Odtnhj

To simplify the input parameters and code, I've generated an input list (in real live it is a lot of csv files with digits inside). All input numbers should be grouped by the single regularity rule. Numbers that can not be described with any regularity rule should be removed from the output result.

input example:

[1010,1020,1050,1030,1022,1880,1940,1900,1920,2010,3100]

output result example:

[[1010,1020,1030,1050],[1880,1900,1920,1940]]

[1010,1020,1030,1050] has increment step 10 and sequence has an exclusion (1040). [1880,1900,1920,1940] has increment step 20, sequence without exclusions.

Starting solution with spark on java

public static void main(String args) throws Exception 
 Logger.getLogger("org").setLevel(Level.OFF);
 SparkConf conf = new SparkConf().setAppName("reduce").setMaster("local[*]");
 JavaSparkContext sc = new JavaSparkContext(conf);

 List<Integer> inputIntegers = Arrays.asList(1010,1020,1050,1030,1022,1880,1940,1900,1920,2010,3100);
 JavaRDD<Integer> integerRdd = sc.parallelize(inputIntegers);
 System.out.println("Check count of objects: " + integerRdd.count());

 System.out.println(integerRdd.collect());
 JavaRDD<Integer> sorted = integerRdd.sortBy(x -> x, true, 1);
 System.out.println("Check sorted collection:");
 System.out.println(sorted.collect());
 /* TODO
 - enrich data with increment step
 - group by increment step (increment step should be calculated in run time)
 - filter out single values 
 */

I think I need to use increment step but I do not know how to manage data from different RDDs (in case of using spark), to have a chance save incremental step in current spark RDD

Does anyone have ideas how to resolve my spark issue or how it can be solved with different tools?

P.S.

Spark (or any equivalent technology) is mandatory because solution must be big data oriented and as result resolved in distributed system.