Odtnhj

I want to get output of a Spark application(which we only use core Spark and people working on the project do not want to change it to Spark SQL) as Parquet or Avro files.

When I look for these two file types, I couldn't find any example without DataFrames, or in general Spark SQL. Can I achieve this without using SparkSQL?

My data is tabular, it has columns but in the processing, all data will be used, not a single column. It's columns are decided at runtime, so there is no "name,ID,adress" kinda generic columns. It looks like this:

No f1 f2 f3 ...
1, 123.456, 123.457, 123.458, ...
2, 123.789, 123.790, 123.791, ...
...

You can’t save an rdd in parquet without converting it to dataframe. Rdd does not have schema but parquet file is in columnar format which needs schema, so we need to convert it to dataframe.

You can use createdataframe api

I tried this and it works like a champ...

public class ParquetHelper

 static ParquetWriter<GenericData.Record> writer = null;
 private static Schema schema;

 public ParquetHelper(Schema schema, String pathName)

 try 
 Path path = new Path(pathName);
 writer = AvroParquetWriter.
 <GenericData.Record>builder(path)
 .withRowGroupSize(ParquetWriter.DEFAULT_BLOCK_SIZE)
 .withPageSize(ParquetWriter.DEFAULT_PAGE_SIZE)
 .withSchema(schema)
 .withConf(new Configuration())
 .withCompressionCodec(CompressionCodecName.SNAPPY)
 .withValidation(true)
 .withDictionaryEncoding(false)
 .build();
 this.schema = schema;
 catch (IOException e) 
 // TODO Auto-generated catch block
 e.printStackTrace();
 
 

 /*
 * 
 */
 public static void writeToParquet(JavaRDD<Record> empRDDRecords) throws IOException 

 empRDDRecords.foreach(record -> 
 if(null != record && new RecordValidator().validate(record, schema).isEmpty())
 writeToParquet(record);
 // TODO collect bad records here
 );

 writer.close();