Odtnhj

I was following the official TF giude to use the tf.data.Dataset API for building data pipeline, but I found it ~2 times slower than my python data pipeline using hdf5.

Here's my experiment result:

TFRecordDataset: 660 thousand samples/second 
hdf5 reader + feed_dict to placeholder: 1.1 million samples/second

My experiment setting is:

- batch size=1000,

- print log every 10 million samples to show the time,

- run a fake model that only takes data as input and do not compute anything.

what's worse is that when I set batch size=10000, it becomes:

TFRecordDataset: 760 thousand samples/second 
hdf5 reader + feed_dict to placeholder: 2.1 million samples/second

Here's my code for reading tfrecord

def parse_tfrecord_batch(record_batch):
 FEATURE_LEN = 29
 dics = 
 'label': tf.FixedLenFeature(shape=(), dtype=tf.int64),
 'feature_id': tf.FixedLenFeature(shape=(FEATURE_LEN,), dtype=tf.int64),
 'feature_val': tf.FixedLenFeature(shape=(FEATURE_LEN,), dtype=tf.float32)
 
 parsed_example = tf.parse_example(record_batch, dics)
 return parsed_example['feature_id'], parsed_example['feature_val'], parsed_example['label']

class fakeModel:
 def __init__(self, train_filenames):
 self.graph = tf.Graph()
 with self.graph.as_default():
 dataset = tf.data.TFRecordDataset(train_filenames)
 dataset = dataset.repeat()
 dataset = dataset.batch(1000)
 dataset = dataset.map(parse_tfrecord_batch, num_parallel_calls=1)
 dataset = dataset.prefetch(1000)
 self.iterator = dataset.make_initializable_iterator()
 self.id, self.wt, self.label = self.iterator.get_next()

 self.train_preds = tf.identity(self.lbl_hldr)

I've tune the num_parallel_calls to 2, 10. Not working.

I've also tuned the prefetch(n) from 1 to 1000, which has little improvement.

My question is:

Is there any way to improve my tfrecord data pipeline? Am I missing something in my code?

Appreciate it for any help.