Odtnhj

Similar to Tensorflow batch_join's allow_smaller_final_batch doesn't work?, I want to pass a batch of images to TensorFlow.

I have 365 images on disk, and my batch size is 100. This means, that the last run must take 65 images. But I cannot achieve that.

Here is what I succeed to do, reproducing the Eypros' answer:

for _ in range(nthreads):
 image_list = load_images(input_queue.dequeue())

image_batch = tf.train.batch_join(image_list, batch_size=100, 
 enqueue_many=True, allow_smaller_final_batch=True, 
 capacity=10)
for n in range(3):
 print (n, len(sess.run([image_batch])))

coord.request_stop()
coord.join(threads)

print (n+1, len(sess.run([image_batch ])))

I get the expected

0 100

1 100

2 100

3 10

But if I set capacity to 65, I don't get the desired 65 files in the last batch, I only get 20. I must add that this happpens when nthreads = 4. When I reduce the number of trheads, the outcome is even worse.

What I tried to do, was to query the input queue, and sleep a bit before coord.request_stop().

numq = sess.run(input_queue.size())
print ('after ', n, ' batches, input_queue size:', numq)
if numq > 0:
 time.sleep(0.08)
 numq = sess.run(input_queue.size())
 print ('after sleep, input_queue size:', numq)

This helps a bit, but if sleep is too long (i.e. the input queue becomes 0), my last sess.run() gets infinitely stuck. I don't know why.

I hate the sleep() hack. I am looking for a clean, efficient and reliable way to consume all images in multithreaded session.

I notice that tf.train.batch_join is deprecated, but I don't know how to convert my simple logic to the suggested tf.data.Dataset.interleave(...).batch(batch_size). That is, I don't understand what to write for interleave. Maybe, if I use Dataset, my problem will be easily resolved?