Odtnhj

While training a pretty standard convolutional net, I discovered a weird bug. Everything starts out fine with a nice loss curve, but suddenly the loss goes to nan. I was able to trace the nans back all the way to the input pipeline.

As you can see, I am printing the errors before and after they are batched using tf.train.shuffle_batch(). The second print comes up as nan, and this is propagated all the way through the question.

What might be causing this? I have played around with different values of capacity, threads, etc.

The code and context is below. Nans are appearing in the batched before/after images, but not the before/after image.

I should note that the tfrecord files have an arbitrary number of examples in them, but I believe that this shouldn't matter for the enqueue/dequeue operations.

def input_pipeline(self, filenames, batch_size, num_epochs=None):
 """Function that creates a highly abstracted input pipeline consisting
 of a bunch of threads and queues given a few simple parameters.

 See https://www.tensorflow.org/versions/r0.10/how_tos/reading_data/index.html#multiple-input-pipelines
 for more information and in-depth explanations.

 Args:
 - filenames: a list of filenames of tfrecords files
 random.shuffle(filenames)

 train_filenames = filenames
 train_filename_queue = (
 tf.train.string_input_producer(train_filenames,
 num_epochs=num_epochs,
 shuffle=True,
 seed=1))

 before_image, after_image, mask_image = (
 self._read_and_decode_reach_tfrecords(train_filename_queue))

 # min_after_dequeue defines how big a buffer we will randomly sample
 # from -- bigger means better shuffling but slower start up and more
 # memory used.
 # capacity must be larger than min_after_dequeue and the amount larger
 # determines the maximum we will prefetch. Recommendation:
 # min_after_dequeue + (num_threads + a safety margin) * batch_size
 min_after_dequeue = 1000
 saftey_margin = 3
 capacity = min_after_dequeue + (3 + saftey_margin) * batch_size
 capacity = 2000

 before_image = tf.Print(before_image,[tf.reduce_mean(before_image + after_image)], "pre_shuffle: ")
 mask_image = tf.Print(mask_image, [tf.reduce_mean(mask_image)], "pre_shuffle_mask: ")

 before_images, after_images, mask_images = (
 tf.train.shuffle_batch(
 [before_image, after_image, mask_image], batch_size=batch_size, 
 capacity=capacity, min_after_dequeue=min_after_dequeue, 
 num_threads=5, seed=1))

 before_images = tf.Print(before_images,[tf.reduce_mean(before_images + after_images)], "post_shuffle: ")