Keras: Image Augmentation- flow - hanging forever and not training

[gattia]

I am trying to train a segmentation network. I am running into issues when trying to use image augmentation. Below is attached my function that sets the generator, fits it (though I dont think I really need to), it then zips the label and image generators.

It is when trying to zip these that my program hangs and never moves on. Playing around it seems that it has to do with the flow function. Whenever I call the flow function it just runs on it's own indefinitely.

I have made some changes to the image.py file to allow for 3D manipulations, but I have check/doublechecked / even gone back and re-implemented my changes in a clean copy of image.py to make sure I wasn't changing anything that should effect this process.
image.py.zip

I've attached a zip of my updated image.py. Most of the functions at the start are left as the 2D counterparts, its within the random_transform that most of the 3D manipulations are located.

Does anyone have any thoughts?

Thanks,

Anthony.

def train(dataLocation):
    print('-'*30)
    print('Creating and compiling model...')
    print('-'*30)
    ## Augmentation Stuff
    seed = 1
    image_datagen = ImageDataGenerator(rotation_range=15, horizontal_flip=True, order=3) 
    mask_datagen = ImageDataGenerator(rotation_range=15, horizontal_flip=True, order=0)

    image_datagen.fit(trainImages2, augment=True, seed=seed) 
    mask_datagen.fit(trainLabels2, augment=True, seed=seed)
    print('start image generator')
    image_generator = image_datagen.flow(trainImages2, batch_size=2, seed=seed)
    print('start label generator')
    mask_generator = mask_datagen.flow(trainLabels2, batch_size=2, seed=seed)
    print('start zip')
    train_generator = zip(image_generator, mask_generator)
    print('done zipping? ')

    print('loadingModelweights')
    #Get and fit model 
    model = get_vnet()
    model.load_weights(dataLocation + 'vNet_Simplified_Femur_March_5_2017_UpdatingWeights_fromTibia.hdf5')
    model_checkpoint = ModelCheckpoint(dataLocation + 'vNet_Simplified_Femur_March_11_2017_UpdatingWeights_fromTibia_Augmentation.hdf5', monitor='loss', save_best_only=True)

    print('-'*30)
    print('Fitting model...')
    print('-'*30)
    model.fit_g

train(dataLocation)

[patyork]

By definition, you can't zip two infinite generators. You'll need to create a third generator to pick one yielded item from each of the original generators, and then yield them.

[gattia]

Thanks for that. Do you have any suggestions or do you know of any resources that have done something similar? All I'll need is something general. I've never done something of the sort and couldn't find anything on a quick search.

If not, no worries, I'll try to figure it out and post back here for future reference.

Thanks,

Anthony

On Mar 11, 2017, at 5:38 PM, Pat York [e-mail] wrote:

By definition, you can't zip two infinite generators. You'll need to create a third generator to pick one yielded item from each of the original generators, and then yield them.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[joelthchao]

def combine_generator(gen1, gen2):
    yield (gen1.next(), gen2.next())

[gattia]

Thanks a million for that advice. I havent had a chance to try and implement (on vacation), will try this afternoon and report back. I had tried something similar... almost exactly the same.... but used return and not yield.

Return just gave me an array. So, yield will return the augmentation generator and when the result of the function is called it will continue to produce new augmentations forever? and not just the same one over and over? Just want to make sure I understand and am using it correctly.

Thanks,

Anthony.

[joelthchao]

So, yield will return the augmentation generator and when the result of the function is called it will continue to produce new augmentations forever? and not just the same one over and over?

Yes, yield will make this function a generator and iteratively produce next value without end.

[gattia]

@joelthchao Im assuming this isn't an issue with the combine_generator yield, but can't think of anything else. Im getting an error when trying to train saying that the generator is yielding None. The error is below, with my code at the bottom.

Thanks again for your help.

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 429, in data_generator_task
    generator_output = next(self._generator)
StopIteration

Traceback (most recent call last):
  File "trainNetwork.py", line 56, in <module>
    network.trainAugmentation(model, trainImages[:70,:,:,:,:], trainLabels[:70,:,:,:,:], dataLocation, newWeights, oldWeights, nb_epoch=60, batch=2, rotation_range=6, horizontal_flip=True, seed=1)
  File "/vol/programs/neuralSeg/simpleOneTissueNetwork/simpleNetwork.py", line 639, in trainAugmentation
    model.fit_generator(combinedGenerator, samples_per_epoch=70, nb_epoch=nb_epoch)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1532, in fit_generator
    str(generator_output))
ValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None

def combine_generator(gen1, gen2):
    yield(gen1.next(), gen2.next())

def trainAugmentation(model, images, labels, dataLocation, newWeights, oldWeights, nb_epoch=60, batch=2, rotation_range=6, horizontal_flip=True, seed=1):
    print('-'*30)
    print('Create image generators...')
    print('-'*30)
    image_datagen = ImageDataGenerator(rotation_range=rotation_range, horizontal_flip=horizontal_flip, order=3) 
    mask_datagen = ImageDataGenerator(rotation_range=rotation_range, horizontal_flip=horizontal_flip, order=0)
    print('-'*30)
    print('Start fitting image generators...')
    print('-'*30)
    image_datagen.fit(images, augment=True, seed=seed)
    mask_datagen.fit(labels, augment=True, seed=seed)
    print('-'*30)
    print('Start image generators...')
    print('-'*30)
    image_generator = image_datagen.flow(images, batch_size=2, seed=seed, shuffle=True)
    mask_generator = mask_datagen.flow(labels, batch_size=2, seed=seed, shuffle=True)
    print('-'*30)
    print('Combine image generators...')
    print('-'*30)
    combinedGenerator = combine_generator(image_generator, mask_generator)
    print('-'*30)
    print('Fitting model...')
    print('-'*30)
    model.load_weights(dataLocation + oldWeights)
    model.fit_generator(combinedGenerator, samples_per_epoch=70, nb_epoch=nb_epoch) 

[joelthchao]

Sorry my bad. I forget to make it loop.

def combine_generator(gen1, gen2):
    while True:
        yield(gen1.next(), gen2.next())

[gattia]

No need to apologize, you're helping me out a ton. That did the trick!

Now I'm having NaN issue with my loss, but I'm sure that's something I've screwed up inside the image augmentation file :).

Thanks!

[gattia]

For anyone that might come along this and want some help with generators, I came across this article today that helps.

http://pybit.es/generators.html

[mongoose54]

@gattia Did you manage to resolve this? I am following the official example from Keras: https://keras.io/preprocessing/image/

# we create two instances with the same arguments
data_gen_args = dict(featurewise_center=True,
                     featurewise_std_normalization=True,
                     rotation_range=90.,
                     width_shift_range=0.1,
                     height_shift_range=0.1,
                     zoom_range=0.2)
image_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(**data_gen_args)

# Provide the same seed and keyword arguments to the fit and flow methods
seed = 1
image_datagen.fit(images, augment=True, seed=seed)
mask_datagen.fit(masks, augment=True, seed=seed)

image_generator = image_datagen.flow_from_directory(
    'data/images',
    class_mode=None,
    seed=seed)

mask_generator = mask_datagen.flow_from_directory(
    'data/masks',
    class_mode=None,
    seed=seed)

# combine generators into one which yields image and masks
train_generator = zip(image_generator, mask_generator)

model.fit_generator(
    train_generator,
    steps_per_epoch=2000,
    epochs=50)

but it loops indefinitely in zip()

[gattia]

It's been about a year since I worked with this, and it's not my forte, however, I think the problem is that your train_generator should be a function with "yield" in it. I've re-written what you have posted in the way that I think will fix your problem.

# combine generators into one which yields image and masks
def combineGenerator(gen1, gen2):
    while True:
        yield(gen1.next(), gen2.next())
train_generator = combineGenerator(image_generator, mask_generator)

Below I have copied snippets of code that Im currently using that rely on this method. Things might be a bit wonky because I wrote a custom version of image augmentation that works in 3D, but you should be able to get the general point.

def combine_generator(gen1, gen2):
    while True:
        yield(gen1.next(), gen2.next())    

image_datagen = threeDimageAug.ImageDataGenerator(rotation_range=rotation_range, horizontal_flip=horizontal_flip, order=1, height_shift_range=height_shift_range, width_shift_range=width_shift_range, slice_flip=True)`
mask_datagen = threeDimageAug.ImageDataGenerator(rotation_range=rotation_range, horizontal_flip=horizontal_flip, order=0, height_shift_range=height_shift_range, width_shift_range=width_shift_range, slice_flip=True)
image_generator = image_datagen.flow(images[:,:,:,:,:], batch_size=batch, seed=seed, shuffle=True)
mask_generator = mask_datagen.flow(labels[:,:,:,:,:], batch_size=batch, seed=seed, shuffle=True)
combinedGenerator = combine_generator(image_generator, mask_generator)

model.fit_generator(combinedGenerator, steps_per_epoch=steps_per_epoch, epochs=iterations_save, callbacks=[csv_checkpoint], verbose=verbose)`

[azmosavi]

@gattia what if we want to write the augmented image and masks on the disk. How can we use the
train_generator = zip(image_generator, mask_generator)
to write on the disk the generated images a long with their masks!

I'm doing the following but the generated augmented masks don't match the generated augmented images:

image_datagen.fit(images, augment=True, seed=seed)
mask_datagen.fit(masks, augment=True, seed=seed)

image_gen=image_datagen.flow(images, batch_size=1,shuffle=False, save_to_dir=dir_image, save_prefix='aug',save_format='png')

mask_gen=image_datagen.flow(masks, batch_size=1,shuffle=False, save_to_dir=dir_mask, save_prefix='aug',save_format='png')

i = 0
for batch in zip(image_gen, mask_gen):
i += 1
if i > 10:
break