Keras: Extending ImageDataGenerator

[tetmin]

Is there an easy way to write generator extensions for Keras? I'd like to use some of the ImageDataGenerator preprocessing steps but also add some of my own such as randomly occluding areas of the image, adding noise etc. If not I can add some of these things to the ImageDataGenerator class in Keras, is that something that would be useful?

Also can I just double check that the ImageDataGenerator does indeed generate batches of examples in a non-locking way, thereby not causing any GPU training bottlenecks?

[oeway]

I would be interested if someone can make the ImageDataGenerator extendable.

Plus, another feature I would like to have is to separate the random variables generator part which used by ImageDataGenerator, thus the variables can be saved somewhere and make the process reproducible, you don't need to actually store the transformed images, just store the random variables, as a dictionary maybe. These random variables can be feed to into a ImageDataGenerator to produce the results.

This going to be super useful in case of dense prediction(such as segmentation), in which we need to do data augmentation on both input image and output image, we can make two ImageDataGenerator, and use the separated random generator to produce the random variables, and then feed into two ImageDataGenerator, so the spatial transform such as rotation are synchronized on both input and output.

I would begin to work on this, please let me know if anyone else have ideas regarding this.

EDIT: perhaps you would think there is a seed option in flow and flow_from_directory, yes, indeed, you can reproduce the image by controlling the seed, but the problem comes if you want to apply different configuration to two ImageDataGenerator, for example, add noise on the input image but not on the output image, in that case, even you use the same seed, the spatial transformation could be different. In that sense, perhaps, we should use different seed for different type of data augmentation function, what do you think?

[oeway]

I am working on improve ImageDataGenerator but would be great anyone interested could provide suggestions and feedbacks.
The main idea to make it more flexible, and support dense prediction dataset. Here are a bunch of ideas I would like to discuss:

• provide a customizable pipeline( a list of functions) to let user add/change their own preprocessing. There will be a pipeline contains random_transform and standardize by default, and user can insert any function in any position of the pipeline. ( For example, #2738 can be resolved, and people can use opencv scikit-image or other libraries)
• reproducible random spatial transform by using a separate random seed from the keyword, so dense prediction data has image-in and image-out can use two different ImageDataGenerator which have the same random spatial transform. (related to #3059)
• provide featurewise_standardize_axis and samplewise_standardize_axis for more flexible standardization, for example, you can pass (0, 2, 3) to do a channel-wise normalization(related to issue #2559 and PR #2879, also mentioned in #2446).

For flow_from_directory():

• remove the limit for channel number and color mode, by using PIL, we can just provide the PIL image mode as a parameter which gives more flexibility on the image format, channel data type and channel number. Image with ARGB mode png file, or F mode(float) tiff which is widely used in many field should be naturally supported. (also related to #3140)
• provide an optional image file reader keyword to replace the underlying PIL image reader (for flow_from_directory). With a file reader function which takes a path as input and return a numpy array, we can provide much more flexible on the image format. We can read hdf5, npz, pickle file, or for example, you can use a library named openslide to read histopathological images. (also related to #1819 for reading time distributed data)

For fit():

• fit is required for feature-wise standardization and ZCA , and it only takes an array as parameter, there is no fit for directory. For now, we need to manually read a subset of the image to do this fit for a directory. One idea is we can change fit() to accept the generator itself(flow_from_directory), of course, standardization should be disabled during fit.

[pengpaiSH]

@oeway When it comes to visual semantic segmentation, I find out that extending ImageDataGenerator is really necessary! Current ImageDataGenerator will only conduct transformations on X instead of Y. However, in image segmentation scenarios, input image X and output mask Y should be transformed simultaneously.

[oeway]

@pengpaiSH I have a working version right now, you are welcome to try it out. https://github.com/oeway/keras/tree/extendImageDataGenerator (branch: extendImageDataGenerator)

For now, you can customize the pipeline and synchronize two ImageDataGenerator by a __add__(+) operator. You can see more extensions in the following example:

from keras.preprocessing.image import ImageDataGenerator,standardize,random_transform
# input generator with standardization on
datagenX = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    featurewise_standardize_axis=(0, 2, 3),
    rotation_range=180,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    fill_mode='reflect',
    seed=0,
    verbose=1)

# output generator with standardization off
datagenY = ImageDataGenerator(
    featurewise_center=False,
    featurewise_std_normalization=False,
    rotation_range=180,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    fill_mode='reflect',
    seed=0)

def center_crop(x, center_crop_size, **kwargs):
    centerw, centerh = x.shape[1]//2, x.shape[2]//2
    halfw, halfh = center_crop_size[0]//2, center_crop_size[1]//2
    return x[:, centerw-halfw:centerw+halfw,centerh-halfh:centerh+halfh]

def random_crop(x, random_crop_size, sync_seed=None, **kwargs):
    np.random.seed(sync_seed)
    w, h = x.shape[1], x.shape[2]
    rangew = (w - random_crop_size[0]) // 2
    rangeh = (h - random_crop_size[1]) // 2
    offsetw = 0 if rangew == 0 else np.random.randint(rangew)
    offseth = 0 if rangeh == 0 else np.random.randint(rangeh)
    return x[:, offsetw:offsetw+random_crop_size[0], offseth:offseth+random_crop_size[1]]

datagenX.config['random_crop_size'] = (800, 800)
datagenY.config['random_crop_size'] = (800, 800)
datagenX.config['center_crop_size'] = (512, 512)
datagenY.config['center_crop_size'] = (360, 360)

# customize the pipeline
datagenX.set_pipeline([random_crop, random_transform, standardize, center_crop])
datagenY.set_pipeline([random_crop, random_transform, center_crop])

# flow from directory is extended to support more format and also you can even use your own reader function
# here is an example of reading image data saved in csv file
# datagenX.flow_from_directory(csvFolder, image_reader=csvReaderGenerator, read_formats={'csv'}, reader_config={'target_size':(572,572),'resolution':20, 'crange':(0,100)}, class_mode=None, batch_size=1)

dgdx= datagenX.flow_from_directory(inputDir, class_mode=None, read_formats={'png'}, batch_size=2)
dgdy= datagenY.flow_from_directory(outputDir,  class_mode=None, read_formats={'png'}, batch_size=2)

# you can now fit a generator as well
datagenX.fit_generator(dgdx, nb_iter=100)

# here we sychronize two generator and combine it into one
train_generator = dgdx+dgdy

model.fit_generator(
        train_generator,
        samples_per_epoch=2000,
        nb_epoch=50,
        validation_data=validation_generator,
        nb_val_samples=800)

[pengpaiSH]

@oeway Thank you for your quick response and your enhancement for ImageGenerator which I think is really really necessary. Below are my confusions:

1. As far as I understand, center_crop and random_crop is our customized augmented transformation, right?
2. In the defined pipeline, datagenX.set_pipeline([random_crop, random_transform, standardize, center_crop]), where does random_transform and standardize come from? They are not even defined such as center_crop.
3. I don't understand why datagenX should be fit_generator()? I think only model should have fit_generator()

[oeway]

Sorry for the confusion, I will write documents about it when I have
time.

1, yes, you just need to provide a function and put in the pipeline,
center_crop is one example, it shows how you can define the function and
how to set arguments for your function.

2, I forgot to put the import line, so random_transform and standardize are
to predefined function which I split from the previous ImageDataGenerator,
they are from the same module as ImageDataGenerator. Just do:

from keras.preprocessing.image import standardize,random_transform

3, fit_generator and fit is for some augment function such as feature_wise
standardization which require a pre computed mean and std value. I added
fit_generator for flow_from_directory(see my previous post for why we need
that), and I generalized the fit interface so that you can now add
customized function which support pre-fit.

I will document all the features if there are more people interested, and
perhaps make a PR. In the meantime, welcome to try and provide feedback.

On Tue, Aug 9, 2016 at 10:49 Pai Peng [email protected] wrote:

@oeway https://github.com/oeway Thank you for your quick response and
your enhancement for ImageGenerator which I think is really really
necessary. Below are my confusions:

1. As far as I understand, center_crop and random_crop is our
   customized augmented transformation, right?
2. In the defined pipeline, datagenX.set_pipeline([random_crop,
   random_transform, standardize, center_crop]), where does
   random_transform and standardize come from? They are not even defined
   such as center_crop.
3. I don't understand why datagenX should be fit_generator()? I think
   only model should have fit_generator()

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/3338#issuecomment-238492225,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdNy2ZFmGvxVU_i2jfVxSHOREIHs7dGks5qeD8ogaJpZM4JXAo4
.

[pengpaiSH]

@oeway Thank you for your clarifying. Perfect! By the way, the transformations defined in the pipeline come in order or not ?

[oeway]

@pengpaiSH Welcome, of course, the idea is use an ordered list to defined the pipeline, and you can change the order of the pipeline.
Let me know if you have any problem.

[landersson]

I also need random cropping, better scaling etc and was just about to start implementing something like this myself. Looks very useful. Will you try to get this merged into the main repo?

[burgalon]

Any plans to merge @oeway branch? I'm specifically interested in the CSV reader option

[ncullen93]

@burgalon It's very straight-forward .. like maybe 10 lines of code.. to extend the data generator to read csv.. Still agree it should be merged but in case you dont want to wait :) . I attached some very unpolished code which works perfectly for me (see 'flow_np_from_directory' function and 'NumpyDirectoryIterator' class specifically) to read in .npy files in the same directory structure as the image version, e.g.:

• dir/
  -class1/
  ... file1.npy
  ... file2.npy
  -class2/
  ... file3.npy

image.zip

[burgalon]

Thanks @ncullen93

[johnny5550822]

@oeway @burgalon @ncullen93 also, I notice that .flow_from_directory has a way to make the original input to target_size, but .flow() does not....maybe we should add this functionality to the ImageDataGenerator so the for example I can resize my image size of 21x21 to 224x224.

I don't find any function like what I have pointed out above.

[ncullen93]

Hm yeah.. I noticed that too. That's probably because the image resizing is done directly on the image object before converting to an array using PIL's resize method (see line 302 in preprocessing/image.py)... .flow() would require resampling a numpy array, probably using scipy's functionality, so I guess it's assumed you will just resize it yourself since the data already fits in memory.

[johnny5550822]

@ncullen93 yup, exactly, but sometime your data may be large array but not image. So it would be nice to provide a way to do resize. I think cv2 can do that nicely. if this functionality is included, I don't need to write a custom datagenerator to do the resize using cv2.

[ncullen93]

true. i don't make those decisions unfortunately. In the meantime, here's a very slightly modified image.py file with a big commented section on line 719 ( look for the "### ADD CODE HERE TO RESHAPE ARRAY TO TARGET SIZE ####") for you to add one line of code to upsample/reshape your array however you like! Just add code to upsample/reshape your array and then you can use "target_size" on the flow() function just like flow_from_directory(). It's very simple. Then replace this image.py file with that in your keras directory, and reinstall. That's all I personally can offer.

image.zip

[johnny5550822]

@ncullen93 Wow, thanks! I will look at that.

[stratospark]

I extended @oeway 's ImageDataGenerator fork to use a multiprocessing Pool. I got a pretty good speedup: https://github.com/stratospark/food-101-keras/blob/master/tools/image_gen_extended.py

This was to fully utilize a Titan X GPU as I write about here: http://blog.stratospark.com/deep-learning-applied-food-classification-deep-learning-keras.html

[ncullen93]

Can you comment a little more on it. Does it support both arrays and directory sampling? Seems to only be arrays from my quick look. Directory sampling is probably the biggest bottleneck in my opinion, since bigger nets typically coincide w/ datasets that dont fit in memory - so the speedup margins on using this w/ directory sampling would be much greater. anyways, implementing just the multiprocessing speedup on the actual keras image generator without the pipeline stuff (since it's not standardized) would be a huge contribution in my opinion

[stratospark]

@ncullen93 I'm planning on cleaning up the code soon and posting a PR so it can hopefully be merged into the main Keras branch. Enough to maintain compatibility with the mainline Keras features.

It was a quick hack to support my use case, augmenting images in memory fast enough to keep my GPU close to full utilization. I had to disable any augmentation related to fitting the model, such as normalization and zca. This was due to the locks not being able to be pickled when doing a multiprocessing map.

Another change I had to make was having to explicitly pass a multiprocessing.Pool. This is due to the fact that Python multiprocessing forks the process, and thus it is easy to run into out of memory errors if you fork after loading your images. I have to create the pool at the very beginning of the script so it uses as little resources as possible. I'm pretty new at this, so there is probably a more elegant way of handling this. One thing that I would like to fix is when interrupting the training in Jupyter Notebook, it kills the processes so I have to restart the kernel and load up all the images to do another run. This really breaks the flow of trying different models and parameters!

To support reproducible results, I also had to create separate random number generators for each process, so each process didn't return the same random numbers (and thus the exact same images as each other). I know I left a few numpy default random generators, so I need to fix that up.

[joeyearsley]

This would fit nicely in the contrib library: https://github.com/farizrahman4u/keras-contrib

[ahundt]

@stratospark I agree with @joeyearsley you should add a PR for keras-contrib, particularly now that keras-2 is out.

[rdelassus]

Be careful with @oeway example, it doesn't work if you have a different reader_config for images and masks. Because he used dictionaries as default arg in the DirectoryIterator class init function.

so every time flow_from_directory is called, read_formats and reader_config are overwritten.

[ahundt]

I've been using and adding to an extended image data generator for fcn

code:
https://github.com/aurora95/Keras-FCN

ongoing work with it:
https://github.com/farizrahman4u/keras-contrib/issues/63

[mptorr]

@oeway at the very end of your code there is this

        model.fit_generator(
            train_generator,
            samples_per_epoch=2000,
            nb_epoch=50,
            validation_data=validation_generator,
            nb_val_samples=800)

There is no reference for validation_generator—where does it come from?

[oeway]

@mptorr validation_generator is get use the same way as train_generator, of course, you may want to change the inputDir to another folder which contains validation data.

[ahundt]

I'm specifically seeking proposals for ImageDataGenerator for segmentation tasks at https://github.com/fchollet/keras/issues/6538, but if there is a better way to generalize I'd love to hear it.

[yash0307]

@oeway, I see that your fork version has not been update for a while. Will your keras/preprocssing/image.py still work with current keras version ?

[hassaan90]

model.fit can work with multiple input/output network, while model.fit_generator only take tuple as input. some one working to match this feature for generator ?

[ahundt]

@hassaan90 your tuple can contain tuples/lists so the generator works for multiple input/output as well.

In [14]: xy = (('x1','x2'), ('y1','y2'))
In [15]: (x1,x2), (y1,y2) = xy
In [16]: print x1, x2, y1, y2
x1 x2 y1 y2

another example:

In [19]: xy = (('a','b'), ('c','d'))
In [20]: (xy1, xy2) = xy
In [21]: (x1, y1) = xy1
In [22]: (x2, y2) = xy2
In [23]: print x1, y1, x2, y2
a b c d

[krishnadusad]

So, I wanna use the datagenerator to flow images from a directory _without_ any classes. This is to train an autoencoder. I've looked up the docs but can't find it. Is this something that has been implemented in the extended version?

[alexhernandezgarcia]

@krishnadusad you don't need much of an extension for that functionality. You can call flow() with y=None and it should work. Alternatively, you could even write your own generator, where you call flow() with random/no labels and then yield only the images. Something like this:

def my_generator(x, batch_size):
    gen = ImageDataGenerator()
    for batch in gen.flow(x, y=None, batch_size):
        yield(batch[0])

[krishnadusad]

Yup, ended up writing my own generator which did exactly that. Anyway, thanks a ton!

[mahernadar]

Hello guys :)

I have a question regarding the flow.from.directory():

My case is a 5 class classification problem and i am trying to fine-tune the VGG16 pre-trained network on my data.

my question is: how can we access the actual data and labels generated by the flow.from.directory()

i am trying to do so with the following code:

datagen = ImageDataGenerator(rescale=1. / 255)

# #generating the tuples
train_tupple_generator = datagen.flow_from_directory(
train_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
class_mode='categorical',
shuffle=False)

 #here i want to store the generated labels
train_labels = np.empty((nb_train_samples,5)) 

def my_generatorT():
    idx = 0
    while 1:
        dataT, labelsT = train_tupple_generator.next()  #python iterator;

      train_labels[idx*batch_size:(idx+1)*batch_size, :] = labelsT
        idx += 1
        yield dataT

bottleneck_features_train = model.predict_generator(
    my_generatorT(), nb_train_samples // batch_size)

np.save(open('bottleneck_features_train_5_classes2.npy', 'wb'),
        bottleneck_features_train)
np.save(open('training_labels.npy', 'wb'),
        train_labels)

I am getting this error in the line "train_labels[idxbatch_size:(idx+1)batch_size, :] = labelsT":

ValueError: could not broadcast input array from shape (20,5) into shape (0,5)

You have to not that i am kind of a beginner in python (:

Thank!

[rdelassus]

Github Issues are usually used to report bugs or ask for enhancements .
If you have questions reguarding your code, you will find people ready to answer them on stackoverflow

[GitHubKay]

I would suggest to implement an automated function with wich we can choose different augmentation for different batches within one batch. Or do i oversee this existing function somehow?

[Niladri365]

@oeway
I am getting AttributeError: 'ImageDataGenerator' object has no attribute 'set_pipeline' while trying to
do preprocessing using the following code
train_datagen = ImageDataGenerator(
rotation_range=30,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
contrast_param = np.random.uniform(0.7,1.3)
brightness_param = np.random.uniform(0.7,1.3)
color_param=np.random.uniform(0.7,1.3)
def contrast_enhancement(x,contrast_param):
x = np.asarray(ImageEnhance.Contrast(x).enhance(contrast_param))
return x
def brightness_adjustment(x,brightness_param):
x=np.asarray(ImageEnhance.Brightness(x).enhance(brightness_param))
return x
def color_adjustment(x,color_param):
x=np.asarray(ImageEnhance.Color(x).enhance(color_param))
return x
train_datagen.set_pipeline([contrast_enhancement, brightness_adjustment, color_adjustment])
train_generator = train_datagen.flow_from_directory(
'dir_name',
target_size=(256,256),
classes=['0','1','2','3','4'],
batch_size=100,
shuffle=True
)

[tinalegre]

@oeway thanks for providing your ImageDataGenerator class. I am wondering how to save some samples of training images/masks after the augmentation has been done? I was trying something like: train_generator = dgdx+dgdy and then x_batch, y_batch = next(train_generator), but didn't work out, I guess because both images and masks belong to the same generator, any suggestions how to solve this issue? Of course, I've passed a save_to_dirparameter when defining the ImageDataGenerator;-)

[oeway]

@NickShahML I guess you checked out the master branch, you need to check out the branch named "extendImageDataGenerator", Or you can simply download this file: https://github.com/oeway/keras/blob/extendImageDataGenerator/keras/preprocessing/image.py and place it with your code, then you can import the function from image.py .

@tinalegre you can write a function to save your images and add that function to your pipeline, something like this:

# the following code is not tested.

x_save_count = 0
def save_x(x, **kwargs):
    img = Image.fromarray(x, 'RGB')
    img.save('x_{}.jpg'.format(x_save_count))
    x_save_count +=1
    return x

y_save_count = 0
def save_y(y, **kwargs):
    img = Image.fromarray(x, 'RGB')
    img.save('y_{}.jpg'.format(y_save_count))
    y_save_count +=1
    return y

# customize the pipeline
datagenX.set_pipeline([random_crop, random_transform, standardize, center_crop, save_x])
datagenY.set_pipeline([random_crop, random_transform, center_crop, save_y])

[tinalegre]

@oeway thanks for the clarification. I'm wondering how would you convert your masks/labels to categorical format?
Where would you call the to_categorical function in the ImageDataGenerator for the masks? Would it make sense from your example to do sth. like: train_generator = dgdx+to_categorical(dgdy)?

[oeway]

@tinalegre if you have a categorical label, you should use the original ImageDataGenerator provided by Keras, the whole purpose of this extension it to do dense prediction where your target is another image. However, if you want, you can still use it, but not like train_generator = dgdx+to_categorical(dgdy), you could do datagenY.set_pipeline([..., to_categorical]), and then train_generator = dgdx+dgdy.

[tinalegre]

Actually, I've a dense prediction problem
(image segmentation), my labels are of size [im_rows,im_cols,1], where
the label-values range from [0,1,...,no_labels-1], don't we need to map
the masks/labels to categorical? At least, this is what I found in the
description of the categorical_crossentropy loss here. I want to use a per-pixel multi-class cross-entropy loss based on the
softmax activations in my model's last layer.

[rdelassus]

@oeway

in your image.py module, we can find this code line 718

class DirectoryIterator(Iterator):

    def __init__(self, directory, image_data_generator,
             color_mode=None, target_size=None,
             image_reader="pil", read_formats={'png','jpg','jpeg','bmp'},
             reader_config={'target_mode': 'RGB', 'target_size':None},
             dim_ordering=K.image_dim_ordering,
             classes=None, class_mode='categorical',
             batch_size=32, shuffle=True, seed=None,
             save_to_dir=None, save_prefix='',
             save_mode=None, save_format='jpeg'):

note that read_formats and reader_config are defaulted to dictionaries. This is bad because if you create a DirectoryIterator with a reader_config A end then create an other with a reader_config B, they will end with both reader_config B. they will share the same dictionnary.

you should instead default it to None, and give them a default value inside the __init__ function core if they are None.

[oeway]

@tinalegre in that case, it make sense, but I don't think the keras function to_categorical(https://github.com/fchollet/keras/blob/master/keras/utils/np_utils.py#L7) will work with an image, you may want to write another to_categorical which takes an image in and output another image, and then you can add it to the pipeline.

@rdelassus Sorry for missing your comments before, I don't understand how the bug comes exactly, and I don't have a test case. However, I made some correction according to your suggestion, could you please check line 718, 734-737, is that what you mean? I can't test at the moment, if you can, please test it and it may help others. Thanks.

https://github.com/oeway/keras/blob/extendImageDataGenerator/keras/preprocessing/image.py

[rdelassus]

@oeway

Yes this is the fix. You have the same problem in other functions (l. 192, 537-538)
see here for a description of why the bug occures.

I didn't code a test case, but I noticed the bug because I don't use the same reader_config for my input image and for the associated label. With the faulty implementation, I always ended with the label reader_cofing for both the input and the label.

[tinalegre]

@rdelassus could you please exemplify what did you mean with the fact that you don't use the same reader_config for the input image and for its associated label? In @oeway example he has datagenX = ImageDataGenerator(params) for the images and datagenY = ImageDataGenerator(params) for the labels, do you mean that the params are not the same for both ImageDataGenerators?

[oeway]

@rdelassus Got it now, I am not aware of that. Ok, now I fixed other places as well. Thanks.
@tinalegre you could look at the link @rdelassus provided.

[tinalegre]

@oeway just one little thing, in python3, one should replace xrange with range, in lines 578, 594, 595

[dangz90]

@oeway I'm trying to synchronize both generator like in your example:

# here we sychronize two generator and combine it into one
train_generator = dgdx+dgdy

but I receive the next error:
TypeError: unsupported operand type(s) for +: 'DirectoryIterator' and 'DirectoryIterator'
I don't know if it's supported in the newer version of keras.

Could you help me with this?

[oeway]

@dangz90 It seems you are not using my extension but the keras's native implementation. In order to use it:

1. download https://github.com/oeway/keras/blob/extendImageDataGenerator/keras/preprocessing/image.py and place it to your current folder
2. import the functions you need from image.py, don't import them form keras.

For newer keras version, I didn't test, but I guess the change should be minor, the one I can see is to change the dim ordering format. You will need to replace all the th to channels_first, and all the tf to channels_last to match the new keras api. If you tested it, please let me know whether it works.

[tinalegre]

@oeway thanks for your example on how to save to disk the images/labels (save_x, save_y). Actually, I need to save those files with their original filenames as found in their directories, instead of just using a counter to assign their names. Any ideas on how to achieve that? I would then pass as parameter as well the path to the output directory.

[mbenami]

Hi
Extending ImageDataGenerator can be done with the preprocessing_function variable
just create a class that does any augmentation you would like to add

for example :

class CustomAugmentation(object):
    def __init__(self,
                 histogram_equalization=False,
                 rot90=False,
                 preprocessor_=False):
        self.histogram_equalization = histogram_equalization
        self.rot90 = rot90
        self.preprocessor = preprocessor_

    def __call__(self, x):
        if self.histogram_equalization:
            if np.random.random() < 0.5:
                x = exposure.equalize_hist(x)
        if self.rot90:
            if np.random.random() < 0.2:
                k = 1
                if np.random.random() < 0.5:
                    k = 3
                x = np.rot90(x, k)
        if self.preprocessor:
            x = self.preprocessor(x)
        return x

preprocessor = CustomAugmentation(histogram_equalization=True,
                                  rot90=True,
                                  preprocessor_=inception_v3.preprocess_input)

train_datagen = ImageDataGenerator(preprocessing_function=preprocessor)

[leptdre]

Is there a way to use the ImageDataGenerator for Large class datasets that cannot fitted in splitted folders or put in RAM. I basically have a numpy array with image file names and another one with onehot encoded labels. Is there a way to use ImageDataGenerator in this case ?

3295

[androlleoni]

Hi everyone, I'm new in the field, I'm trying to use the random crop, but I cannot find out a way to include standardize and random_transform from keras.imageDataGenerator. Please someone help me!!

[kechan]

@oeway
I noticed this is a relatively old issue and it is still open. Do you know if your work has been merged into Keras? I am on Keras 2.1.4 now but I don't think this is there. I also took a quick look at your GitHub. Is it possible to do this by subclassing ImageDataGenerator, so I can stay on Keras 2.1.4 (which my project depends on)?

Also, do you have any other new doc/write up?

Thanks!

[jkjung-avt]

I shared how I extended Keras' ImageDataGenerator to support random cropping in this blog post:
Extending Keras' ImageDataGenerator to Support Random Cropping.
https://jkjung-avt.github.io/keras-image-cropping/

I think the same technique could be easily adapted as a solution to the original question.

"Is there an easy way to write generator extensions for Keras? I'd like to use some of the ImageDataGenerator preprocessing steps but also add some of my own such as randomly occluding areas of the image, adding noise etc. If not I can add some of these things to the ImageDataGenerator class in Keras, is that something that would be useful?"

[ChristianEschen]

I am doing semantic segmentation, thus I have both input images X and label images Y of size 1440x1920.
I would like to train a model with 224x224 input size. In order not to
lose information it would be nice to random crop the images of size
224x224.
Furthermore, I would like to do random crop sampling from all
categories from the label image Y in order to have batches with equal class
distributions.
I have investigated https://jkjung-avt.github.io/keras-image-cropping/#comment-3887776756 for random crop.
The problem is that the information for doing random crop from the classes in the label images comes from the label image Y. Thus I guess we need to store the x_index and y_index for random crop from classes in label image Y.
However, I can't see how this is possible.
Eg. you have 2 generators.
datagenY = ImageDataGenerator()
datagenX = ImageDataGenerator()

and I do:

dgdy= datagenY.flow_np_from_directory()
dgdx= datagenX.flow_np_from_directory()

And syncronizing the generators:
training_generator=zip(dgdx,dgdy)

And if I modify image.py I can add a random_crop_from_categories.
However, I do not have access to the pixel index from datagenY in datagenX.

Is it possible to implement a random_crop_from_categories in the image.py file to sample equally from the categories in Y and apply the same crop in X?

[gledsonmelotti]

In the keras command, you have to define the image type, that is, color_mode = 'rgb' or 'grayscale' when using the command flow_from_directory with model.fit_generator. So how do I change color_mode in flow_from_directory to use with 2, 4, 5 and 6 channels?

I thank you for your attention,
Gledson

[jkjung-avt]

@ChristianEschen, I copied my reply on Disqus over here.

You can define your paired_crop_generator() to crop the same region from X (images) and Y (masks) simultaneously. The code below is just quick and dirty hack, but hopefully you get the idea.

def paired_crop_generator(batches, crop_length=224):
    '''
    Assuming batch_X.shape is (N,1440,1920,3) and batch_Y.shape is (N,1440,1920,1),
    this function would crop the same (224,224) region from the XY pair for each image
    in the batch, then return (yield) the cropped batch.
    '''
    while True:
        batch_X, batch_Y = next(batches)
        batch_Xcrops = np.zeros((batch_X.shape[0], crop_length, crop_length, 3))
        batch_Ycrops = np.zeros((batch_X.shape[0], crop_length, crop_length, 1))
        height, width = batch_X.shape[1], batch_X.shape[2]
        dy, dx = random_crop_length, random_crop_length
        for i in range(batch_X.shape[0]):  # loop through all images in the batch
            # select a random top-left corner of the crop, for current image in the batch
            x = np.random.randint(0, width - dx + 1)
            y = np.random.randint(0, height - dy + 1)
            # take the crop (same region) from both X and Y
            batch_Xcrops[i] = batch_X[i][y:(y+dy), x:(x+dx), :]
            batch_Ycrops[i] = batch_Y[i][y:(y+dy), x:(x+dx), :]
        yield (batch_Xcrops, batch_Ycrops)

......
training_generator = zip(dgdx,dgdy)
train_crops = paired_crop_generator(training_generator, 224)

net_final.fit_generator(train_crops, ......)

[jkjung-avt]

@gledsonmelotti, I believe Keras' flow_from_directory() API could only handle 'rgb' or 'grayscale' images. For training data with 2, 4, 5, 6, ... channels, you'd need to implement your own data loading code.

[tinalegre]

@oeway I've a model with multiple-output loss functions and need to provide for each of those losses the ground-truth masks, sth. like.: train_generator = imGen+maskGen+maskGen+maskGen, for three multiple-output loss functions. Any idea, on how to achieve that with the extension you did to the ImageGenerator class? i.e. how to combine the multiple generators in a correct way?

[mbenami]

Hey @tinalegre
have you try to load the 3 mask you need and insert them into 1 list?
then you just need to extract the element in custom loss function that handles this.

[oeway]

@tinalegre @mbenami If your output images are with the same size, what I would do is to combine these them into one image with multiple channels, so you can generate the data as if there are one input image and one output image. with my extension, most of those transformation functions they can support any number of channels.
For the network itself, you will need to use K.concatenate to to combine your outputs into one output image with multiple channels, and then, I will add a customized objective function:

def customized_objective(y_true, y_pred):
        # split the tensor into different channels
        ch1_true = y_true[:, :, :, :1]
        ch2_true = y_true[:, :, :, 1:2]
        ...
        return mse(ch1_true, ch1_pred) + L1(ch2_true, ch2_pred)

At least, this is how I do it for outputs, good luck.

[tinalegre]

@mbenami @oeway thanks for your feedback, I ended up following the solution of presented here and it worked, basically I created a custom function that given the generators of the form (x,y), it returns the output in the form (x, [y,y,y])

[gledsonmelotti]

@jkjung-avt Thank you very much.

[tinalegre]

@oeway @mbenami thank you for your feedback, my previous solution was actually wrong. I followed your suggestion @oeway and I changed my network by using a Lambda function to concatenate the outputs as follows:

    def concatenate_activations(x):
        return K.concatenate(x)

    ....
    outputs = Lambda(concatenate_activations, name='Concatenate_activations')(outputs)
    model = Model(inputs=inputs, outputs=outputs)

without the Lambda it was not working as Keras expected a Tensor as output parameter. When compiling the model, I'm doing: model.compile(loss=multi_loss, optimizer=optimizer, metrics=metrics)

def multi_loss(y_true, y_preds, nb_classes=2, nb_combloss=10, single_weight=1.0, combine_weight=1.0):
    y_combine_preds = y_preds[:, :, 0:(nb_classes*nb_combloss)]
    y_single_preds = y_preds[:, :, (nb_classes*nb_combloss):(nb_classes*nb_combloss+nb_classes)]

    def combine_loss(preds):
        loss = 0
        for ii in range(1, nb_combloss+1):
            ii_curr = (nb_classes*(ii-1))
            pred = preds[:,:, ii_curr:ii_curr+nb_classes]
            loss += categorical_crossentropy(y_true, pred)
        return loss

    def single_loss(preds):
        loss = categorical_crossentropy(y_true, preds)
        return loss

    return single_weight * single_loss(y_single_preds) + combine_weight * combine_loss(y_combine_preds)

where y_preds is of shape [:,65536,22], y_single_preds is of shape [:,65536,2] and y_combine_preds is of shape [:, 65536,20]. It seems to be that I did something wrong, as I'm getting an error of: Invalid argument: Incompatible shapes: [8,65536,2] vs. [8,65536,22], any ideas?

[oeway]

@tinalegre It seems to me that you need to make y_true has the same shape as y_pred, by padding with zeros for example. I guess Keras expect you will have the same shape for y_true and y_pred, so you just need to pass the check of the Keras engine.

[tinalegre]

@oeway the problem is somehow in the metrics = ['accuracy', 'mean_squared_error'] functions, for some reason, for instance, the mean_squared_error that I've only one output. Basically, what I've a binary segmentation problem with binary masks as one_hot_vectors and I'm summing up a set of categorical_crossentropy losses. Any ideas whether I need to modify the mean_squared_error so that it averages the mean_squared_error from all the losses?

[oeway]

@tinalegre I think mean_squared_error expect you to have the same shape for y_pred and y_true. It doesn't make sense to me that why you need mean_squared_error if it's a classification problem. I would simply remove mean_squared_error. Or if you want to somehow have it, you definately need to make your own customized metric function which accepts different shapes for y_pred and y_true.

[XiaoTonyLuo]

@oeway Hi I am confused on the memory allocation scheme of ImageDataGenerator. I write a generator, the structure is similar to @jagiella 's example sulution. But when I usefit_generator to train my network I found that the memory of the generator accumulates as epoch increases and finally stopped with out of memory error, and when I try ImageDataGenerator there is no such problem. Could you please give me some advice on it? Thank you!

[amineio]

For Transforming Images and masks (synchronously) for segmentation for example, take a look on keras documentation (checked recently) 👍
Example of transforming images and masks together

[tinalegre]

@amineio thanks for the info. did you use it? and how would you manage one-hot encoding for the masks?

[amineio]

@tinalegre there is only two classes (black and white masks) the final layer looks like that :
outputs = Conv2D(1, (1, 1), activation='sigmoid') (c9)
You can refer to many Unet implementations for further details, exp : zhixuhao

[bernardohenz]

@oeway
first, I want to say that I've been using your code, and it feels so good the ability to control the order of transformations on the pipeline (and even create new transformations and add them to the pipeline).

Currently, I am using the ImageDataGenerator` as a pre-processing step before prediction, so I must run them even when testing the models accuracy. My question is: when usingevaluate_generatorwithuse_multiprocessing=True```, is it guarantee that each sample will be evaluated only once?

I've been search in the documentation, but the only thing I could find is

The use of keras.utils.Sequence guarantees the ordering and guarantees the single use of every input per epoch when using use_multiprocessing=True.

Does your code account for that?

[oeway]

@bernardohenz thanks for your feedback. I don't have time to look in more details right now. I think probably it's not guaranteed, Keras introduced keras.utils.Sequence to work with multiprocessing(a warning related to that), and my implementation do not use that. I think you will need to investigate on how to change the generator to inherit form keras.utils.Sequence. Maybe you want to look at this tutorial:
https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly.html

[bernardohenz]

hi @oeway

I've checked this link already, but the generator already offers so much features (flow_from_directory, several transformations, pipeline ordering) that would take some time to implement in the keras.utils.Sequence. I believe that _adapting_ the Sequence inside the generator would be faster, but right now I don't know if it is possible. I'll take a look into that.

[bernardohenz]

@oeway I've adapted your code to use the keras.utils.Sequence. It is working both fow flow and flow_from_directory. I created a repo with the Datagen and test files: https://github.com/bernardohenz/ExtendableImageDatagen

[wt-huang]

Closing as this is resolved

[rstml]

Just for the reference, here's another way of adding image manipulation (random & center crop in my case) to the pipeline before Keras resizes images. It's done by monkey patching keras_preprocessing.image.utils.loag_img function as I couldn't find any other way.

https://gist.github.com/rstml/bbd491287efc24133b90d4f7f3663905

[Nestak2]

@stratospark @oeway Are any of the scripts here merged into the main keras? 3 years have gone and default keras still doesn't allow the use of csv/npy ...

[tinalegre]

@oeway in your code you showed us how to sychronize two generator and combine it into one: train_generator = dgdx+dgdy. Now, imagine, I would like to combine 3 generators, i.e. 1x for the input image (dgdx), and 2x outputs: segmentation mask (dgdx) and depth mask (dgdz). I was therefore trying to do something like: train_generator = dgdx+dgdy+dgdz but it didn't work.
=> TypeError: unsupported operand type(s) for +: 'zip' and 'DirectoryIterator'
Any idea, how could I integrate this into my model, i.e. 2 outputs with your generator?

@pengpaiSH I have a working version right now, you are welcome to try it out. https://github.com/oeway/keras/tree/extendImageDataGenerator (branch: extendImageDataGenerator)

For now, you can customize the pipeline and synchronize two ImageDataGenerator by a __add__(+) operator. You can see more extensions in the following example:

from keras.preprocessing.image import ImageDataGenerator,standardize,random_transform
# input generator with standardization on
datagenX = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    featurewise_standardize_axis=(0, 2, 3),
    rotation_range=180,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    fill_mode='reflect',
    seed=0,
    verbose=1)

# output generator with standardization off
datagenY = ImageDataGenerator(
    featurewise_center=False,
    featurewise_std_normalization=False,
    rotation_range=180,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    fill_mode='reflect',
    seed=0)

def center_crop(x, center_crop_size, **kwargs):
    centerw, centerh = x.shape[1]//2, x.shape[2]//2
    halfw, halfh = center_crop_size[0]//2, center_crop_size[1]//2
    return x[:, centerw-halfw:centerw+halfw,centerh-halfh:centerh+halfh]

def random_crop(x, random_crop_size, sync_seed=None, **kwargs):
    np.random.seed(sync_seed)
    w, h = x.shape[1], x.shape[2]
    rangew = (w - random_crop_size[0]) // 2
    rangeh = (h - random_crop_size[1]) // 2
    offsetw = 0 if rangew == 0 else np.random.randint(rangew)
    offseth = 0 if rangeh == 0 else np.random.randint(rangeh)
    return x[:, offsetw:offsetw+random_crop_size[0], offseth:offseth+random_crop_size[1]]

datagenX.config['random_crop_size'] = (800, 800)
datagenY.config['random_crop_size'] = (800, 800)
datagenX.config['center_crop_size'] = (512, 512)
datagenY.config['center_crop_size'] = (360, 360)

# customize the pipeline
datagenX.set_pipeline([random_crop, random_transform, standardize, center_crop])
datagenY.set_pipeline([random_crop, random_transform, center_crop])

# flow from directory is extended to support more format and also you can even use your own reader function
# here is an example of reading image data saved in csv file
# datagenX.flow_from_directory(csvFolder, image_reader=csvReaderGenerator, read_formats={'csv'}, reader_config={'target_size':(572,572),'resolution':20, 'crange':(0,100)}, class_mode=None, batch_size=1)

dgdx= datagenX.flow_from_directory(inputDir, class_mode=None, read_formats={'png'}, batch_size=2)
dgdy= datagenY.flow_from_directory(outputDir,  class_mode=None, read_formats={'png'}, batch_size=2)

# you can now fit a generator as well
datagenX.fit_generator(dgdx, nb_iter=100)

# here we sychronize two generator and combine it into one
train_generator = dgdx+dgdy

model.fit_generator(
        train_generator,
        samples_per_epoch=2000,
        nb_epoch=50,
        validation_data=validation_generator,
        nb_val_samples=800)

[zachwarner]

Hi all. Many thanks to @oeway and @bernardohenz for their very helpful solutions. Just checking back in two years later: is there a similar adaptation of ImageDataGenerator now that the Keras team have refactored this code?

The specific functionality I'm looking for is just precise reproducibility; I'd like ImageDataGenerator to be seedable but do not need the other features. For reference, my workflow is ImageDataGenerator -> flow -> fit_generator and I'm running Keras 2.3.1 with TF 2.0 backend in python 3.6.9 in a docker container with base Ubuntu 18.04.

[prarshah]

@ChristianEschen, I copied my reply on Disqus over here.

You can define your paired_crop_generator() to crop the same region from X (images) and Y (masks) simultaneously. The code below is just quick and dirty hack, but hopefully you get the idea.

def paired_crop_generator(batches, crop_length=224):
    '''
    Assuming batch_X.shape is (N,1440,1920,3) and batch_Y.shape is (N,1440,1920,1),
    this function would crop the same (224,224) region from the XY pair for each image
    in the batch, then return (yield) the cropped batch.
    '''
    while True:
        batch_X, batch_Y = next(batches)
        batch_Xcrops = np.zeros((batch_X.shape[0], crop_length, crop_length, 3))
        batch_Ycrops = np.zeros((batch_X.shape[0], crop_length, crop_length, 1))
        height, width = batch_X.shape[1], batch_X.shape[2]
        dy, dx = random_crop_length, random_crop_length
        for i in range(batch_X.shape[0]):  # loop through all images in the batch
            # select a random top-left corner of the crop, for current image in the batch
            x = np.random.randint(0, width - dx + 1)
            y = np.random.randint(0, height - dy + 1)
            # take the crop (same region) from both X and Y
            batch_Xcrops[i] = batch_X[i][y:(y+dy), x:(x+dx), :]
            batch_Ycrops[i] = batch_Y[i][y:(y+dy), x:(x+dx), :]
        yield (batch_Xcrops, batch_Ycrops)

......
training_generator = zip(dgdx,dgdy)
train_crops = paired_crop_generator(training_generator, 224)

net_final.fit_generator(train_crops, ......)

@jkjung-avt @XiaoTonyLuo
Thankyou, I tried it, but using this method the model.predict method never completes, any suggestions how to tackle it?