Keras: Different results from fit_generator and fit

[darugar]

Keras 1.0.0 with Theano 8.1.0 on linux, Titan X GPU:

I get different convergence characteristics from fit vs fit_generator - fit converges to a reasonable solution, but fit_generator does not.

Here's the code: https://gist.github.com/darugar/b83c70284acce912542431f6999578ae

I have a model with 5 convolution layers, 3 dense (I see the same issue with other models).

When I load all of the training data into memory and train the network with fit I get a decrease in loss, increase in accuracy.

When I load the same model with the same data, but this time via a data generator and fit_generator, I get increasing loss, poor accuracy.

I've confirmed that the data returned by the data generator is the same as the data in memory by collating the generated data and comparing with the in-memory.

I'm using a thread based model to load data into the generator in the background, but otherwise nothing exotic. I use only a single thread for reader, so hopefully no strange thread interaction issues.

My data is 9x64x64, 70k training samples, 30k validation.

Loss graph for fit (in-memory data). Blue is training loss, green is validation:

Loss graph for fit_generator:

[darugar]

It looks like the difference in behavior was due to the lack of data shuffling in my data generator - I tried fit with shuffle=False and got poor results, and then I integrated data shuffling into my generator and I started seeing good results with fit_generator. Moral of the story: shuffle your data properly!

[pengpaiSH]

@darugar I have run into the same problem. Note that fit_generator() does not have the shuffle parameter. Do you mean that you are shuffling the data inside your customized generator?

[ZhuoyangZhan]

I think he did

[AloshkaD]

I've integrated a random generator to fix this issue like this
`class Generators:
def generate_train_batch(self, data,img_rows, img_cols, batch_size = 16):

    batch_images = np.zeros((batch_size, img_rows, img_cols, 3))
    batch_masks = np.zeros((batch_size, img_rows, img_cols, 1))
    while 1:
        for i_batch in range(batch_size):
            i_line = np.random.randint(len(data)-1)

            img,parcel_id, binary_img, orginal_size, area, image_corner, coordinates_boundary= main(data, i_line , size=(img_rows,img_cols),extract_parcel= True,augumentation=False,
                                  balance_classes=False, display_gt = False, display_mask = False, 
                                  display_binary_img=False)
            #img_rows, img_cols =  img.shape[:2]

            batch_images[i_batch] = img
            batch_masks[i_batch] = binary_img
        yield batch_images, batch_masks`

[Cerno-b]

It looks like the difference in behavior was due to the lack of data shuffling in my data generator - I tried fit with shuffle=False and got poor results, and then I integrated data shuffling into my generator and I started seeing good results with fit_generator. Moral of the story: shuffle your data properly!

This a thousand times! I had the exact same issue and switching on shuffle resolved it perfectly, thanks a lot for your insight!