Keras: modify generator inside fit_generator during training

[raulsoutelo]

Hi,

model.fit_generator takes a generator as input. This generator is used to feed the batches to train the model.

Is it possible to modify this generator during training? I am aware that some training parameters can be modified using callbacks, but it seems that the generator is not possible, am I right?

I imagine that I could create my own fit_generator but I was wondering if there is an easier way.

Thanks in advance!

[Dref360]

If you use a keras.utils.Sequence, you can implement on_epoch_end and modify the object between the epoch.

[raulsoutelo]

Thanks for response Dref360, it seems to be working properly! Is it possible to access the model you are training from on_epoch_end?

I tried to read it from disk (saved using ModelCheckpoint callback after each epoch) but there are issues with both functions trying to access the file at the same time.

Thanks!

[Dref360]

Unfortunately, the Sequence's on_epoch_end call happens in another thread so it won't be thread-safe to load the model. You could re-implement ModelCheckpoint to make it thread-safe with your Sequence object, but it would be complicated.

The Sequence API was not designed to interact with the training loop.