Keras: Can I use callbacks in train_on_batch ?

[pengpaiSH]

@fchollet As we all know, Keras provide a callbacks interface in model.fit() function. However, it requires loading the training dataset into memory previously. When I use train_on_batch( ), I didn't find out where I can put my callbacks (e.g. EarlyStopping, History, etc.). Any suggestions ?

[fchollet]

Clearly if you are training on each batch from inside your own script, then the notion of per-batch or per-epoch callback is not longer relevant. You are free to do whatever you want between batches or between epochs: it's your own code.

[pengpaiSH]

@fchollet Thank you for you reply. fchollet, you are always that helpful : )

[naotokui]

what about TensorBoard callback? Are there any way to use TensorBoard with train_on_batch()?

[teramototoya]

@naotokui Please see https://gist.github.com/joelthchao/ef6caa586b647c3c032a4f84d52e3a11

[hellojialee]

@naotokui
Thank you for your sharing.
But
if batch_no % 10 == 0: X_val, Y_val = np.random.rand(32, 3), np.random.rand(32, 1) logs = model.train_on_batch(X_val, Y_val) write_log(callback, val_names, logs, batch_no//10)
How can we write the loss on the validation data while don't train the model on the validation data?

[teramototoya]

@USTClj I think that using test_on_batch will work

[hellojialee]

@teramototoya Thank you so much, I have just forgotten the test_on_batch function. :)

[keyochali]

you can do it manually!
you can just use save weights:
model.save_weights('my_model_weights.h5')
and for tensorboard you can use:
tensorboard.on_epoch_end() tensorboard.on_train_end(None)

[sravanje]

you can do it manually!
you can just use save weights:
model.save_weights('my_model_weights.h5')

This will not save the optimizer state though, right? What's the workaround for that?

[moha23]

@fchollet I'm using model.train_on_batch and after each epoch I validate using model.predict (also in batches), due to memory leak issues I need to use clear_session() and load_model() after and before each model.predict, respectively. Since I need to continue training after validation I use model.save() to save the weights, since it also saves the 'the optimizer and its states'. Is it instead possible to save only the weights but not the architecture (I want to make some changes in architecture for validation) but also be able to pick up training after validation?