Keras: Using the model parameters which gave best validation performance

[shreyassaxena]

Hello,
I have a small issue/suggestion. While training a deep learning network, it would be quite helpful if:

[1] As of now lets say I run for 20 epochs, but on seeing the validation error I know my model achieved the best performance at 10 epochs. Ideally I would re-run with full data, uptill 10 epochs. However, it might be good if the best validation model is saved in memory and one can have an option, either to predict with the model at last epoch or the one with best validation error?
I saw this as a feature somewhere, but I think this is not yet supported in Keras?

Ofcourse, once could run for 5 epochs save the model and the validation error rate, and run again with new initialization for next 5 epochs and so on. I am going to try this option, but would have been great if using pre-trained model was given as an example?

[2] In the output, it would also return the data logged for validation and training error. This would be useful to compare diff. architectures by plotting them

[wepe]

for [1] , my method is:

best_accuracy = 0.0
for e in range(nb_epoch):
    print('Epoch', e)
    print("Training...")
    progbar = generic_utils.Progbar(X_train.shape[0])
    for i in range(nb_batch):
        train_loss,train_accuracy = model.train(X_train[i*batch_size:(i+1)*batch_size], Y_train[i*batch_size:(i+1)*batch_size],accuracy=True)
        progbar.add(batch_size, values=[("train loss", train_loss),("train accuracy:", train_accuracy)] )

    #save the model of best val-accuracy
    print("Validation...")
    val_loss,val_accuracy = model.evaluate(X_val, Y_val, batch_size=1,show_accuracy=True)
    if best_accuracy<val_accuracy:
        best_accuracy = val_accuracy
        cPickle.dump(model,open("./model.pkl","wb"))

by this method , the best model is saved.
hope it help!

[shreyassaxena]

Yeah perfect! I was almost done with it, was finishing up. Still your code looks in a better shape :).

Thanks again, maybe this should be added as a default pipeline in Keras @fchollet ? i.e. saving the best validation model separately ?

[fchollet]

There are currently no plans to make early stopping a Keras feature, since it's fairly easy for the user to implement it (as pointed out by @wepe) and the range of potential behaviors is large enough that standardization might be harmful to model performance...

Regarding [2], this is already the case. The .fit() method will return a training history (dictionary) with all successive training and validation losses / accuracy.

[shreyassaxena]

Ok, thanks I was not knowing about [2]. I will close this issue.

[philipperemy]

If you don't want to pickle your model

    W = m.get_weights()

    val_loss_list = []
    np.random.seed(1337)
    for ep in range(1, nb_epoch):
        m.fit(X_train,
              y_train,
              batch_size=batch_size,
              nb_epoch=1, verbose=0,
              validation_data=(X_test, y_test))

        val_loss_list.append(m.evaluate(X_test,
                                        y_test, verbose=0,
                                        batch_size=batch_size))

    best_epoch = np.array(val_loss_list).argmin()

    # reset model
    m.reset_states()
    m.set_weights(W)

    np.random.seed(1337)
    m.fit(X_train,
          y_train,
          batch_size=batch_size,
          nb_epoch=best_epoch+1, verbose=2,
          validation_data=(X_test, y_test))

[Queequeg92]

you can just rewrite the callback class and save the best model parameters in on_batch_end() function.