Keras: How can I get both test accuracy and validation accuracy for each epoch

[philokey]

Hi, everyone.

I can use model.evaluate() to calculate the test accuracy for the last epoch, but how can I get both test accuracy and validation for each epoch?

[joelthchao]

You can train the model with only one epoch and evaluate it. Then use a for loop to repeat this procedure for several epochs. However, doing test in each epoch doesn't like what we should do when training a model, test should be only done after you finished training.

[joelthchao]

Well, I wrote a callback for this purpose.

class TestCallback(Callback):
    def __init__(self, test_data):
        self.test_data = test_data

    def on_epoch_end(self, epoch, logs={}):
        x, y = self.test_data
        loss, acc = self.model.evaluate(x, y, verbose=0)
        print('\nTesting loss: {}, acc: {}\n'.format(loss, acc))

Then, you can call fit with callbacks

model.fit(X_train, Y_train, validation_data=(X_val, Y_val), 
          callbacks=[TestCallback((X_test, Y_test))])

However, since there is no on_evaluate_end in the Callback class. So actually it does testing before evaluation and results in bad output format Q_Q.

Epoch 1/2
 9984/10000 [============================>.] - ETA: 0s - loss: 0.6496 - acc: 0.7921
Testing loss: 0.2133, acc: 0.9319
10000/10000 [==============================] - 8s - loss: 0.6491 - acc: 0.7922 - val_loss: 0.2133 - val_acc: 0.9319

[carlthome]

Having an on_evaluate_end would be great and would totally work for me on this one https://github.com/fchollet/keras/issues/2521

[zbyte64]

evaluate now only returns loss and not accuracy, rendering the above work around broken :(

Edit: but passing in metrics=['accuracy'] during compile works fine for me

[patyork]

evaluate() will return the list of metrics that the model was compiled with. So if your compile metrics list include 'accuracy', then this should still work.

E.g.

tmp = Sequential()
tmp.add(Dense(20, input_shape=(10,)))

tmp.compile(optimizer='adadelta', loss='mse', metrics=['mse', 'accuracy'])
tmp.evaluate(np.asarray([np.zeros((10))]), np.asarray([np.zeros((20))]))

yields

1/1 [==============================] - 0s
[0.0, 0.0, 1.0]

..which is [loss, mse, accuracy] as a list.

[piotrbazan]

model.fit returns history object which contains information about training an validation accuracy and loss:

{'acc': [0.9843952109499714],
 'loss': [0.050826362343496051],
 'val_acc': [0.98403786838658314],
 'val_loss': [0.0502210383056177]
}

[Alhawawreh]

can anyone help me, pleae? the evaluation function in keras has a very bad performance although the result of training and validation is very good. what is the problem ?

[Hisairnessag3]

@manallllll This is a question better suited for stackoverflow but you should always post code if you want help

[nobeldang]

@joelthchao i defined my custom callback as per your code but its showing an error that too many values to unpack at line
self.model.evaluate(...)

[samra-irshad]

print('\nTesting loss: {}, acc: {}\n'.format(loss, acc))

Your method works, however, Keras is not logging test results in log file, its doing that for validation set but not for test set

[KhawYewOnn]

Epoch 1/2
9984/10000 [============================>.] - ETA: 0s - loss: 0.6496 - acc: 0.7921
Testing loss: 0.2133, acc: 0.9319
10000/10000 [==============================] - 8s - loss: 0.6491 - acc: 0.7922 - val_loss: 0.2133 - val_acc: 0.9319

@joelthchao is 0.9319 the testing accuracy or the validation accuracy? Notice that acc:0.9319 is exactly the same as val_acc: 0.9319. Also, Testing loss: 0.2133 is the exact same value as val_loss: 0.2133. I ran the code as well, and I notice that it always print the same value as validation accuracy. I notice that somehow self.model.evaluate(x, y) is not using the value in x and y, but instead uses the validation data.