Keras: model.evaluate with 'rmse' returns incorrect scores?

[emjotde]

Hi,
I have the following weird situation: I cannot recreate the way keras calculates RMSE on a test set. It seems to return too good results (RMSE is too low). We seems worrying, since this is also my objective during training.

model.load_weights("simple.3.h5")
model.compile(loss='rmse', optimizer='adagrad')

# Calculate test set score with Keras
score = model.evaluate(X_test, y_test, batch_size=batch_size)

# Calculate test set score manually after prediction
predict = model.predict(X_test, batch_size=batch_size)
score2 = np.sqrt(np.mean(np.square(predict - y_test)))

print('Test score:', score, score2)

And I get the following output:

Test score: 17.0371643621 24.382910713

Is there any explanation why those two scores would be different that I am missing? There is no modification of X_test, y_test between the evaluate, predict calls.
Thanks,
Marcin

[emjotde]

Or could this actually be the average RMSE over batches as during training?

[fchollet]

It's not the same mathematical formula. What Keras computes is:

np.sum([np.sqrt(np.mean(np.square(predict_batch - y_batch)))] for predict_batch, y_batch in batches]) / nb_samples

i.e. Keras computes the loss batch by batch. To avoid inconsistencies I recommend using MSE instead.