Keras: How does model.fit () calculate loss and acc ? Documentation will be helpful.

[whatdhack]

I think it uses functions like binary_entropy , binary_accuracy etc. Is any smoothing used ? Some description in the documentation will be good ( pointers to the exact code will also help).

[raymond-yuan]

Loss and accuracy are calculated as you train, according to the loss and metrics specified in compiling the model. Before you train, you must compile your model to configure the learning process. This allows you to specify the optimizer, loss function, and metrics, which in turn are how the model fit function knows what loss function to use, what metrics to keep track of, etc. The loss function (like binary cross entropy) documentation can be found here and the metrics (like accuracy) documentation can be found here

In summary, in order for you to train (and for fit to calculate loss and acc):

1. Instantiate model

model = Sequential()
...

1. Compile the model. This is where the functions like binary cross entropy are specified for the model to call fit. The built in loss functions are documented and implemented here.

# For a binary classification problem
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

1. Train by calling fit

model.fit(data, one_hot_labels, epochs=10, batch_size=32)

[whatdhack]

Thanks , that is a little bit more information, but not exactly the answer to what I was asking. My question is what exactly is the formula used for calculating loss and accuracy per batch ( e.g. the numbers that get displayed during model.fit() ) . If you know where that is exactly done in the code , that answer should be right there.

[raymond-yuan]

Yes, like I linked, the formulas for calculating loss per batch can be found here. The exact implementation is backend specific. For example if you were using tensorflow look here

[whatdhack]

Thanks again - I have seen those code by the way. Let me try again. The following is an output from a model.fit () session .

5/125 [>.............................] - ETA: 4:35 - loss: 1.1342 - acc: 0.5250

Which exact variable holds the loss (e.g. 1.1342) and accuracy (e.g. 0.5250) ?

[raymond-yuan]

I believe that the loss is stored here; similarly the accuracy is handled by the handle_metrics function. However, you shouldn't be working with these variables directly, if you want to calculate the loss and accuracy for your model given some data use model.evaluate

[whatdhack]

Thanks again. Basically my question is what mathematically the loss and accuracy displayed during model.fit() are. Is it averaged over the batch , or some other function is used to calculate the loss/acc that gets displayed on the screen. The answer is still not there, hence need to keep looking till the answer is found and document it.

[raymond-yuan]

For training loss, keras does a running average over the batches. For validation loss, a conventional average over all the batches in validation data is performed. The training accuracy is the average of the accuracy values for each batch of training data during training.

[whatdhack]

Good. Is the running average over just the batch or there is carryover from previous batch ? Would be great to see the code for that.

[raymond-yuan]

The code is here, BaseLogger is applied to every Keras model and will update things like loss and acc. The training loss is carried over from the previous batch: it's the average of the losses over each batch of training data.

[whatdhack]

You are probably close. If the code ( see below) indeed is responsible for the fit() per batch loss/acc output, then the mystery is probably is how 'v' got into the logs list before entering the on_batch_end() function . This function primarily seems to be for keeping track variables for on_epoch_end () calculations.

def on_batch_end(self, batch, logs=None):

.
.
.
for k, v in logs.items():
if k in self.stateful_metrics:
self.totals[k] = v < ----
else:
if k in self.totals:
self.totals[k] += v * batch_size
else:
self.totals[k] = v * batch_size

[raymond-yuan]

'v' gets added into the batch logs here. outs is the output of the train function, f, which is built when you call fit on your model, which will output your loss and metrics from here given some inputs. An example of the log list looks like this (if you were to add a print(logs) in the on_batch_end method)

{'loss': 1.8399812, 'batch': 2, 'size': 32, 'acc': 0.5625}
{'loss': 1.6450999, 'batch': 3, 'size': 32, 'acc': 0.53125}
{'loss': 1.6244895, 'batch': 4, 'size': 32, 'acc': 0.40625}
{'loss': 1.4333215, 'batch': 5, 'size': 32, 'acc': 0.5625}
{'loss': 1.1506481, 'batch': 6, 'size': 32, 'acc': 0.75}
{'loss': 1.0616728, 'batch': 7, 'size': 32, 'acc': 0.65625}
{'loss': 1.313815, 'batch': 8, 'size': 32, 'acc': 0.46875}
{'loss': 1.2656202, 'batch': 9, 'size': 32, 'acc': 0.5625}
{'loss': 1.2441672, 'batch': 10, 'size': 32, 'acc': 0.5625}
{'loss': 0.99731344, 'batch': 11, 'size': 32, 'acc': 0.65625}
{'loss': 1.0144608, 'batch': 12, 'size': 32, 'acc': 0.71875}
{'loss': 1.2803181, 'batch': 13, 'size': 32, 'acc': 0.4375}
{'loss': 1.2766386, 'batch': 14, 'size': 32, 'acc': 0.5}
{'loss': 0.87459761, 'batch': 15, 'size': 32, 'acc': 0.6875}

[whatdhack]

That is great. So, it is still not clear if the per batch loss/acc is the average over just that batch or something else.

[kaiche12]

Sorry, how do Keras calculate loss (e.g mse) when doing predict_on_batch. For example you have 10 batches, the model normally will return 10 losses

[DayChan]

@raymond-yuan When we choose "acc" to calculate accuracy, how does keras choose the calculating function? I found that there are several accuracy functions.