Keras: what is this warning in the callbacks?

[ghost]

UserWarning: Method on_batch_end() is slow compared to the batch update (0.151899). Check your callbacks.

what is that warning? and why?

Thank you!

• [x] Check that you are up-to-date with the master branch of Keras. You can update with:
  pip install git+git://github.com/fchollet/keras.git --upgrade --no-deps

• [ ] If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• [x] If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with:
  pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• [ ] Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

[bstriner]

Literally what it says, You are running something like saving the model or rendering images after each batch and it is taking longer than the batches themselves.

Are you doing something each batch that could be causing the issue? Or are you just asking how the warning can be disabled?

Cheers

[ghost]

yes i'm doing "reduce_lr" and "early_stopping" callbacks,
actually this message only appear if my learning rate is high (.1)
but if it is .001 it disappears.

[bstriner]

Neither ReduceLROnPlateau nor EarlyStopping have batch end callbacks, only epoch end callbacks, at least in the current version. Does your version of Keras have batch_end callbacks in those 2 classes? Might be time to update. If that isn't the issue, maybe you just have really small batches and epochs so anything is slow compared to your batches.

[xiaohk]

Is it possible to get this warning without using any callbacks?

I am using a DataGenerator to feed data, which has an on_epoch_end method. I am also getting this warning:

UserWarning: Method on_batch_end() is slow compared to the batch update (0.586719). Check your callbacks.

[plesqui]

@xiaohk I am having the same issue. I scripted my own data generator where I read data from an .h5 file and perform data augmentation. I suspect that the GPU processes the batch through the network faster than the time it takes to generate the next batch. I will keep you updated if I find out how to solve it...

Cheers!

[plesqui]

Hi @xiaohk.

It seems like the issue was caused by my CPU that generates the training batch (consisting of 8 training examples) much slower than the GPU processes it. I have increased the number of training examples per batch to 32 and the warning disappears.

Another approach that might solve this problem is to set use_mutiprocessing = TRUE in the fit_generator and select the appropriate number of workers to generate your training batches more efficiently. More information about this is described in: [https://keras.io/models/sequential/]
Hope it helps!

[Demetrio92]

Somehow this thread did not really help me understanding what's wrong, so here's my attempt to explain it.

---

The warning means exactly what it's saying (duh!)

Method on_batch_end() is slow compared to the batch update (factor_of_slowness)

So if it says compared to the batch update (2.059934) it means your validation (or whatever you're doing there) takes twice the time of training (which given a small batch size and a serious dev set is a realistic scenario).
Apparently it warns you if the factor is above 0.5, and does not warn otherwise.

[msloryg]

@Demetrio92 the threshold seems to be different for Method (on_train_batch_begin):

WARNING: tensorflow:Method (on_train_batch_begin) is slow compared to the batch update (0.121552).
Check your callbacks.

Does somebody know what the difference in the warning means?

[allen-ash]

@ghost Here is my self-define weightsSaver, only every self.N iterations tf will show this warning.

class WeightsSaver(Callback):
    def __init__(self, N):
        self.N = N
        self.batch = 0

    def on_batch_end(self, batch, logs={}):
        if self.batch % self.N == 0:
            name = '.\\model\\synDS_weights%09d.h5' % self.batch
            self.model.save_weights(name)
        self.batch += 1

Does yours show every iteration or some iterations once?

[Demetrio92]

This discussion is going waaay off-topic, guys. Consider StackOverflow for specific problems and solutions.

---

Otherwise, you pollute the thread and anyone trying to figure out what does the warning in the OP mean will be reading through the code snippets on how to save weights on every N iterations.
Which is discussed here thoroughly, including the built-in CheckPoint option.

[gledsonmelotti]

Neither ReduceLROnPlateau nor EarlyStopping have batch end callbacks, only epoch end callbacks, at least in the current version. Does your version of Keras have batch_end callbacks in those 2 classes? Might be time to update. If that isn't the issue, maybe you just have really small batches and epochs so anything is slow compared to your batches.

@bstriner How do you eliminate this "UserWarning: Method on_batch_end() is slow compared to the batch update (0.151899). Check your callbacks"?

Best Regards

[left-brain]

Any solution for this warning message?

[aryanVijaywargia]

@left-brain Increasing batch_size did the trick in my case

[Demetrio92]

Any solution for this warning message?

reading this thread is the solution.

[dsvanidze]

In my case it was the Tensorboard callback https://github.com/tensorflow/tensorflow/issues/37876#issuecomment-674450985.

[errabundo]

WARNING:tensorflow:Callbacks method on_train_batch_end is slow compared to the batch time (batch time: 0.0032s vs on_train_batch_end time: 0.1142s)

[Demetrio92]

the level of idiocy in this thread is unbelievable.

[KeithWM]

In my case, increasing the batch size resulted in a memory error. Fortunately, the kwarg experimental_steps_per_execution can be supplied in the model.compile() call, reducing how often on_batch_end callbacks are evaluated. This removed the warning, and should be reducing my training time by ~50%.

[Anhar2]

In my case, increasing the batch size resulted in a memory error. Fortunately, the kwarg experimental_steps_per_execution can be supplied in the model.compile() call, reducing how often on_batch_end callbacks are evaluated. This removed the warning, and should be reducing my training time by ~50%.

May I ask to what did you set it? thought it should be a number representing the number of steps per execution, but didn't work and cannot find references.