Keras: ProgBar has poor performance in jupyter notebook

[clouds56]

When I use model.fit() in jupyter notebook (with verbose) the chrome which host jupyter client frozen. the output shows:

Epoch 1/10
5000/5000 [==============================] - 1s - loss: 0.3613 - acc: 0.9016     
Epoch 2/10
5000/5000 [==============================] - 1s - loss: 0.3593 - acc: 0.9026     
Epoch 3/10
5000/5000 [========================>.....] - ETA: 0s - loss: 0.3447 - acc: 0.9059

I dump it with xxd and find there's lots of '\x08' at the end of line.

00000000: 4570 6f63 6820 312f 3130 0a35 3030 302f  Epoch 1/10.5000/
00000010: 3530 3030 205b 3d3d 3d3d 3d3d 3d3d 3d3d  5000 [==========
00000020: 3d3d 3d3d 3d3d 3d3d 3d3d 3d3d 3d3d 3d3d  ================
00000030: 3d3d 3d3d 5d20 2d20 3173 202d 206c 6f73  ====] - 1s - los
00000040: 733a 2030 2e33 3631 3320 2d20 6163 633a  s: 0.3613 - acc:
00000050: 2030 2e39 3031 3620 2020 2020 0808 0808   0.9016     ....
00000060: 0808 0808 0808 0808 0808 0808 0808 0808  ................
00000070: 0808 0808 0808 0808 0808 0808 0808 0808  ................
00000080: 0808 0808 0808 0808 0808 0808 0808 0808  ................
00000090: 0808 0808 0808 0808 0808 0808 0808 0808  ................
000000a0: 0808 0808 0808 0808 0808 0808 0808 0808  ................
000000b0: 0808 0808 0808 0808 0808 0808 0808 0808  ................
000000c0: 0808 0808 0808 0808 0808 0808 0808 0808  ................
000000d0: 0808 0808 0808 0808 0808 0808 0808 0808  ................
000000e0: 0808 0808 0808 0808 0808 0808 0808 0808  ................
000000f0: 0808 0808 0808 0808 0808 0808 0808 0808  ................
00000100: 0808 0808 0808 0808 0808 0808 0808 0808  ................
00000110: 0808 0808 0808 0808 0808 0808 0808 0808  ................
00000120: 0808 0808 0808 0808 0808 0808 0808 0808  ................
00000130: 0808 0808 0808 0808 0808 0808 0808 0808  ................
00000140: 0808 0808 0808 0808 0808 0808 0808 0808  ................

I think we could use ipywidgets to improve the performance.
Here's an example
The code snippets I'm using is:

from keras.layers import Dense, Activation
from keras.models import Sequential

model = Sequential()
model.add(Dense(output_dim=1024, input_dim=image_size*image_size))
model.add(Activation("relu"))
model.add(Dense(output_dim=10))
model.add(Activation("softmax"))

model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])

model.fit(train_dataset[:5000], train_labels[:5000], nb_epoch=10, batch_size=128)

Any ideas?

[bstriner]

TQDM is also a good option (https://github.com/tqdm/tqdm).

I like the idea of additional progress bars but that would introduce additional dependencies. Should probably put that together as a separate module.

I'll be interested in what you come up with. I'll post a TQDM/Keras example when I get a chance.

Cheers,
Ben

[clouds56]

I agree with tqdm (which is finally using ipywidgets), and I've created a pull request that show a demo of ipywidgets, it's just a demo and I'm not that familiar with the source code of keras.
Consider start from the pull request and migrate to something like tqdm.

[bstriner]

I just put together a module for keras/tqdm integration. Try this out:

https://github.com/bstriner/keras-tqdm/blob/master/examples/keras_progress_bars.ipynb
https://github.com/bstriner/keras-tqdm

After some fiddling, the TQDM display looks really good

Please let me know if it works for you. I'd be interested in how the performance is in comparison.

To use it, set verbose=0 to stop the built-in keras progress bar, and set callbacks=[TQDMCallback()] to use the TQDM progress bar.

Cheers,
Ben

[tentrillion]

I ran into the same issue and tried the solution from @bstriner . It works great and I would like to see something like this come into keras at some point. The only hiccup for me was that the install script didn't work as I'd guessed. I had to do a sys.path.insert(1, '<path_to_repo\keras-tqdm\>' to get things working.

[bstriner]

Glad to help! @tentrillion I just made some updates to the setup and it should be working correctly if you pull the latest changes. If it isn't please let me know what OS/python you are using so I can try to troubleshoot.

Cheers and happy new year everybody!

[clouds56]

It's great!
Although model.evalute doesn't support callbacks and has to use legacy progress bar.

Happy new year!

[bstriner]

Just released on PyPI with some minor tweaks. Should be able to just pip install keras-tqdm to use it.

@clouds56 Great point on evaluate. Seems like evaluate should support callbacks and wouldn't be too hard to implement. Maybe start a new issue and PR to add callbacks to evaluate.

Basically just some changes to training.py to copy some of the callback code from the training loop to the test loop. API for training loop callbacks should be the same as the testing loop callbacks, except maybe that there is only one epoch.

https://github.com/fchollet/keras/blob/master/keras/engine/training.py

Cheers,
Ben

[pkozela]

I hit the similar problem once I moved to python 3.6. Somehow, in notebooks, the Progbar started printing those '\x08' at the end of line - making the notebook freeze (it works OK though outside of jupyter notebooks).

Looks like it's coming from the line:
sys.stdout.write('\b' * prev_total_width)

[wiso]

Similar problems, here what I see in jupyter 4.2.1, python 2.7.13:

416/600 [===================>..........] - ETA: 0s
Out[22]:

[AlxVoropaev]

I have the same problem! Python 3.4.3 keras 1.2.2

[pcampr]

Same in Python 3.6, Keras 1.2.2. The characters were rendered as ticks (Chrome)...
(as a workaround I use verbose=2, which shows only progress at the end of each epoch)

[EvenOldridge]

I was able to get around this by using callbacks to record the information I was interested in and setting verbose=0. Prior to doing so chrome would become unusable as soon as I launched fit_generator, but now it's fully responsive.

In my case I just wanted on epoch end stats so I used:

class LossHistory(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.loss = []
        self.acc = []
        self.val_loss = []
        self.val_acc = []


   def on_epoch_end(self, batch, logs={}):
        self.loss.append(logs.get('loss'))
        self.acc.append(logs.get('acc'))
        self.val_loss.append(logs.get('val_loss'))
        self.val_acc.append(logs.get('val_acc'))

history = LossHistory()

and then called:

model.fit_generator(batches, samples_per_epoch=batches.nb_sample, nb_epoch=1,
                validation_data=val_batches, nb_val_samples=val_batches.nb_sample, verbose=0, callbacks=[history])

hope this helps.

[clouds56]

Hi all, @bstriner have implement a progress bar using tqdm, please checkout https://github.com/bstriner/keras-tqdm.

#!/usr/bin/bash
pip install keras-tqdm

#!/usr/bin/python3
from keras_tqdm import TQDMNotebookCallback
# keras, model definition...
model.fit(X_train, Y_train, verbose=0, callbacks=[TQDMNotebookCallback()])

It is a fancy package, worth a try!

[stolsvik]

The TQDM integration of @bstriner is _awesome!_ (once I got past a boatload of problems, in particular _"Widget Javascript not detected. It may not be installed properly."_ and _"The installed widget Javascript is the wrong version. It must satisfy the semver range ~2.1.4."_..!)

(Only nitpick is that it would be good if one could use the exact same code for Notebook and command-line usage - could it sniff out whether it was inside a notebook environment or not?)

The problem with Keras' progress notebook integration should definitely be addressed, as this was my first excursion into Keras at all, and I really started wondering what was wrong. Finally got that MNIST example to run using command line, and then realized it had to be something with the Jupyter integration.

That such an "in your face" functionality doesn't work out of the box was painful.

Wasn't it for the evident problems with nbextensions, @bstriner's code should definitely be the default for Keras "verbose=1" when run in a Jupyter environment - it is really slick!

[bstriner]

@stolsvik Thanks! If you let me know the details of what issues you ran into I'll add something to the readme.

[deltaz0]

In case anyone still runs into this issue, the new jupyter kernel doesn't have this problem. I was able to fix with:

conda upgrade notebook