Ipywidgets: Provide example of widget executing nonblocking code

[aggFTW]

Summary

Add something like this example to the documentation:

import threading
from IPython.display import display
import ipywidgets as widgets
import time
progress = widgets.FloatProgress(value=0.0, min=0.0, max=1.0)

def work(progress):
    total = 100
    for i in range(total):
        time.sleep(0.2)
        progress.value = (i+1)/total

thread = threading.Thread(target=work, args=(progress,))
display(progress)
thread.start()

---

Original Issue

From what we understand, ipywidgets is a serverextension + nbextension + custom messages that use the shell channel, which blocks on kernel executing code. Is it possible at all, to create a widget that is constantly refreshing its output regardless of whether the kernel's shell channel is busy or not?

We'd be willing to contribute changes to ipywidgets if you can devise a way to make this possible.

Thanks!

[jasongrout]

What about launching a new thread and communicating updates from the thread?

[aggFTW]

Mmmm... can you point me to the code that is updating interact widgets, for example? How is the widget sending updates from the thread?

I'm looking for the code myself :)

[maartenbreddels]

This example works:
https://gist.github.com/maartenbreddels/3378e8257bf0ee18cfcbdacce6e6a77e
It also works when you set thread_safe to False, but I know from experience that it will fail sometimes.

[aggFTW]

😮 thanks! I'm playing with it now 👍 it seems like it will work perfectly!

[SylvainCorlay]

I think that it is worth creating an example notebook for something like this (and a progress bar) ...

[jasongrout]

+1 on an example notebook. That would be a great addition!

[maartenbreddels]

Can we actually get a confirmation that updating a widget from a different thread can lead to issues, or should this maybe be handled in the ipywidgets code, if not already.

[aggFTW]

I've been playing with the code a little bit and I see that by using the IOLoop with the callback, the work method is actually executed in MainThread (even when launched with threading.Thread), which means that if the MainThread ever sleeps, the widget will also not be updated.

I thought that not using the IOLoop would prevent it from being blocked by a sleep in the MainThread. However, sleeping still prevents the refresh. I believe that's because the IOLoop is the one sending the messages, no matter what. If that is true, it seems like the IPython kernel actually prevents widgets from being updated even if running on their own thread.

Any thoughts on how to achieve a refresh regardless of what's happening on the MainThread or that the kernel is busy?

[jasongrout]

CCing @minrk for details about how messages and threads work. IIRC, messages are only sent from the main thread.

[minrk]

It is no longer true that messages are only sent from the main thread (IOPub generally goes on a dedicated background thread, as of ipykernel 4.3), but requests are only received on the main thread. So you should be able to publish safely from a background thread. However, when I made that patch for 4.3, I was only looking at print/display, and missed the comms threadsafety bit, so widgets still don't work in background threads, even though they should. ipython/ipykernel#149 fixes this, and allows widgets to update from background threads.

[jasongrout]

For completeness, here is the example @maartenbreddels posted above:

import threading
from IPython.display import display
import ipywidgets as widgets
import time
def get_ioloop():
    import IPython, zmq
    ipython = IPython.get_ipython()
    if ipython and hasattr(ipython, 'kernel'):
        return zmq.eventloop.ioloop.IOLoop.instance()
ioloop = get_ioloop()
thread_safe = True
def work():
    for i in range(10):
        def update_progress(i=i):
            print "calling from thread", threading.currentThread()
            progress.value = (i+1)/10.
        print i
        time.sleep(0.5)
        if thread_safe:
            get_ioloop().add_callback(update_progress)
        else:
            update_progress()
print "we are in thread", threading.currentThread()

thread = threading.Thread(target=work)
progress = widgets.FloatProgress(value=0.0, min=0.0, max=1.0, step=0.01)
display(progress)
thread.start()

[jasongrout]

Note that with threads working, you should be able to delete the ioloop stuff in the example. Here is a cleaned-up example. Also note that because printing is sent to the current cell, you don't want to print inside the thread.

import threading
from IPython.display import display
import ipywidgets as widgets
import time
progress = widgets.FloatProgress(value=0.0, min=0.0, max=1.0)

def work(progress):
    total = 100
    for i in range(total):
        time.sleep(0.2)
        progress.value = (i+1)/total

thread = threading.Thread(target=work, args=(progress,))
display(progress)
thread.start()

[jasongrout]

CC @cpcloud @jreback