Dash: [BUG] Callback doesn't update page layout if same callback is fired again before previous callback completes

[adm78]

Describe your context

dash                 1.3.1        
dash-core-components 1.2.1        
dash-html-components 1.0.1        
dash-renderer        1.1.0        
dash-table           4.3.0

 - OS: Linux (Debian 10)
 - Browser: Firefox
 - Version 60.9.0esr (64-bit)

Describe the bug

I have a callback who's input is the "n_intervals" attribute of a dcc.Interval component. If the callback function takes longer to return than the length of the interval driving it, then page layout is never changed by this callback. The updating of page layout silently fails. My current temporary fix is to make the intervals for these callbacks much larger than I would like, making the application sluggish.

Expected behavior

The page layout should be updated with the latest return value available, even if the callback has fired again and is awaiting a new return value.

[byronz]

@adm78 does the following code illustrate your problem?

import time
import dash
import dash_html_components as html
import dash_core_components as dcc
from dash.dependencies import Input, Output


app = dash.Dash(__name__)
app.layout = html.Div(
    [

        html.Div(id="out"),
        dcc.Interval(id='freq', interval=1*1000, n_intervals=0)
    ]
)


@app.callback(Output("out", "children"), [Input("freq", "n_intervals")])
def cb(intervals):
    time.sleep(2)
    return " CB => Out {}".format(intervals)

if __name__ == "__main__":
    app.run_server(debug=True)

[adm78]

@byronz yes, exactly. No page update with interval=1*1000. With interval=1*3000 everything works as expected.

[byronz]

@adm78 the behavior of the renderer, in this case, is by design to avoid rerendering if a callback has been triggered multiple times by a cascade of other changes… it's waiting for the last one but just keep on pending with new requests. you can see that from network tab too.

there is some internal callback relevant discussion going on, and this will be addressed together whenever we end up with a final conclusion.

[Jerry-Ma]

@adm78 @byronz

I recently discovered that I can use the following trick to "throttle" the interval so that it is only fired after the previous function is returned:

        # in layout, create dcc.Interval(id='timer', interval=1000, max_intervals=0)
        @app.callback(
                [
                    Output('output_component', 'children'),
                    Output('timer', 'max_intervals'),
                    Output('timer', 'interval'),
                    ],
                [
                    Input('timer', 'n_intervals'),
                    ]
                )
        def update(n_intervals):
            # if you can calculate the time it takes to execute the function below,
            # it can be used to update the interval so that the callback fires at a roughly regular frequency
            result, time_used = call_and_timeit(long_running_function)()
            new_interval = 1000 - time_used    # the makes the callback fire at rate of 1 per second.
            if new_interval < 0:
                new_interval = 100  # some minimum time
             return result, (n_intervals or 0) + 1, new_interval

Of course, I am very looking forward to being able to have a server side push mechanism using websocket. I've seen a great example here #1110 . Unfortunately, the author does not release the source code.

Hopefully this is helpful.

[MM-Lehmann]

@Jerry-Ma I guess you want to post your intermediate results from your long_running_function? The way you've implemented this above would actually introduce a lag of up to one second between each run of the function, not accounting for the time it needs to render. Also, there is a lot of data going around the network.
Why not run the heavy bit async and post the results to a caching service (I use memcached) which is polled every interval by your update function? It just takes whatever has accumulated at that point without interfering with the actual (now non-throttled) computation. Note: you might have to have multiple workers running to make this work.

[Jerry-Ma]

@MM-Lehmann Thanks for your comment. I am not sure I fully understand it, though:

I guess you want to post your intermediate results from your long_running_function? The way you've implemented this above would actually introduce a lag of up to one second between each run of the function, not accounting for the time it needs to render.

The name of the function is causing the confusion. I should have "named" it as func_that runs_at_non_constant_time_that_may_be_larger_than_timer_interval. For example, it can be a function that returns from a remote db query which may take a range of time to finish, and we would like to poll the resource as frequent as possible. dcc.Interval does not work because it stuck when it occurs that the func execution is slower than the predefined interval.

The work around is mimicking a recursive calling and does the job at least to the degree that I have tested.

Also, there is a lot of data going around the network.

I am not sure how this introduces a lot of data. The approach only passes two additional ints from the server to the browser.

Why not run the heavy bit async and post the results to a caching service (I use memcached) which is polled every interval by your update function? It just takes whatever has accumulated at that point without interfering with the actual (now non-throttled) computation. Note: you might have to have multiple workers running to make this work.

This is what I have been doing with actual real long_running_functions, but as I said in the above, the naming is confusing and actually this posted workaround is to solve the need to invoke a function repeatedly while the execution time could vary a lot.

With that being said, there is another use case of this recursive calling pattern, that is when the frontend rendering time dominates the refresh rate. This could happen for making live plots that contains variable (user could set this) amount of data.

For example, it may take 1 sec to render 10k points in a scatter plot, and 2 sec to render 20k points, but the number of points to be rendered are dynamic. To set up a live updating figure of this kind, it would make sense to use the recursive update (rendering) approach.