Altair: Interactive chart slow with large number of data points

[kuchenrolle]

When I create an interactive scatter plot where some text element is displayed on mouseover, I've noticed that this becomes unusably slow if the data points go into the thousands. If I take the example from here, then all is nice on the 400 data points from the cars package, but if I increase the number of data points to 4000, then things get very slow, and with 20000 (the size of my data set), it becomes much worse:

cars = alt.load_dataset("cars")
cars = pd.concat([cars]*10)
cars["Horsepower"] = cars["Horsepower"].sample(frac=1).reset_index()

pointer = alt.selection_single(on='mouseover', nearest=True, empty='none')

base = alt.Chart().encode(
    x='Miles_per_Gallon', y='Horsepower'
)

chart = alt.layer(
    base.mark_point().properties(selection=pointer).encode(color='Origin'),
    base.mark_text(dx=8, dy=3, align='left').encode(
    text=alt.condition(pointer, 'Name', alt.value(''))
    ),
    data=cars
)

chart

(It does not necessarily make all that much sense to have 20k data points that can be hovered over for more information, but I'm wondering whether there's a way to speed this up or if it will generally be a slow.)

[jakevdp]

This has to do with the vega-lite javascript libraries. @kanitw or @domoritz may have ideas.

[domoritz]

Often SVG performs better for interactive charts because browsers are optimized to track changes. We should allow Altair to change the renderer. Other than that, we are limited by Vega's performance, which is mostly dependent on the browser.

[jakevdp]

SVG vs Canvas is controlled by the renderer metadata... I'm going to be chatting with @ellisonbg today about how we can best give users access to that.

[domoritz]

Excellent. Can you loop me into the conversation? I'm going to work on some Vega/python related things this week.

[aldanor]

Wondering re: this as well, is it posible to siwtch between canvas/svg renderers from Altair itself?

[aldanor]

Just to add, in some browsers and platforms, vega-rendered SVG looks a ton nicer (e.g. on ff nightly) than canvas/png, e.g. font kerning is correct -- it would be nice if it was possible to enable svg rendering in jupyterlab/notebook (plus if you export the notebook it's all vector graphics and thus magnifiable?).

[jakevdp]

Not all frontends support it yet, but you can pass renderer metadata via the renderers registry... for example:

alt.renderers.enable(renderer_name, embed_options={'renderer': 'svg'})

where renderer_name is jupyterlab, notebook, colab, etc.

I plan to document this better once the next version of JupyterLab is released, because the current release ignores any metadata.

[aldanor]

Does the master branch of jupyterlab support this already? Hoping for a new version soon then in case it does as it’s a bit of a showstopper for us :)

Would this be controlled globally only? (i.e., being able to chart.display('svg') might also be nice to have)

[domoritz]

Yes, it's already in master.

[jakevdp]

I like the idea of allowing chart.display(renderer='png'). It would actually be fairly straightforward to implement, I think. It would essentially look something like this:

class Chart(object):
     # ... all the other stuff...

    def display(self, **kwargs):
        from IPython.display import display
        with alt.renderers.enable(alt.renders.active, embed_options=kwargs):
            display(self)

with maybe a bit more book-keeping to make sure globally-enabled keywords are also used...

It would also help to use explicit keyword args so they'd be more self-documenting

[aldanor]

@jakevdp One last question here, would renderer metadata also be supported in Jupyter HTML exporter, or would that have to be done separately? (or maybe it's already done?)

I.e., if you supposedly do a chart.display(renderer='svg') and then export your notebook to HTML, would you see an embedded SVG?

[jakevdp]

Notebook export is an entirely different thing. Currently, exported HTML does not include any live altiar charts. See #767

[jakevdp]

OK, I've implemented chart.display(renderer='svg') in #925

[jredd]

Does Vega have any render thrashing mitigation built it? Something like Fastdom which combines read/writes into single animation frames and such?

[domoritz]

Vega has some optimizations and only renders dirty elements. You can read the renderer code here: https://github.com/vega/vega/tree/master/packages/vega-scenegraph. If you want to help make it faster, we would love your contribution.

[jredd]

I've only ever done optimization on my D3 charts and while I was able to get a massive performance increase with Fastdom I wonder if on a package scale using something like the shadow Dom would be better? I've actually never contributed to open source so I have no idea what to do there.

[domoritz]

We would love to have your help @jredd. You fork the Vega repo, make and push code changes, and then when you are done send a pull request from your fork to ours.