Incubator-superset: Jinja template + dataframe access from Separator'esque Viz

[kalium99]

Make sure these boxes are checked before submitting your issue - thank you!

I have checked the superset logs for python stacktraces and included it here as text if any

• [x] I have checked the superset logs for python stacktraces and included it here as text if any
• [x] I have reproduced the issue with at least the latest released version of superset
• [x] I have checked the issue tracker for the same issue and I haven't found one similar

Superset version

0.26

Expected results

A Separator'esque widget that can render not just html or text, but can interpolate variables and even include branching logic. I'd imagine that being able to insert Jinja templating with access to the pandas dataframe would suffice.

The reason for it, is that I want to create more descriptive text that might help to explain the data being visualised. For example, the final rendered text may look something like:

"USA's population is growing slower (by 5%) than Australia's"

And the Jinja template may look something like (It's been years since I've done Jinja, so forgive any errors):

  <text>
     {% if df[df.country == 'USA'].growth <  if df[df.country == 'Australia'].growth %}
     USA's population is growing slower by ( {{ df[df.country == 'USA'].growth }} )
     than Australia's.
     {% else %}
     USA's population is growing faster than Australia's.
     {% endif %}
   </text>

I have spent only 5 minutes looking at this, but I suspect we would need a viz object that builds a dataframe based on the form data (i.e columns, metrics, group by etc) as other Viz objects do, but also have a textarea like MarkupViz to save our template.

Anyway, the implementation details would need to be worked out, but for me this would be a great addition as well as for others potentially.

Thoughts ?

Actual results

This feature does not exist AFAIK.

Steps to reproduce

[kalium99]

Here's a screen-shot of the first iteration of it.

[mistercrunch]

One limitation here is having the ability to mix data from different datasources. I'm wondering if we should have something more like:

{% set df = run_sql('SELECT sum(units) AS units FROM sales') %}
We sold a total of <strong>{{ df['units'][0] }}</string> units

To allow for multiple queries / datasources.

A few other considerations:

• should this be a feature of the markdown dashboard-native component?
• Should this be more of a client-side templating? Meaning you could see the text right away and loading spinners within the text while the async calls are happening. I think Jinja is probably fine easier and opens the power of pandas...
• clearly some security concerns here, we should probably make this viz type unavailable by default.

[kalium99]

@mistercrunch
TL;DR; It's probably more secure the way it's currently implemented although its ease of use is not great so maybe it should be disabled by default for that alone?

should this be a feature of the markdown dashboard-native component?

Are you referring specifically to the Dashboardv2 Markdown viz (my patch was actually against the 0.26 branch, which was probably a mistake. I will have to rebase it on the correct branch)? I only looked at Dashboard V2 yesterday. As for the Markdown viz in the original Dashboard I did originally consider this. In effect though, it seems to me that the Jinja template is a data visualisation tool. In the sense that we specify datasources, metrics, columns etc, and then we represent that specific data in a way that hopefully tells some story. The markup viz is just to give some context to other parts of the dashboard. This is also reflected in the control panel; the Jinja viz essentially needs the same control panel elements as other data based viz and would differ from that of the Markup viz.

Should this be more of a client-side templating? Meaning you could see the text right away and loading spinners within the text while the async calls are happening. I think Jinja is probably fine easier and opens the power of pandas...

That doesn't sound like a bad idea, but It's not where my skill set lies, so I've gone with what I know :-) And I also think that having the power of Pandas is genuinely useful.

clearly some security concerns here, we should probably make this viz type unavailable by default.

I'd have to double check, but IIRC if it's done as I've implemented it then there shouldn't be. The df is generated via the Query in the control panel, and subject to all the same access checks that any other viz tool is subject too. One of my concerns is actually that people will file a bunch of bugs against it because they have trouble using it. Here are some gotcha's:

• If you have the metric COUNT(*) then the dataframe column label is 'count', however if your metric is COUNT(year) then it's 'COUNT(year)'
• Jinja doesn't support all valid Python syntax. This will fail in Jinja df.groupby('column1').apply(lambda x: x.std())
• Other things that make it not totally user friendly that I haven't come across.

PS I do like the thinking of mixing data sources etc, but I'm not sure about the security implications etc. I think it would be good to consider doing something like that at a later date.

[ankoh]

@kalium99 +1.
What's the status of this?
Can you share a version that is based on a more recent master commit? I'd love to play around with it!
Making a single data frame accessible via markdown would be phenomenal already!

[ankoh]

I added a hacky proof-of-concept for those who want to play around with the first iteration that @kalium99 described.
I am not too familiar with superset and could miss something but it seems to work.

0001-Add-markup-table-viz.txt

[kalium99]

Sorry, just seeing this comment now. I have been using it on our local Superset instance. Will push a patch up this evening but from memory it doesn't look totally different to yours @ankoh

[kaitorecca]

@kalium99 : Look around for your PR but cannot found it. Have make a PR for this?

[awoehrl]

Any update? That feature would be really helpful.

[kalium99]

Apologies I dropped off the planet for a while. I'm no longer using Superset, so I'm less invested in this. The patch from @ankoh was pretty much the same as what I had, so I suggest someone take that, add whatever else is needed (like tests, documentation etc) and create a PR from it. I'm currently focusing on other things though. Cheers

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[monkevlar]

Any updates?

[micimize]

Latest incarnation #6771 was closed as a #wont-fix