Incubator-superset: [SIP-4] [Embeddable Charts] Create Superset JS Client

[williaster]

@kristw @conglei @mistercrunch @betodealmeida @hughhhh

cc @john-bodley @michellethomas @graceguo-supercat @timifasubaa

[SIP] [Embeddable Charts] Create Superset Client

This is the first of a few SIPs to progress toward a world where we can embed Superset Charts as React components in other web applications. At a high level this will require @superset/xxx NPM packages with data providers + chart components.

Motivation

This specific SIP is for creating a Superset JS client to handle asynchronous requests for data including authentication.

The motivation for this change is two-fold
1) address technical debt by moving from ajax to fetch
2) encapsulate all request logic into a single JS class, which will enable issuing requests for data from other web applications

Proposed Change

Write a single JS SupersetClient class that

• handles CSRF tokens and verifies authentication
• supports GET/POST/PUT/DELETE methods
• supports customization of headers, etc. if needed

After this we can write corresponding React components to handle fetching specific types of Superset data (including in other apps):

• <DatasourceProvider /> returns available datasources
• <MetadataProvider /> given a datasource id, returns metadata on all of its columns, metrics, etc.
• <QuerydataProvider /> given a query, returns the data for that query; this can be uncoupled from visualizations over time.

We can do this in three phases:
1) Write the client, considering all existing request types
2) Refactor existing ajax requests to leverage new client
3) Write react components

New or Changed Public Interfaces

Example usage of the client:

import SupersetClient from 'xxx';

// Singleton instance of the client
const client = SupersetClient.getInstance({
  host: 'http://0.0.0.0:8088',
  cors: 'enable', // talking to the same domain or not
});

// verify authentication and fetch CSRF
client.init()
  .then(...) // now make some requests
  .catch(...); // in another app, link to Superset for Auth

// later
client.get({ endpoint: 'datasources/' });
client.post({ endpoint: 'explore_json/', postPayload: ... });

Authentication and CORS

In order to support requests made from other web applications, we must

• enable CORS (cross origin request sharing) and
• ensure authentication

The Superset backend already supports enabling CORS for specific endpoints and resources. This is an example config.py:

# CORS Options
ENABLE_CORS = True
CORS_OPTIONS = { 
  "resources": { 
    "*": {
      "origins": "http://localhost:9001", "supports_credentials": True, 
    }
  }
}

By setting the fetch parameter { credentials: 'include' } we can also forward credentials for the current user.

Proof of concept

This is a proof of concept using the SupersetClient code in another web application (PR to come). It demonstrates

• ability to verify user authentication
• ability to fetch and use CSRF token in requests
• support for GET and POST requests for Superset data that ultimately powers a chart.

New dependencies

This should only require the whatwg-fetch polyfill, which has been added / considered in other PRs (eg #5524) and has no license concerns.

Migration Plan and Compatibility

We don't anticipate that any migration will be necessary or breaking compatibility changes.

Rejected Alternatives

We considered a client that also handles redux state management, but this seems too complex to start and could be added as another layer in the future if it was needed for embeddable charts (we think it won't be)

Major open questions

A major open question is WHERE this code (and other future @superset/... JS npm packages) should ultimately live. The options are:

1) within the apache/incubator-superset repo under a src/packages/ subfolder, or

2) a separate repository (which?!)

Pros of apache/incubator-superset

• Centralized, all code in one place
• Member management in one place
• Centralized place for all contributions, dependencies, etc.

Cons of apache/incubator-superset

• Combine issues from all components. (could address with per-package tags)
• Possible confusion if/when superset app version dependencies differ from the most-up-to-date packages/* version
  (ie superset app does not pull from local package source code, rather it pulls from npm for a particular version of that source code)
• Deprecating a package means deleting a code from central repo instead of leaving the git repo alone.

• Combined git log can be very polluted

  Overall it seems we should move packages to a different repsoitory. Where should this be? (note: I grabbed the npm @superset namespace for publishing so we should be good there! 👍

[tiffon]

Hello, full-disclosure, I'm not a user of superset (although, it looks awesome!), I'm just browsing and had a question:

Why go with React components instead of an iframe based approach? Seems like the latter affords more flexibility from implementation, maintenance and usage perspectives?

[williaster]

@tiffon good question.

we already support embedding charts (and dahsboards) with an iframe via a url?standalone=true flag. we've customized the queries of these charts with some success, but it doesn't allow for a pattern where you might want to expose vis controls easily. you could do this within the Superset code with url parameters but that adds more complexity.

instead we'd like to introduce modular data providers (for data sources and queries) and embeddable charts to enable greater flexibility in apps which hope to leverage the Superset backend. you could imagine wanting to build a complex visualization in another web app, but wanting to leverage Superset data to power it. This would also support that use case.

finally, we also expect that the modularity that's enforced by this approach will also greatly reduce coupled code/related bugs.

[victornoel]

@williaster great work, I would most certainly be a user of this feature!

Two small questions:

• Do you already have an idea how it will be possible to have the same functionality as a widget filter box for several charts embedded in a client?
• How do you see the embedding of a chart already defined in superset (instead of building one from the client side in your example)?

[williaster]

@victornoel also good questions!

I think it's actually easier to answer your second question first. It's a bit beyond the scope of this SIP (another one will be posted shortly that describes this in depth) but we are hoping to refactor the frontend code into a series of NPM packages, for the time being you could imagine @superset/connection (for data-related components) and @superset/charts(for chart related components) packages. You could then import individual Superset visualizations or a component we're thinking of as a "<SuperChart id={123} />" component that could take a chart id and render any vis type.

As for exposing controls like group bys and filters in another application, here's an overview of some different usage scenarios we have thought about with associated components. If the app using the embeddable charts knows about column metadata, it can create the appropriate group bys and filters that then affect the query for the chart. (note this doesn't use the filter widget specifically, the other application would need to write the logic for controls; in the long-run we would also hope to create @superset/controls package to expose these types of controls out of the box)

[victornoel]

@williaster thanks for your answer, this new endeavour around embeddable charts and chars as plugins is really a great advancement for superset I think :)

[williaster]

@mistercrunch what are your thoughts on where JS packages should live? I can create this within the apache/incubator-superset repo for now, but would be ideal to settle on a long term solution.

Overall it seems like there are more pros to moving it _outside_ of the superset-repo, but I also see the merits of keeping it in one.

[mistercrunch]

@williaster I'm open to both approaches, though apache makes things harder on us as we cannot create repos easily under the apache/ and they also make it harder to make official Apache releases. Ideally we'd create a superset Github org with all the repos with need, and all committers would be admins, thought that is not the apache way

I think it's fairly easy to revisit this decision too. The question is if not under apache/, where? This may require a mailing list thread with our apache mentor Ashutosh.

Options:

• create a superset GH org, move many of our smaller repos there, keep the main repo under apache, there we'd have full admin rights, no need to ask Apache infra for anything
• all packages under the current repo
• work with Apache infra at creating the many repos as we need them

We may be able to dig out Ashutosh's GH account and tag him here.

[williaster]

@mistercrunch one clarification here is that we don't need _multiple_ repos for the ui packages. It's pretty common to have one mono repo (eg xxx/superset-ui) that then contains multiple packages. It's almost easier to have them all in one place when there are inter-dependencies, and tools like lerna.js exist to help manage this + publishing. vx and data-ui are both written this way so I'm pretty familiar with how to do it.

So an example repo dir structure would look like

xxx/superset-ui

// root package.json that has deps like lerna.js and shared things like build, testing, linting
package.json 

// each of these packages can be developed independently, and have dependencies on each other 
// lerna.js is used to manage inter-package dependencies + publishing.
packages/ 
  superset-core/
    package.json
    README.md
    src/ // source for THIS package only 
    tests/
  … 
  superset-plugin-vis-timeseries/
  superset-plugin-vis-histogram/
  …
  superset-preset-vis-core/
  superset-preset-vis-airbnb/
  superset-preset-vis-lyft/

so you could do that in a separate repo, or include it within apache/incubator-superset and update the structure to be

superset/
  assets/
    // this defines the deps of the overall Superset app (including superset 
    // npm packages) + manages the build. This allows the packages to be developed 
    // and versioned independently from the Superset app 🎉
    package.json 
    webpack.config.js
    … 
    src/
      // these do NOT have package.json’s, their deps are defined in the root.
      // when an app has a dependency on a package (e.g., @superset/core), 
      // it is referencing the npm version of that code, not the code in the local 
      // repository. In development you can symlink such deps
      apps/
        explore/
        dashboard/
        …

      // each of these packages can be developed separately from the app.
      // lerna.js is used to manage inter-package dependencies + publishing.
      packages/ 
        superset-core/
          package.json
          README.md
          src/ // source for THIS package only 
          tests/
        … 
        superset-plugin-vis-timeseries/
        superset-plugin-vis-histogram/
        …

[arpit-agarwal]

@williaster @kristw
I understand that idea is to make superset JS Charts available in external react application via this SIP.
However we can also enable 3rd party react components (charts) to be used within superset dashboards.
The majority of work planned for Embeddable chart (#5692 ) and ChartPlugin (#5882) will be reused.

We need to expose a interface that will be adhered by visualisation to be embeddable in suoerset dashboard. The interface will be derived from refactoring proposed in multiple SIP

One open question at this point of time is how and when to pull third party code and security sandboxing.
We can refer Kibana for some aspirations https://www.elastic.co/blog/developing-new-kibana-visualizations

[williaster]

@graceguo-supercat adding my responses to your SIP-4 Dashboard refactor comments here.

Is there any existed Superset dependencies that depends on jquery?

With my series of PRs, the intention is to remove all jQuery ajax calls. This is one step toward completely removing jQuery altogether, which is used in other places for DOM manipulation and possibly react-bootstrap. This is a large enough task that doing it in chunks is probably the best way forward.

When we need a little change on FAB, I need to fix it in another codebase, tested it in another codebase with many context i am not familiar, wait it build and generate a new version, then i can use it in Superset. Is it possible that make the new toolkit inside Superset, but generate standalone modules, to share with embeddable chart?

Couple thoughts

• Above I enumerated some pros/cons of whether @superset-ui packages should live within this repo, or a separate one. The Pros of moving it to a separate repo seemed to outweigh the cons, so we are moving forward with the separate superset-ui repo.

• While you're right that it may add some additional overhead to address an issue or develop a feature for superset in the separate superset-ui repo, this can be largely addressed using npm link which simply sym-links package @superset-ui/xxx in node_modules to your local @superset-ui/xxx dev repo.

For the long term, when we develop some new features (like improve logging or batch-fetch etc, maybe?), I need to consider should i add this additional function in Superset or in the toolkit code base?

We discussed this in person a bit. To start we'll play it by ear, if something would be useful in another repository, we could make it into a package.

[williaster]

@arpit-agarwal thanks for the thoughts. I think that the chart plugins will enable embedding 3rd party react components/charts in a dashboard.

@kristw is working on fleshing out some of the requirements for the plugin system now in SIP-6. Thanks for the link to how kibana supports plugins, we'll 👀

[victornoel]

@williaster I'm trying to plan our migration to superset and we are wondering if there is some kind of tentative timeline for the availability of this feature.
Say to have embedded charts from superset in another application without any specific customization.

Thanks for any information, even if there is no commitment on them :)

[williaster]

Hey @victornoel 👋 The SupersetClient described here has been implemented in a new separate superset-ui repository and is exported by @superset-ui/core. With the correct CORS configuration, this should allow you to make requests to a Superset backend in any application and I'm refactoring the Superset application to use it now.

@kristw is working on untangling all of the visualizations to support the actual Chart Plugin system in described in SIP-6. The plugins themselves are coming along well (e.g., this PR) and should be wrapped up in Q4 this year.

The trickier part of this will be defining consistent APIs for query + data munging, and fully migrating query specifications and data munging to JS from Python. @conglei is working the latter right now. I would say that work will be complete in Q1 next year most likely.

[victornoel]

@williaster thanks a lot for the detailed answer, all of this new architecture is very exciting, both SupersetClient and the new plugin system (that I will also exploit in the future).

So basically, we can already (with current master and @superset-ui/core) see charts defined in superset in third party apps. It means it is already time for me to start experimenting with it!

One last small question: any idea when the next stable superset will be released?

[sashank]

Yes waiting for a stable release of this feature

[williaster]

Going to close this as complete: @superset-ui/connection now powers all async requests in the Superset app, and several other modular packages are under active development in @superset-ui to power other ideas described here and captured in other SIPs.

Noting that this SIP was used more as formalizing architecture design and not for any formal voting.

[rojan507]

I am trying to embed chart into my angular app .What is the best way to do it?
Do you have an example?
Thanks

[williaster]

Hi @rojan507 this is not quite ready to embed in an external app. It is likely that this ability will be added chart by chart because we have to migrate backend code for every vis type to the client to allow for JS-only plugins. You can follow SIP-6 and see the first plugin PR (word cloud) for further details.

[victornoel]

@williaster I thought this was already possible as long as the chart and its backend code was already included in superset?

[kishikishore]

Hello Friends,
This is kishore from india ,it does n't make sense of apache superset embedded in react component . could anyone give the briefly of these example and attach example code also here.