Incubator-superset: [SIP-32] Proposal for moving frontend assets to the base of the repo

[nytai]

[SIP] Proposal for moving frontend assets to the base of the repo

Motivation

Currently frontend assets are embedded in the superset package under superset/assets/, which is symlinked insuperset/assets/static. This is really confusing for newcomers to the project as it's unclear why code appears to be duplicated. Additionally this is a UNIX symlink so the link isn't created on Windows machines.

Moreover is seems there's code (eg, MANIFEST.in, Dockerfile ) that tries to work around the fact that the entire frontend exists in the superset module and most of these files are not needed during runtime. Things like whitelisting files and removing files after builds are workarounds to the actual issue and only add complexity to the problem.

Proposed Change

• Move the frontend code into a top level directory, something like frontend or client
• Configure webpack to output the built/dist files to superset/assets/dist. This can include the package.json (needed for version info) and any other necessary runtime files.
• remove the symlinks.
• update build scripts
• make sure everything works (local builds, docker builds, python package, CI, etc.)
• update documentation.

New or Changed Public Interfaces

[no changes]

New dependencies

[no changes]

Migration Plan and Compatibility

No major/breaking changes should be necessary. The bulk of the work would be around making sure no existing builds/scripts break and documentation is updated. Since most of the changes will only be moving files, rebases should be automatic, though this will probably require a lot of existing branches to be rebased.

Rejected Alternatives

1. Remove the symlink and move everything to superset/static/assets and continue to support the whitelisting in MANIFEST.in and the cleanup in Dockerfile and possibly other places.
2. Continue to support the current file organization and continue to field questions such as: https://github.com/apache/incubator-superset/issues/8804

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label #enhancement to this issue, with a confidence of 0.85. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[nytai]

🏷 #SIP

[nytai]

🏷 SIP

[nytai]

🗑 🏷 SIP

[nytai]

looks like the bot doesn't like #s 😭

[willbarrett]

🏷 #SIP

[nytai]

I just ran black superset and it looks like it checked and reformatted some python code in node_modules. Aside from the question of what are python files doing node_modules, this raises the concern that linters/formatters are running on files we don't care about.

[rusackas]

I would LOOOOOVE to no longer see the (apparent) code duplication. It not only makes things confusing at the start, but tedious (e.g. searching) even when you know your way around a bit.

I prefer "client" to "frontend" for some reason, but I'm not stuck on it.

Let me know if/where I can help in this effort!

[mistercrunch]

+1, I have no strong opinions on the topic, but the folder name could also be javascript

[mistercrunch]

Don't forget to notify the dev@ mailing list about the SIP, it should really just be a pointer and the conversation should happen here on Github (rich content, versioned, reactions, bots, ...).

[DiggidyDave]

I would even go a step farther and explicitly break out top level dirs for 2 distinct, separate projects:

• superset (backend service): all things python go under here, and no python goes above it (unless it is decoupled from superset's backend service codebase, ie utilities, scripts, etc, for working with the full project)
• superset-frontend: nodejs stuff fully contained under here

...and move all requirements*.txt, and setup.py etc--literally everything that is scoped to the backend--out of the root and into that backend svc root dir.

This would be more explicit, much much cleaner and self-documenting (IMO), and it would allow each project root to have its own well-scoped Dockerfile, .gitignore, etc in the root. It would make it clear where the test infrastructure lives for the different bits of frontend and backend (right now the whole thing is powered by python tox orchestration) and would be more intuitive for folks who are used to one stack or the other.

[craig-rueda]

I would even go a step farther and explicitly break out top level dirs for 2 distinct, separate projects:

• superset (backend service): all things python go under here, and no python goes above it (unless it is decoupled from superset's backend service codebase, ie utilities, scripts, etc, for working with the full project)
• superset-frontend: nodejs stuff fully contained under here

...and move all requirements*.txt, and setup.py etc--literally everything that is scoped to the backend--out of the root and into that backend svc root dir.

This would be more explicit, much much cleaner and self-documenting (IMO), and it would allow each project root to have its own well-scoped Dockerfile, .gitignore, etc in the root. It would make it clear where the test infrastructure lives for the different bits of frontend and backend (right now the whole thing is powered by python tox orchestration) and would be more intuitive for folks who are used to one stack or the other.

Only potential problem I see with @DiggidyDave 's option is that we would have nested superset dirs. A slight variation to this solution could be:

• superset-py - superset top-level package lives in here, along with py related stuff
• superset-frontend - node stuff

Thoughts?

[suddjian]

+1 on distinct top-level dirs. To chime in on the naming discussion, I think just calling them "frontend" (or "client") and "backend" would be good. Using "superset" in the directory names seems superfluous. Disagree with the name "javascript" as that's just one of at least 3 languages in use.

This should actually be SIP-32. Looks like not all SIPs are labelled correctly (~fixing~ not fixing since the bot can't do #).

[DiggidyDave]

FWIW I personally also prefer names that explicitly contain frontend and backend, but I was afraid it would get the opposite pushback. FWIW my "real" suggestion would be superset-frontend and superset-backend, which leaves open the possibility in the future of having some other subproject that is not superset that is another app or service in the root dir also.

I have a very minor objection to either superset-py or javascript simply because those are coupling the thing being named with the tech currently in use. (Not that we are liable to have a sudden switch to typescript or go or anything like that, but sometimes other things get mixed in a bit, as noted by @suddjian sometimes we sprinkle in other tech... so less specificity in the name w.r.t. the tech in name would be good imo.