Incubator-superset: Loading custom 'Viz' plugin

[robinbakkerdemcon]

Hello,

Recently I have started setting up Apache Superset and to archieve what I am looking for (a custom table view) I want to make a plugin. I have followed this tutorial (maybe 4 times) https://preset.io/blog/2020-07-02-hello-world/

When I follow this tutorial and write npm run dev-server it will start the dev-server at: http://localhost:9000. It perfectly loads my plugin, but the problem is that I work from home and do everything through Putty on another laptop within the company network. Now, since I work from home I want to test my plugin, how can I archieve to run the dev-server on IP '0.0.0.0'?

Normally to run my superset server I write superset run -h 0.0.0.0 -p 8088 this perfectly loads up superset and I can access it at home but this does not load the hello-world plugin!

What I already tried

1. I created a new plugin from scratch following the same steps
2. I tried running npm run dev-server --host 0.0.0.0

So I see two possibilities:

1. Can I load my plugin when running superset run -h 0.0.0.0 -p 8088
2. Can I access the dev server from home?

[robinbakkerdemcon]

Further I would like to mention that this issue is nearly the same as issue #10433 . Running npm run prod also throws the following error:

I have tried every solution from this issue (copying it manually instead of npm link, deleting node_modules etc.).

[mistercrunch]

@rusackas

[rusackas]

Hmmm... I haven't personally run into this, so I don't have a good answer without doing some investigation, but I'm glad the plugin process itself is working!

My _suspicion_ is that you'll have to dig into incubator-superset's webpack.config.js to move away from localhost and port 9000.

The alternative, if you have superset running for you in other circumstances, would be to go further towards production by actually building the plugin as a module, publish it on NPM, and import the package in your instance of Superset.

[robinbakkerdemcon]

Hello and thanks for the response. I have edited the webpack.config.js and edited the following line:

const PREAMBLE = [path.join(APP_DIR, '/src/preamble.ts')];
if (isDevMode) {
  PREAMBLE.unshift(
    `webpack-dev-server/client?http://172.17.6.165:${devserverPort}`,
     // I've kept the dev server port the same. Only changed the IP!
  );
}

But when I run sudo npm run dev-server it will listen on http://localhost:9000 again. Is there also a tutorial for building my plugin as module and then importing it from (a locally hosted??) NPM server to superset?

[robinbakkerdemcon]

@rusackas
Alright so I have been busy with this problem again. Now, I have installed a desktop environment on the machine rnning Superset and through VNC I can access the dev server that is running locally, I verify this because the browser tab shows [DEV]. I have tried to use the plugin now and the command npm run dev-server correctly shows

I assumed that I would work, but it doesn't work. It does not show up in Visualizations whatever I do.

My package.json

My MainPreset.js
Imports

Configure

Suspected problem

When I run the npm run dev-server at the end I get these errors:

[robinbakkerdemcon]

Update, I have reinstalled everything, even Ubuntu server 20.04. Reinstalled Superset through this tutorial
https://medium.com/@kharekartik/a-better-guide-to-building-apache-superset-from-source-2c8dbad38b2b

Created a plugin from scratch using the preset.io hello world tutorial, and still no result. Is there anyone who is able to help my with this issue? I want to use Superset but I cannot find a solution to this particular problem. @rusackas @mistercrunch

Currently the plugin does not load inside the dev-server environment. The symlink message does appear but no plugin in my visualization list (Charts -> create new chart). When I want to run yarn build inside the superset-ui folder I get a message that 'build is no file or directory' maybe this is related to the issue?

[avicenteg]

@robinbakkerdemcon if you want to use yarn build first you'll need to use yarn install, and maybe you'll also need npm install. I'm running in the same problem than you... Are you using docker environment?

[robinbakkerdemcon]

@robinbakkerdemcon if you want to use yarn build first you'll need to use yarn install, and maybe you'll also need npm install. I'm running in the same problem than you... Are you using docker environment?

Yes I have done that, yarn install and npm install but sadly I am running into the same issue as before. I am not running in a docker environment. I have installed all python dependencies inside a Virtual Environment as recommended. I am sorry that you have the same problem, I tried reinstalling everything (even Ubuntu Server) and installing everything from scratch again using the guides provided by Apache Superset but it still won't work.

For now I have given up on trying everything I can, but if someone is able to help me (and now you) with this problem I still want to try and get this working.

[returnzer0]

I'm running into the same issue when adding custom plugins, I'm using docker.

When I add a new custom visualization plugin using this tutorial and run docker-compose, I get this error

superset_node            | npm ERR! code E404
superset_node            | npm ERR! 404 Not Found - GET https://registry.npmjs.org/@superset-ui%2fplugin-chart-hello-world - Not found
superset_node            | npm ERR! 404 
superset_node            | npm ERR! 404  '@superset-ui/plugin-chart-hello-world@^0.0.0' is not in the npm registry.
superset_node            | npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
superset_node            | npm ERR! 404 It was specified as a dependency of 'superset-frontend'
superset_node            | npm ERR! 404 
superset_node            | npm ERR! 404 Note that you can also install from a
superset_node            | npm ERR! 404 tarball, folder, http url, or git url.
superset_node            | 
superset_node            | npm ERR! A complete log of this run can be found in:
superset_node            | npm ERR!     /root/.npm/_logs/2020-11-30T21_19_48_140Z-debug.log

I can run the dev server using npm run dev-server inside the superset-frontend folder and it gets served via localhost:9000, but docker doesn't seem to play that well, the frontend at localhost:8088 and logo keeps spinning.

[nytai]

for developing a custom plugin, you'll want to use the dev-server. Docker is running in prod mode, so it will not handle any npm liking magic

[returnzer0]

@nytai Thanks for the reply, once I'm finishing developing the plugin and want to show it in prod, is that possible? Or are custom plugins limited to dev only?

[nytai]

You'd have to publish the plugin as an npm package and then add it to superset. I'm not sure if we'll be merging community plugins into upstream master yet, but you can always maintain your fork with a commit that adds the custom viz plugin. Of course that means building your own custom prod version of superset from source, but you should have all the tools necessary.

We have a dynamic plugin marketplace type feature in the works, but not sure when that will be coming.

[rusackas]

@returnzer0 the most common path for that would be to publish the package on NPM (including the built assets) from your own repo, so that you can then import the NPM module as a dependency in your Superset implementation.

[returnzer0]

@rusackas Thank you, I have another question in hopes of not swaying away from the original issue.

If I don't really care about publishing the custom viz on npm, but I simply want to create a docker-compose build which I can then host anywhere, privately, is there any way to go about it? I think that my concern and @robinbakkerdemcon are quite similar. (please correct me if they aren't)

Currently, I'm just getting npm linking errors, and following #10433 doesn't get me anywhere. docker-compose up still throws module not found errors. I think the problem exists in the docker-entrypoint.sh script, where it installs the npm packages.

[nytai]

You could try building the viz plugin and installing it using the file:... reference instead of linking, mentioned here https://stackoverflow.com/a/17371987/13977852

[nytai]

actually, I bet the issue with docker (via npm link) is that the linked plugin isn't in the docker file system. I'd try moving the plugin into superset-frontend (which should be volume mounted) and try pointing to it using the file:... reference (which should be easier to debug than npm link)

[ktmud]

Not sure I fully understood the problem, but I'd recommend try not to rely on Docker for the dev build.

You may even try to start Superset with Docker, but checkout another copy of superset source code to have a clean build.

The webpack dev server allows you to point to arbitrary remote host (even if it is running on a different version of Superset), so as long as you have the backend successfully running somewhere, you can always build the dev server locally to get more predictable build results.

# Run dev-server at 8080 and proxy to local port 9000
npm run dev-server -- --port 8088 --superset=http://localhost:9000

[avicenteg]

I'm running into the same issue when adding custom plugins, I'm using docker.

When I add a new custom visualization plugin using this tutorial and run docker-compose, I get this error

superset_node            | npm ERR! code E404
superset_node            | npm ERR! 404 Not Found - GET https://registry.npmjs.org/@superset-ui%2fplugin-chart-hello-world - Not found
superset_node            | npm ERR! 404 
superset_node            | npm ERR! 404  '@superset-ui/plugin-chart-hello-world@^0.0.0' is not in the npm registry.
superset_node            | npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
superset_node            | npm ERR! 404 It was specified as a dependency of 'superset-frontend'
superset_node            | npm ERR! 404 
superset_node            | npm ERR! 404 Note that you can also install from a
superset_node            | npm ERR! 404 tarball, folder, http url, or git url.
superset_node            | 
superset_node            | npm ERR! A complete log of this run can be found in:
superset_node            | npm ERR!     /root/.npm/_logs/2020-11-30T21_19_48_140Z-debug.log

I can run the dev server using npm run dev-server inside the superset-frontend folder and it gets served via localhost:9000, but docker doesn't seem to play that well, the frontend at localhost:8088 and logo keeps spinning.

Hello @returnzer0 I reach a solution with docker. I'm using a old plugin (JS instead of TS) but I think it should work.

1. I created a folder in the superset-frontend where I put my new plugins (p.e: superset-ui/plugins).
2. I create manually an entrance in the package.json, linking the version of the plugin with a file.
   "@superset-ui/legacy-plugin-chart-mychart": "file:superset-ui/plugins/legacy-plugin-chart-mychart"
3. I run npm install in superset-fronted in order to update the package-lock.json file. After the process is complete, delete the node-modules folder and its content.
4. In the Dockerfile, I added:
   COPY ./superset-frontend/superset-ui /app/superset-frontend/superset-ui
   This command should be before this other command.

RUN /frontend-mem-nag.sh \
    && cd /app/superset-frontend \
        && npm ci

1. In the docker-compose I created a volume (this volume was present in old versions of superset).

x-superset-volumes: &superset-volumes
  - ./docker/docker-init.sh:/app/docker-init.sh
  - ./docker/pythonpath_dev:/app/pythonpath
  - ./superset:/app/superset
  - ./superset-frontend:/app/superset-frontend
  - node_modules:/app/superset-frontend/node_modules  <- this line was changed
  - superset_home:/app/superset_home

volumes:
  superset_home:
    external: false
  db_home:
    external: false
  node_modules:   <- this line was changed
    external: false
  redis:
    external: false

1. In the docker-compose I use install without the --f option for the superset-node.

  superset-node:
    image: node:12
    container_name: superset_node
    command: ["bash", "-c", "cd /app/superset-frontend && npm install --global webpack webpack-cli && npm install && npm run dev"]
    env_file: docker/.env
    depends_on: *superset-depends-on
    volumes: *superset-volumes

1. I docker-compose up.
2. Probably, when you do that, you won't have any error, but the new plugin will be still missing. In this case, goes to the folder of your plugin and check if there are a node_modules folder. If it is in there, delete it.
3. If you haven't still, shut down the superset instance, and change the superset-node command in docker-compose, deleting the npm install option.

  superset-node:
    image: node:12
    container_name: superset_node
    command: ["bash", "-c", "cd /app/superset-frontend && npm install --global webpack webpack-cli && npm run dev"]
    env_file: docker/.env
    depends_on: *superset-depends-on
    volumes: *superset-volumes

1. docker-compose up again, and it should be working, at least for me.

The steps from 1-8 should be repeated when you add a new plugin, but if you are doing some changes in your plugin you can change the files in the plugin and the changes will be reflected in your instance when you restart it.

I hope this works for you. I know how frustating is it, I spend 2 whole weeks to understand it.

[returnzer0]

Thank you, everyone! It was indeed an issue with npm package linking (or lack of).

I tried what @avicenteg outlined and I am able to run it with docker-compose up.

I would add that, in step 6, I didn't change anything. Instead, I went into docker/docker-frontend.sh bash file to change the commands, but after running it once, I realized that if I added any new plugins, I'd need to bring down the volumes as well, so that npm can install packages, which is something I want to avoid, just in case. So I keep the -f and --no-optional flags in the bash script so that new packages are installed regardless, every time I run docker-compose up.

@robinbakkerdemcon hope this also shines some light on your issue, I am able to run it via docker and expose a port on my gateway and can access it from anywhere and all of the plugins are loading just fine for me.

Edit: I wanted to add that this method works for new files (typescript), and for both class and functional react components.

[DrissiReda]

I have tried publishing my npm package, adding it to package.json and adding it to src/visualizations/presets/Mainpreset.js and I'm getting the same error:

ERROR in ./src/visualizations/presets/MainPreset.js
Module not found: Error: Can't resolve '@reda_drissi/legacy-preset-chart-deckgl-maptiler' in '/app/superset-frontend/src/visualizations/presets'

While building the docker image with the official dockerfile.

I also ran "npm install" locally to build the appropriate package-lock.json so that "npm ci" in the Dockerfile finds the right file.

[returnzer0]

Did you try adding it via the file:... directive? I don't have any experience with publishing the package, but try to see if you can first include it locally.

[DrissiReda]

@returnzer0 that's how I started, it was a local package that I tried using with npm link then with only file: ..., both didn't work, also only the dev build command is supposed to work with those, not the production build.