Generator: How to use pre-install template?

[aleung]

Reason/Context

We have all API files managed in one git repo and have CI pipeline to generate document from these files. I'm going to support asyncapi and use @asyncapi/generator to generate HTML document. The CI pipeline runs in a Docker container so I need to pre-install both generator and html-template into the Docker image.

There are several reasons that we don't want generator to install template on-the-fly: we can't secure the internet connection in the CI environment; we don't want to waste time to download in every CI job; and also our strategy is that everything in CI should be immutable. The way described in #449 to mount a volume doesn't fit for our CI neither.

I have tried to install both generator and template in Dockerfile but generator doesn't use the global installed template. It looks for template under its own node_module folder: /usr/local/lib/node_modules/@asyncapi/generator/node_modules/

RUN npm add -g @asyncapi/generator @asyncapi/html-template

I also tried to install template using generator CLI, but the command failed because it only download and install template during generation.

RUN npm add -g @asyncapi/generator
RUN ag -i @asyncapi/html-template

Finally I made it work by adding a dummy asyncapi file into the image then generate it during Docker build:

COPY docker-content/dummy-asyncapi.yaml .
RUN npm add -g @asyncapi/generator \
    && ag -o /tmp/output -i dummy-asyncapi.yaml @asyncapi/html-template \
    && rm dummy-asyncapi.yaml && rm -rf /tmp/output

However it's just a workaround and not looks elegant.

Description

I see there might be two alternatives to resolve the problem:

Alt#1

Generator uses the global installed template. It skips on-the-fly installation if template already exists.

Alt#2

Generator CLI provides template management command. For example ag install-template <template> to install the template into its node_modules folder.

[github-actions[bot]]

Welcome to AsyncAPI. Thanks a lot for reporting your first issue.

Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

[derberg]

@aleung Hey, thanks for raising this issue. I think it is very much related to https://github.com/asyncapi/generator/issues/425. @laat tried to fix it already.

Tbh it is frustrating to me that there are so many installation related issues and I could not find time to fix them yet 😞 . There are few installation-related issues that we should address imho. Atm I'm involved in major refactoring of our CI pipelines and it will take me few more days to complete and then I could jump into it and fix it (if I manage 😅 ). Unless you want to give it a try?

Only one comment

we can't secure the internet connection in the CI environment;

Shouldn't you do what corporations do usually, setup an npm mirror in an internal network, set proxies, and basically make sure that npm install takes a package from an internal, scanned mirror?

[aleung]

@derberg Thank you for your quick response. My workaround works so it isn't in a hurry. I'm still new on this project and I'll try to see if I can contribute something.

Shouldn't you do what corporations do usually, setup an npm mirror in an internal network, set proxies, and basically make sure that npm install takes a package from an internal, scanned mirror?

Yes, we do have setup npm mirror. However there may be some restrictions on the modules/versions available on the mirror. It isn't a blocking issue so far but I wish my CI job can have less dependency on the environment.

[derberg]

@aleung hey, this https://github.com/asyncapi/generator/pull/517 should fix the issue. Have a look if you can, especially the integration tests to make sure the way I check "using templates as dependency" makes sense

[aleung]

@derberg Thanks. I've looked into the test project. It should already cover the case in this issue.

[derberg]

@aleung thanks for taking the time 🙇🏼

[asyncapi-bot]

:tada: This issue has been resolved in version 1.2.0 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket:

[derberg]

Release 1.2 enables support for latest node and npm and fixes all known installation issues. In case you see some other issues, please create a new issue. Please use --debug flag when you experience issues with the generator and in the issues mention the troubleshooting logs. Thanks!

[aleung]

@derberg I tried the new release and got error:

# ag --version
1.4.0

# ag -o ./ --force-write -p singleFile=true index.yaml @asyncapi/html-template --debug
Errors while trying to resolve package location at undefined TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at validateString (internal/validators.js:120:11)
    at Object.join (path.js:1039:7)
    at /usr/local/lib/node_modules/@asyncapi/generator/lib/generator.js:353:41
    at new Promise (<anonymous>)
    at Generator.installTemplate (/usr/local/lib/node_modules/@asyncapi/generator/lib/generator.js:339:12)
    at Generator.generate (/usr/local/lib/node_modules/@asyncapi/generator/lib/generator.js:168:73)
    at Generator.generateFromString (/usr/local/lib/node_modules/@asyncapi/generator/lib/generator.js:254:17)
    at async /usr/local/lib/node_modules/@asyncapi/generator/cli.js:132:9 {
  code: 'ERR_INVALID_ARG_TYPE'
}
Template installation started because the template cannot be found on disk
^C

# ls /usr/local/lib/node_modules/\@asyncapi/html-template/
filters  macros        package.json  postcss.config.js  tailwind.config.js  test
hooks    node_modules  partials      README.md      template

# cat /usr/local/lib/node_modules/\@asyncapi/html-template/package.json | grep version
  ......
  "version": "0.18.1"

# which ag
/usr/local/bin/ag

# ls -l /usr/local/bin/ag
lrwxrwxrwx 1 root staff 46 Mar  5 05:37 /usr/local/bin/ag -> ../lib/node_modules/@asyncapi/generator/cli.js

This is how I installed them in Dockerfile:

RUN PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true npm add -g @asyncapi/generator @asyncapi/html-template

[derberg]

@aleung the issue is that you are installing html-template here as a global dependency, this cannot work. Just take any project in node. Let's say you want to use lodash there, but do not want to add it to dependencies and instead you install it globally. Require/import will not work in this case. This is the case you try to do here.

in your case we look for the module here /usr/local/lib/node_modules/@asyncapi/generator/node_modules but globally installed HTML template is here /usr/local/lib/node_modules/@asyncapi/html-template

I understand that you want to preinstall all in a docker image so later generation is super fast.

Workaround 1:
set this env variable export NODE_PATH=/usr/local/lib/node_modules or this if previous doesn't work export NODE_PATH=/usr/lib/node_modules. You just need to point NODE_PATH to a location where global packages are, you can check it with npm root -g

Workaround 2:
The quick workaround would be to move sources of HTML template, after installation under the generator.

Please let me know how it went

[aleung]

Ok, then I can rollback to use my previous workaround: generate a dummy asyncapi file during Docker build to install the template. Not a problem to me.

However, I saw a lot of other tools which use npm package as plugin can use global installed plugin. Since ag can be installed globally, it ought to support global installed plugin as well.

[jonaslagoni]

Just to quickly mention this, the generator supports local path to templates so the following could be done if I am not mistaken:

ag -o ./ --force-write -p singleFile=true index.yaml /usr/local/lib/node_modules/@asyncapi/html-template --debug

[jonaslagoni]

Furthermore @derberg I can see there is a resolve-global package we can take advantage of. I actually see this as a great use-case given this issue as well as https://github.com/asyncapi/generator/issues/523

[aleung]

Another one: requireg

[jonaslagoni]

@aleung would you like to create a feature request? Didn't realize this issue is closed 😄

[derberg]

yeah, no need to create another issue. I don't want it to keep waiting again few months. I'll give it a try this week with resolve-global as the author is the same as for the other package we use for resolving deps. Most important for me will be to make sure we can test this properly. To make sure tests can run on local, I'll have to first install somehow the template globally inside the test

[aleung]

@jonaslagoni Actually this issue has not been solved yet :smile: . My original issue description is about using pre-installed global template.

@derberg Thanks for your active support!

[derberg]

yeap, I have to admit I missed that -g in npm add -g @asyncapi/generator @asyncapi/html-template also applies to the html-template

[derberg]

PR is opened https://github.com/asyncapi/generator/pull/530
just need few hours to figure out integration tests of this global package support

[aleung]

I'm sorry to report that the global installed generator doesn't work any more in v1.5.0. Created the issue #531.

[derberg]

PR ready for review
@aleung the other issue, let's leave it as separate one

[asyncapi-bot]

:tada: This issue has been resolved in version 1.6.0 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket: