Openpgpjs: "Cannot find module 'asmcrypto.js/dist_es5/..." on `npm install && npm test`

[jpbochi]

I'm having almost exactly the same behavior as described https://github.com/openpgpjs/openpgpjs/issues/866

I attempted the same fix as suggested there, namely update my npm. It didn't work.

To avoid any possible issues with my local machine, I configured a CI pipeline on GitLab, and it fails there also. This is the build log:

$ node --version
v12.5.0
$ npm --version
6.9.0
$ npm install
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 786 packages from 1143 contributors and audited 7464 packages in 41.275s
found 334 vulnerabilities (3 low, 11 moderate, 320 high)
  run `npm audit fix` to fix them, or `npm audit` for details
$ npm test

> [email protected] pretest /builds/jpbochi/openpgpjs
> grunt

Running "browserify:openpgp" (browserify) task
>> Error: Cannot find module 'asmcrypto.js/dist_es5/aes/cfb' from '/builds/jpbochi/openpgpjs/src/crypto'
Warning: Error running grunt-browserify. Use --force to continue.

Aborted due to warnings.
npm ERR! Test failed.  See above for more details.

And this is the build config:

image: node:12.5

test_all:
  script:
    - node --version
    - npm --version
    - npm install
    - npm test

I also tried node v10.16.0 and v8.16.0 (the same ones used in the last travis-ci builds, and none worked. As far as I could tell , the directory node_modules/asmcrypto.js/dist_es5 was never created in any of the attempts.

[jpbochi]

I finally managed to get a build passing. I updated the dependency to point to github:jpbochi/asmcrypto.js#11e3dd0650718924b8bbff450a947cad4d4c4b15 (https://github.com/jpbochi/asmcrypto.js/commit/11e3dd0650718924b8bbff450a947cad4d4c4b15), in which I committed the dist folders I built locally.

I tried to run prepare asmcrypto.js inside node_package/asmcrypto.js inside this repo here, but it would not work for reasons I could not figure out. It would either fail to run or cause lots of unit tests to fail. The only way I made it work was by cloning https://github.com/openpgpjs/asmcrypto.js separately, and running npm install && npm run prepare there. I used node v12.4.0 and npm v6.9.0 for that.

[chesnokovilya]

It is difficult to determine exactly what causes this bug. Could you compare you setup with testing pipeline? Example here: https://travis-ci.org/openpgpjs/openpgpjs/jobs/556438060

[jpbochi]

@chesnokovilya My failure is on npm install itself, and I can see you're not doing anything special there.

One extra test I did was to do npm install --loglevel silly, and I saw this:

npm sill prepareGitDep 1> added 82 packages from 560 contributors and audited 132 packages in 9.179s
npm sill prepareGitDep found 2 vulnerabilities (1 moderate, 1 high)
npm sill prepareGitDep   run `npm audit fix` to fix them, or `npm audit` for details
npm sill prepareGitDep 
npm sill prepareGitDep 2> npm WARN install Usage of the `--dev` option is deprecated. Use `--only=dev` instead.
npm sill prepareGitDep npm WARN lifecycle [email protected]~prepare: cannot run in wd [email protected] node -r esm build.js (wd=/root/.npm/_cacache/tmp/git-clone-8b5fc547)
npm sill prepareGitDep 

Here's a link to the full build log on GitLab.

Right now, there are only two reasons I can think of to explain why your build passes, and mine doesn't:

1. In Travis CI, the build runs inside a custom Ubuntu 14.04.5 LTS machine, while my build run npm install inside official node docker images. They use Debian. This might affect permissions under /root/.npm/_cacache/tmp/git-clone-8b5fc547. Or maybe it's because Travis doesn't run npm as root. Or something else related to the OS.

🜚 docker run --rm node:12.5 cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

1. Your Travis CI build is caching the node_modules folder, and somehow it has a working version of that dependency.

[chesnokovilya]

Yeah, it seems to be permissions issue in your docker container:
https://docs.npmjs.com/misc/scripts#user
May be you could confirm this?

[jpbochi]

It was the permissions indeed. I did npm config set unsafe-perm true before npm install and everything passed.

For completeness, this are the new "npm sill prepareGitDep" logs after that config change:

npm sill prepareGitDep 1>
npm sill prepareGitDep > [email protected] prepare /Users/jp/src/jp/.npm/_cacache/tmp/git-clone-22b3fc50
npm sill prepareGitDep > node -r esm build.js
npm sill prepareGitDep
npm sill prepareGitDep Build complete
npm sill prepareGitDep added 82 packages from 560 contributors and audited 132 packages in 112.187s
npm sill prepareGitDep found 2 vulnerabilities (1 moderate, 1 high)
npm sill prepareGitDep   run `npm audit fix` to fix them, or `npm audit` for details
npm sill prepareGitDep
npm sill prepareGitDep 2> npm WARN install Usage of the `--dev` option is deprecated. Use `--only=dev` instead.
npm sill prepareGitDep Circular dependency: dist_es5/bignum/bignum.js -> dist_es5/bignum/extgcd.js -> dist_es5/bignum/bignum.js
npm sill prepareGitDep Circular dependency: dist_es8/bignum/bignum.js -> dist_es8/bignum/extgcd.js -> dist_es8/bignum/bignum.js
npm sill prepareGitDep

Nothing like expressing your thoughts out loud. I thought about the permission, but didn't pursue further. :)