Cli: [BUG] EPERM issues since v6.11.0

[rowanoulton]

What / Why

We recently upgraded from npm 6.0.1 to 6.12.0 and several of our engineers are encountering persistent EPERM issues when they run npm install.

From all angles the folders in question have the right permissions set, and we've ruled out nvm as a possible reason for this failure too. No combination of uninstalling and reinstalling or cache clearing has resolved it.

Whenever they run npm install, they get output like this:

code EPERM
npm ERR! syscall spawn
npm ERR! errno EPERM
npm ERR! Error: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at Object.spawn (child_process.js:540:9)
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24
npm ERR!  OperationalError: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at Object.spawn (child_process.js:540:9)
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24
npm ERR!     at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!   cause: Error: spawn EPERM
npm ERR!       at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!       at Object.spawn (child_process.js:540:9)
npm ERR!       at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24
npm ERR!       at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!     errno: 'EPERM',
npm ERR!     code: 'EPERM',
npm ERR!     syscall: 'spawn'
npm ERR!   },
npm ERR!   stack: 'Error: spawn EPERM\n' +
npm ERR!     '    at ChildProcess.spawn (internal/child_process.js:394:11)\n' +
npm ERR!     '    at Object.spawn (child_process.js:540:9)\n' +
npm ERR!     '    at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:216:24\n' +
npm ERR!     '    at /Users/REDACTED/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24',
npm ERR!   errno: 'EPERM',
npm ERR!   code: 'EPERM',
npm ERR!   syscall: 'spawn',
npm ERR!   parent: 'webapp'
npm ERR! }
npm ERR! 
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR! 
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.
npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/REDACTED/.npm/_logs/2019-11-12T21_45_00_580Z-debug.log

I walked back the version of npm until 6.10.3, where installs work again. The mention of pacote in the above logs seems suspicious when you compare it to the release notes for 6.11.0:

dc8f9e52f [email protected]: Infer the ownership of all unpacked files in node_modules, so that we never have user-owned files in root-owned folders, or root-owned files in user-owned folders. (@isaacs)

Note that upgrading to 6.13.0 does not fix the problem.

Also note that we have some private git+ssh urls in our package.json

Where

npm public registry

How

Current Behavior

Install fails with above error output, EPERM

Steps to Reproduce

• npm i -g [email protected] && npm install

Expected Behavior

Install works as before

[isaacs]

It looks like the issue is involving a git repo dependency that has a prepare script of some kind.

Can you share some details about your dependencies? I know that they're private git deps, but how are you referencing them? What are they doing?

[rowanoulton]

Hi @isaacs, thanks for the quick response.

In addition to public npm registry dependencies, we have two types of private modules:

1. Internal modules we publish to our Artifactory registry. These are namespaced and configured with .npmrc. They take the format of @company-name/module
2. Dependencies on repositories from our Github Enterprise instance, of the format git+ssh://[email protected]:company/repo-name.git#<full sha1 commit hash>

Does that answer your question?

[doochik]

I think this issue is related to https://github.com/npm/cli/issues/286

Since v6.11 npm calls git with the directory owner user
https://github.com/npm/pacote/blob/v9.5.8/lib/util/git.js#L216

Examples:
I'm user1 and group1. Run npm install in directory /workdir which is owned by user2 and group2. npm runs git with uid=user2 and gid=group2 and fails to start with Error: spawn EPERM. Because user1 hasn't rights to start process for user2.

Another example is for docker. I have docker container for build and mount /workdir inside. Inside container /workdir is owned by unknown user. And we have error

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git@<private server>/admin-template.git
npm ERR! 
npm ERR! No user exists for uid 1000
npm ERR! fatal: Could not read from remote repository.

[isaacs]

Ok, thanks. So this isn't about prepare scripts, it's just failing to run git because the uid/gid is getting set to the owner of the folder that it's unpacking to. The private-ness of the git repo is a red herring in this case, I think, because what you're describing seems to be about spawning git in the first place, regardless of the repo you're fetching from.

I think the problem here is that it's mixing "folder owner uid" with "euid of child process" under the same uid/gid config param.

We'll need to dig into it a bit more, because if we just roll that back, we'll be back to the state of checking out root-owned files in user-owned locations. Setting an effective uid on the spawned process does make the files default to being owned by that user, which is nice, but we should be skipping that unless the current user is root.

So, it might _still_ fail if the user1 doesn't have permission to write to the folders owned by user2, but I think that would've been a problem before 6.11 as well. I'm assuming that /workdir has a mode of 775 rather than 755?

[isaacs]

Note (for myself, mostly) that means you'll end up with files being owned by user2 in a folder owned by user1, but since the files checked out _are_ owned by user2, then user2 should have permission to chown them over to user1.

[CNSKnight]

+1 npm 10.-12.

$ npm install --save https://github.com/CNSKnight/mifterialize-tree.git
npm ERR! code EPERM
npm ERR! syscall spawn
npm ERR! errno EPERM
npm ERR! Error: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at spawn (child_process.js:540:9)
npm ERR!     at execFile (child_process.js:224:17)
npm ERR!     at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
npm ERR!     at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)
npm ERR!     at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)
npm ERR!     at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24
npm ERR!  OperationalError: spawn EPERM
npm ERR!     at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!     at spawn (child_process.js:540:9)
npm ERR!     at execFile (child_process.js:224:17)
npm ERR!     at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
npm ERR!     at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)
npm ERR!     at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)
npm ERR!     at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!   cause: Error: spawn EPERM
npm ERR!       at ChildProcess.spawn (internal/child_process.js:394:11)
npm ERR!       at spawn (child_process.js:540:9)
npm ERR!       at execFile (child_process.js:224:17)
npm ERR!       at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
npm ERR!       at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)
npm ERR!       at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)
npm ERR!       at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24 {
npm ERR!     errno: 'EPERM',
npm ERR!     code: 'EPERM',
npm ERR!     syscall: 'spawn'
npm ERR!   },
npm ERR!   stack: 'Error: spawn EPERM\n' +
npm ERR!     '    at ChildProcess.spawn (internal/child_process.js:394:11)\n' +
npm ERR!     '    at spawn (child_process.js:540:9)\n' +
npm ERR!     '    at execFile (child_process.js:224:17)\n' +
npm ERR!     '    at tryCatcher (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)\n' +
npm ERR!     '    at ret (eval at makeNodePromisifiedEval (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/bluebird/js/release/promisify.js:184:12), <anonymous>:16:23)\n' +
npm ERR!     '    at opts.retry.retries (/home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/pacote/lib/util/git.js:192:14)\n' +
npm ERR!     '    at /home/steeve/.nvm/versions/node/v12.13.0/lib/node_modules/npm/node_modules/promise-retry/index.js:29:24',
npm ERR!   errno: 'EPERM',
npm ERR!   code: 'EPERM',
npm ERR!   syscall: 'spawn',
npm ERR!   parent: 'ANCILLARIES'
npm ERR! }

[doochik]

I'm assuming that /workdir has a mode of 775 rather than 755?

Yes, 775. I have write access to workdir.
In docker I’m root and have access to whole fs

[bertrama]

I believe this is also related to issue #296 .

[sppatel]

I am seeing similar issues too. What is strange is that if I run npm install w/ the package locks removed, npm install succeeds with 6.12.0. At that point I will have fresh package-lock files. I then wipe node_modules and re-run npm install and it fails with the EPERM error.

If I roll back npm versions v10.16.2 (the version I was on previously) - no issues.

[sppatel]

Two questions on the above comment that I don't quite understand.

(1) Why is this not seen on an npm install w/o a package lock file.
(2) Why is this not more widespread and seen consistently between users? I could not get others to repro.

[SebCorbin]

I'm encountering this issue too in a directory created by docker (I think, it's been a while).

I've fixed that by changing the group of the directory to my default user group, all runs fine now

[bertrama]

One example way to reproduce:

• Installing a dependency from a git repository
• Running in linux or osx.
• The package directory’s group is not the user’s default group.
• The package directory has the setgid bit turned on (i.e. chmod g+s or 2775) such that the not-default-group is preserved when npm install creates new directories.

In my ubuntu environment, my default username/group might be bertrama/bertrama and I’m a member of a group cdrom. Steps for me to reproduce are:

mkdir -m 2775 eperm-test
chgrp cdrom eperm-test
cd eperm-test
npm init -y
npm add left-pad@git://github.com/left-pad/left-pad

I haven't tested, but you may be able to substitute chgrp cdrom eperm-test with chgrp "$(groups | awk '{print $2}')" eperm-test.

[isaacs]

Fixed in pacote 9.5.10, will be included in the next npm v6 release.

[sppatel]

Really appreciate the quick turn-around on this. Thank you. Is there any rough (and I mean rough 😄 ) estimate as to when the next release is targeted for?

[isaacs]

in about an hour

[isaacs]

Maybe less. Just waiting on CI, then there's about 15 minutes of manual steps to do. https://github.com/npm/cli/pull/577

[sppatel]

Confirmed fixed! TY!

[dmikalova]

I'm still getting this issue on NPM 6.14.4 with a container volume mount in Jenkins. Jenkins docker runs with a user that doesn't exist on the container, but the volume mounts are owned by that user, and I end up using root for unrelated reasons. NPM then fails because the owner of the dir doesn't exist. In brief, where user 11028 doesn't exist:

docker run -t -d -u 11028:100 --user root:root -w /tmp/jenkins-ea2356b0/workspace/x -v /tmp/jenkins-ea2356b0/workspace/x:/tmp/jenkins-ea2356b0/workspace/x:rw,z x
...
npm install
...
npm ERR! No user exists for uid 11028

NPM is the only thing that ends up having this behavior. Copying the dir to a temp dir and running let's me continue with the build.