Cli: [BUG] bin files are lost when installing on a v1 lockfile, and can only be retained by regenerating entire lock

[G-Rath]

Running npm i with a lock file that has lockfileVersion: 1 seems to not pull in the bin details for at least some packages, meaning the .bin folder is not generated.

If I remove both package-lock.json & node_modules, npm i generates the expected lock that has the bin property, but this results in dependencies being upgraded which means more work for me since that can have all kinds of knock on effects (eslint plugin changes, security, etc).

Current Behavior:

No .bin folder exists.

package-lock.json:

{
  "name": "myfolder",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "version": "1.0.0",
      "license": "ISC",
      "dependencies": {
        "typescript": "^4.0.3"
      }
    },
    "node_modules/typescript": {
      "version": "4.0.3",
      "resolved": "https://registry.npmjs.org/typescript/-/typescript-4.0.3.tgz",
      "integrity": "sha512-tEu6DGxGgRJPb/mVPIZ48e69xCn2yRmCgYmDugAVwmJ6o+0u1RI18eO7E7WBTLYLaEVVOhwQmcdhQHweux/WPg=="
    }
  },
  "dependencies": {
    "typescript": {
      "version": "4.0.3",
      "resolved": "https://registry.npmjs.org/typescript/-/typescript-4.0.3.tgz",
      "integrity": "sha512-tEu6DGxGgRJPb/mVPIZ48e69xCn2yRmCgYmDugAVwmJ6o+0u1RI18eO7E7WBTLYLaEVVOhwQmcdhQHweux/WPg=="
    }
  }
}

Expected Behavior:

The .bin folder exists, with tsc in it.

package-lock.json:

{
  "name": "myfolder",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "version": "1.0.0",
      "license": "ISC",
      "dependencies": {
        "typescript": "^4.0.3"
      }
    },
    "node_modules/typescript": {
      "version": "4.0.3",
      "resolved": "https://registry.npmjs.org/typescript/-/typescript-4.0.3.tgz",
      "integrity": "sha512-tEu6DGxGgRJPb/mVPIZ48e69xCn2yRmCgYmDugAVwmJ6o+0u1RI18eO7E7WBTLYLaEVVOhwQmcdhQHweux/WPg==",
      "bin": {
        "tsc": "bin/tsc",
        "tsserver": "bin/tsserver"
      },
      "engines": {
        "node": ">=4.2.0"
      }
    }
  },
  "dependencies": {
    "typescript": {
      "version": "4.0.3",
      "resolved": "https://registry.npmjs.org/typescript/-/typescript-4.0.3.tgz",
      "integrity": "sha512-tEu6DGxGgRJPb/mVPIZ48e69xCn2yRmCgYmDugAVwmJ6o+0u1RI18eO7E7WBTLYLaEVVOhwQmcdhQHweux/WPg=="
    }
  }
}

Steps To Reproduce:

Run the following in a new folder:

#!/usr/bin/env bash

npx 'npm@6' init -y
npx 'npm@6' install typescript

stat node_modules/.bin

npm i
npm ci

stat node_modules/.bin

Example:

Users/G-Rath/myfolder
❯ ../script.sh
Wrote to /c/Users/G-Rath/myfolder/package.json:

{
  "name": "myfolder",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {

    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}


npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN [email protected] No description
npm WARN [email protected] No repository field.

+ [email protected]

added 1 package from 1 contributor and audited 1 package in 3.3s
found 0 vulnerabilities

  File: node_modules/.bin
  Size: 512             Blocks: 0          IO Block: 512    directory
Device: eh/14d  Inode: 7599824377958213  Links: 1
Access: (0755/drwxr-xr-x)  Uid: ( 1000/  g-rath)   Gid: ( 1000/  g-rath)
Access: 2020-10-14 14:20:34.154173400 +1300
Modify: 2020-10-14 14:20:34.154173400 +1300
Change: 2020-10-14 14:20:34.154173400 +1300
 Birth: -

up to date, audited 1 package in 635ms


found 0 vulnerabilities

added 1 package, and audited 1 package in 3s

found 0 vulnerabilities
stat: cannot stat 'node_modules/.bin': No such file or directory

Environment:

• OS: Ubuntu 18.04 (via WSLv1, Windows 10)
• Node: 14.4.0
• npm: 7.0.0

[mdarse]

Encountered the same issue. A workaround for me was to trigger lock file migration using npx npm@7 dedupe instead.

EDIT: I did the dedupe that after the removal of the node_modules directory.

I did a comparison of both lock files (the one I get with dedupe & the failing one), it appears that in addition to the packages.*.bin properties that got lost, packages.*.engines are gone too. I hope this can give a hint to someone.

❯ node -v
v14.14.0
❯ npm -v
7.0.5

[mizukami234]

Same problem here.
I tried dedupe @mdarse suggested, but it didn't solve all packages.

It seems that some of pkgMetaKeys fields in shrinkwrap.js get different results between migrating and regenerating. I think fixDependencies might have a bug.

[mansona]

I've just come across this now, and running npm dedupe worked for me. Although when working with a colleague they didn't fix the issue by running npm dedupe and needed to delete node_modules and package-lock.json and reinstall 👍

Edit: that seemed to work for me locally but for me to get it to work in CI I needed to delete node_modules and package-lock.json and reinstall too

[mdarse]

@mizukami234 just to be clear, I did that after deleting node_modules (but not the lock file), my message missed that part.

[davidje13]

I'm trying to narrow this down. If I install an entirely new dependency which has bins, they get loaded correctly. But if I delete the whole node_modules folder and reinstall, all the previously-existing dependencies which have bins don't get loaded (but the new dependency's bins do). It looks like the package-lock.json only includes "bin" entries for the newly added packages, which I suspect is the problem.

The dedupe command does nothing for my case. The only workarounds I've found so far:

• destroy the package-lock entirely (though I can't do that because a bunch of other packages have broken recently! cough fsevents cough
• if possible, bump the version of the dependency which has no bin entries (worked for one of the dependencies that I had, but another one didn't have an update to apply)
• _downgrade_ the dependency then install then re-_upgrade_ it and install

Looks like the old version didn't bother recording "bin" data, so the migration doesn't have a way to translate it and just leaves it out, but the new version won't install binaries unless the "bin" section is present in package-lock.json (seems to be some aggressive caching).

I'd suggest the best fix would be to provide an _explicit_ "there are no binaries" record (e.g. a "bin" entry with an empty list), so that absence means "check this" rather than "do nothing".

[ruyadorno]

A word to folks that contributed to this fix, we fixed the problem at the origin of this issue so that the migration from lockfile v1 -> v2 will not drop the bin files info. That said, if you already ran into the problem and currently have an invalid package-lock v2 file a possible solution is to reinstate a v1 package-lock into place and make the migration again e.g:

# Make sure you got [email protected] or newer:
$ npm install -g npm@7

# npm@6 install:
$ npx npm@6 install

# npm@7 install to migrate package-lock file again:
$ npm install

# You should have a fixed-up package-lock.json file now :)