Berry: [Bug] bin scripts don't work for packages from the GitHub registry

[rkistner]

• [ ] I'd be willing to implement a fix

Describe the bug

Packages installed from the github registry don't get their bin scripts listed in yarn.lock. They are then not listed under yarn bin, and cannot be run directly using yarn run. The scripts are still present in node_modules/.bin if the node-modules linker is used.

It appears to be an issue with the GitHub package registry API that doesn't include the bin field. However, it is quite unexpected that the bin scripts work with npm and yarn 1, but not yarn 2.

To Reproduce

Install this package: https://github.com/rkistner/yarn2-github-bin-script-issue/packages/321236. It is a public package, but you still need to authenticate with a github token.

1. Configure the scope:

npmScopes:
  rkistner:
    npmRegistryServer: "https://npm.pkg.github.com/rkistner/"
    npmAuthToken: "your-token-here"

1. Install the package: yarn add @rkistner/yarn2-github-bin-script-issue

2. yarn bin -> not listed.

3. Check yarn.lock -> no bin field under the @rkistner/yarn2-github-bin-script-issue package.

(Cannot reproduce with Sherlock, since an access token is required).

Environment if relevant (please complete the following information):

• OS: Tested on Ubuntu
• Node version: Tested on v12.6.0
• Yarn version: Tested on 2.2.1

Additional context

Add any other context about the problem here.

Downloading the tgz file gives a package.json file like this, with the bin field as expected:

{
  "name": "@rkistner/yarn2-github-bin-script-issue",
  "repository": "https://github.com/rkistner/yarn2-github-bin-script-issue",
  "version": "1.0.3",
  "main": "index.js",
  "license": "MIT",
  "bin": "./bin.js",
  "publishConfig": {
    "registry": "https://npm.pkg.github.com"
  },
  "files": ["bin.js"]
}

However, via the GitHub registry API on https://npm.pkg.github.com/@rkistner/yarn2-github-bin-script-issue (need access token once again), we get back the response below. Note the lack of a bin field.

I haven't been able to 100% confirm, but I _think_ this is what is used to generate the info in yarn.lock, which is causing the issue here.

{
  "name": "@rkistner/yarn2-github-bin-script-issue",
  "description": "",
  "dist-tags": {
    "latest": "1.0.3"
  },
  "versions": {
    // (other versions here)
    "1.0.3": {
      "_from": "",
      "_id": "@rkistner/[email protected]",
      "_nodeVersion": "12.6.0",
      "_npmUser": {},
      "_npmVersion": "6.14.2",
      "_shasum": "",
      "author": null,
      "bugs": {
        "url": "https://github.com/rkistner/yarn2-github-bin-script-issue/issues"
      },
      "dependencies": null,
      "description": "",
      "devDependencies": null,
      "dist": {
        "integrity": "sha512-t1+ihH6KSIJlI+glDV25dIABAuqnB7HBSrXiCrGGIhbSpMm6bbpF9djeAjCjcgelNgo2mMQ0f8FYpfSc5LRfkw==",
        "shasum": "c6d343b50cf757658c87663f38e5efd5fd4445e7",
        "tarball": "https://npm.pkg.github.com/download/@rkistner/yarn2-github-bin-script-issue/1.0.3/2616e60234d610b7aa9d44ab3442bcaae594a1eab916809a458200c8980812d7"
      },
      "gitHead": "28671a1c4eeb7a3aad7833e9d16ece684e6f3acb",
      "homepage": "https://github.com/rkistner/yarn2-github-bin-script-issue#readme",
      "license": "MIT",
      "main": "index.js",
      "name": "@rkistner/yarn2-github-bin-script-issue",
      "peerDependencies": null,
      "repository": {
        "type": "git",
        "url": "git+https://github.com/rkistner/yarn2-github-bin-script-issue.git"
      },
      "scripts": null,
      "version": "1.0.3"
    }
  },
  "time": {
    "1.0.0": "2020-07-20T14:53:55.000Z",
    "1.0.1": "2020-07-20T14:54:40.000Z",
    "1.0.2": "2020-07-20T14:54:56.000Z",
    "1.0.3": "2020-07-20T15:21:34.000Z"
  }
}

[arcanis]

cc @phanatic

The lockfile is generated based on the metadata returned by the registry. Otherwise, we'd need to fetch all the tarballs during the resolution, which would be a significant regression. It worked in the v1 because we didn't cache the bins in the lockfile, but that's not an option anymore since we now keep the packages in their archives as much as possible. If we were to always re-read the bin fields from there, yarn run would be significantly slower.

[rkistner]

Thanks @arcanis, makes sense to architect it this way. One thing that may help on Yarn's side is to detect mismatches like these and warn the user, since it's quite hard to debug. Not sure whether it's worth it though, especially if it gets fixed on GitHub's side.

I've also contacted GitHub support, so they are aware of the issue. I'll post here if there is any update from their side.

[yarnbot]

Hi! 👋

This issue looks stale, and doesn't feature the reproducible label - which implies that you didn't provide a working reproduction using Sherlock. As a result, it'll be closed in a few days unless a maintainer explicitly vouches for it or you edit your first post to include a formal reproduction (you can use the playground for that).

Note that we require Sherlock reproductions for long-lived issues (rather than standalone git repositories or similar) because we're a small team. Sherlock gives us the ability to check which bugs are still affecting the master branch at any given point, and decreases the amount of code we need to run on our own machines (thus leading to faster bug resolutions). It helps us help you! 😃

If you absolutely cannot reproduce a bug on Sherlock (for example because it's a Windows-only issue), a maintainer will have to manually add the upholded label. Thanks for helping us triaging our repository! 🌟

[brummelte]

They don't work for private packages hosted on a GitLab registry either:

https://gitlab.com/gitlab-org/gitlab/-/issues/275992

[jagu-sayan]

Workaround:

# With `nodeLinker: node-modules` in `.yarnrc.yml` file
yarn exec ./node_modules/.bin/<your command>