Vscode-vsce: Created VSIX via `vsce package --yarn` does not include not-hoisted deep dependency

Created on 23 Mar 2020 · 7Comments · Source: microsoft/vscode-vsce

Think about a simple package like this, developed with yarn 1.22.4:

{
  "name": "vsce-yarn-lacked-dependency",
  "description": "example",
  "version": "0.0.1",
  "publisher": "yhatt",
  "repository": "https://github.com/yhatt/vsce-yarn-lacked-dependency",
  "engines": {
    "vscode": "^1.40.0"
  },
  "activationEvents": [
    "*"
  ],
  "main": "./extension.js",
  "devDependencies": {
    "vsce": "^1.74.0"
  },
  "dependencies": {
    "markdown-it": "^10.0.0"
  }
}

// extension.js
const md = require("markdown-it")();

module.exports = {
  activate: () => console.log(md.render("Hello, world!"))
};

The creation of extension through yarn vsce package --yarn will success, but will fail to activate in VS Code.

abstractExtensionService.ts:396 Activating extension 'yhatt.vsce-yarn-lacked-dependency' failed: Cannot find module 'entities/lib/maps/entities.json'
Require stack:
- /Users/yuki.hattori/.vscode/extensions/yhatt.vsce-yarn-lacked-dependency-0.0.1/node_modules/markdown-it/lib/common/entities.js
- /Users/yuki.hattori/.vscode/extensions/yhatt.vsce-yarn-lacked-dependency-0.0.1/node_modules/markdown-it/lib/common/utils.js
- /Users/yuki.hattori/.vscode/extensions/yhatt.vsce-yarn-lacked-dependency-0.0.1/node_modules/markdown-it/lib/index.js
- /Users/yuki.hattori/.vscode/extensions/yhatt.vsce-yarn-lacked-dependency-0.0.1/node_modules/markdown-it/index.js
- /Users/yuki.hattori/.vscode/extensions/yhatt.vsce-yarn-lacked-dependency-0.0.1/extension.js
- /Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/loader.js
- /Applications/Visual Studio Code.app/Contents/Resources/app/out/bootstrap-amd.js
- /Applications/Visual Studio Code.app/Contents/Resources/app/out/bootstrap-fork.js.

It would work correctly if created through npm with yarn vsce package.

I've compared the structure in both of VSIX archives, and noticed the VSIX from yarn has included only a dependency [email protected] hoisted from devDependenciess (vsce -> [email protected] -> [email protected]).

VSIX from yarn VSIX from npm

VSIX from yarn	VSIX from npm
. └── node_modules ├── argparse │ └── lib │ ├── action │ │ ├── append │ │ └── store │ ├── argument │ └── help ├── entities (1.1.2) │ ├── lib │ ├── maps │ └── test ├── linkify-it │ └── lib ├── markdown-it │ ├── bin │ ├── dist │ └── lib │ ├── common │ ├── helpers │ ├── presets │ ├── rules_block │ ├── rules_core │ └── rules_inline ├── mdurl ├── sprintf-js │ ├── demo │ ├── dist │ ├── src │ └── test └── uc.micro ├── categories │ ├── Cc │ ├── Cf │ ├── P │ └── Z └── properties └── Any	. └── node_modules ├── argparse │ └── lib │ ├── action │ │ ├── append │ │ └── store │ ├── argument │ └── help ├── linkify-it │ └── lib ├── markdown-it │ ├── bin │ ├── dist │ ├── lib │ │ ├── common │ │ ├── helpers │ │ ├── presets │ │ ├── rules_block │ │ ├── rules_core │ │ └── rules_inline │ └── node_modules │ └── entities (2.0.0) │ └── lib ├── mdurl ├── sprintf-js │ ├── demo │ ├── dist │ ├── src │ └── test └── uc.micro ├── categories │ ├── Cc │ ├── Cf │ ├── P │ └── Z └── properties └── Any

.
└── node_modules
    ├── argparse
    │   └── lib
    │       ├── action
    │       │   ├── append
    │       │   └── store
    │       ├── argument
    │       └── help
    ├── entities (1.1.2)
    │   ├── lib
    │   ├── maps
    │   └── test
    ├── linkify-it
    │   └── lib
    ├── markdown-it
    │   ├── bin
    │   ├── dist
    │   └── lib
    │       ├── common
    │       ├── helpers
    │       ├── presets
    │       ├── rules_block
    │       ├── rules_core
    │       └── rules_inline
    ├── mdurl
    ├── sprintf-js
    │   ├── demo
    │   ├── dist
    │   ├── src
    │   └── test
    └── uc.micro
        ├── categories
        │   ├── Cc
        │   ├── Cf
        │   ├── P
        │   └── Z
        └── properties
            └── Any

.
└── node_modules
    ├── argparse
    │   └── lib
    │       ├── action
    │       │   ├── append
    │       │   └── store
    │       ├── argument
    │       └── help
    ├── linkify-it
    │   └── lib
    ├── markdown-it
    │   ├── bin
    │   ├── dist
    │   ├── lib
    │   │   ├── common
    │   │   ├── helpers
    │   │   ├── presets
    │   │   ├── rules_block
    │   │   ├── rules_core
    │   │   └── rules_inline
    │   └── node_modules
    │       └── entities (2.0.0)
    │           └── lib
    ├── mdurl
    ├── sprintf-js
    │   ├── demo
    │   ├── dist
    │   ├── src
    │   └── test
    └── uc.micro
        ├── categories
        │   ├── Cc
        │   ├── Cf
        │   ├── P
        │   └── Z
        └── properties
            └── Any

Look at the difference of the structure about entities package.

The extension expects to use a not-hoisted [email protected] in dependencies ([email protected] -> [email protected]), not [email protected] in vsce's devDependency.

In fact, there is not-hoisted [email protected] in /node_modules/markdown-it/node_modules while development even if installed packages by yarn. I suppose the process of packaging via yarn has some wrong.

Workaround

An available workaround is just using npm by omit --yarn.

However, it has not a worth in a few case. The author of extension has to use --yarn if using resolutions field (only supported in yarn), for resolving some vulnerabilities in deep dependency. vsce package via npm would throw an error due to the different structure of node_modules between npm and yarn.

For example, the following is to fix a vulnerability in deep dependency of Puppeteer v2.1.0 by yarn.

{ "dependencies": { "puppeteer": "^2.1.0" }, "resolutions": { "**/extract-zip/mkdirp": "^0.5.3", } }

vsce package using npm will fail by the different structure.

$ vsce package ERROR Command failed: npm list --production --parseable --depth=99999 --loglevel=error npm ERR! missing: [email protected], required by [email protected]

Sometimes I've met this trouble in the extension developed by me: https://github.com/marp-team/marp-vscode/pull/35, https://github.com/marp-team/marp-vscode/pull/130#issuecomment-602164173

UPDATE: npm v7 has shipped with support for yarn.lock and can generate fully deterministic dependency tree. Just using npm v7 might not need to worry about incorrect packaging.

bug help wanted

Source

yhatt

👍9 👀2

All 7 comments

The same for me too. Extension cannot be package neither by yarn, nor by npm.

npm ERR! missing: @babel/[email protected], required by [email protected]
npm ERR! missing: @babel/[email protected], required by [email protected]
npm ERR! missing: @babel/[email protected], required by [email protected]

And so on.

YevhenKap on 4 Jun 2020

We faced the same issue with gitlab-vscode-extension, this is my writeup:

Problem

The correct npm.ts implementation discards module version and packages incorrect version of NPM module.

For our extension, this happens because we have two different versions of ajv in our dependencies. Production code uses v6 and test code v5. The vsce ls (and so vsce package) picks the dev dependency.

Example project

The steps that follow can reproduce the issue reliably on gitlab-vscode-exteions.

clone the extension [email protected]:gitlab-org/gitlab-vscode-extension.git
checkout git checkout v3.0.0
install dependencies yarn
observe that the production code depends on ajv@6 by running yarn list --prod
├─ [email protected] │ ├─ ajv@^6.5.5
vsce ls --yarn will say that we will package the module straight in node_modules
vsce ls --yarn | grep ajv ... node_modules/ajv/package.json
that's incorrect version:
sh cat node_modules/ajv/package.json | grep version "version": "5.5.2", cat node_modules/har-validator/node_modules/ajv/package.json | grep version "version": "6.12.2",

Possible fixes

The issue is in the flattening logic. When we remove the version and then "ignore" all other versions we are potentially using incorrect dependencies. The only way to prevent this is IMO honouring the dependency structure given by yarn and not attempting to flatten it.

This can result in very subtle and hard to debug issues. Or production incidents

viktomas on 26 Jun 2020

In case someone made the same mistake as me, I just fixed it like this:

yarn && vsce package --yarn

johnsoncodehk on 28 Sep 2020

👎2

How about always using npm instead of yarn for dependency resolution if vsce was using npm >= v7? npm v7 can generate fully deterministic dependency tree based on yarn.lock so can expect more reliable packaging.
https://blog.npmjs.org/post/621733939456933888/npm-v7-series-why-keep-package-lockjson

Someone may feel like a bit strange to use npm even if running vsce with --yarn, but it would be much better than encountering production incident as like as brought in Marp, GitLab, and Microsoft Azure Logic Apps.

yhatt on 16 Oct 2020

Let me write down I met this trouble in our extension marp-vscode again. How to reproduce:

git clone https://github.com/marp-team/marp-vscode.git
cd ./marp-vscode
git checkout 5f285dc
yarn && yarn package

The following error would output to developer console if tried to activate created VSIX.

abstractExtensionService.ts:717 Activating extension 'marp-team.marp-vscode' failed: Cannot find module 'cssesc'
Require stack:
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/postcss-selector-parser/dist/selectors/className.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/postcss-selector-parser/dist/parser.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/postcss-selector-parser/dist/processor.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/postcss-selector-parser/dist/index.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/postcss-minify-selectors/dist/index.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/node_modules/@marp-team/marp-core/lib/marp.js
- /Users/yhatt/.vscode-insiders/extensions/marp-team.marp-vscode-0.15.1/lib/extension.js
- /Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/vs/loader.js
- /Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/bootstrap-amd.js
- /Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/out/bootstrap-fork.js.
_logMessageInConsole @ abstractExtensionService.ts:663

It can fix by adding --no-yarn option to vsce command in package npm-script: marp-team/marp-vscode#178

yhatt on 19 Oct 2020

We have the same problem when publish Vetur.
We downgrade to old version for solve this problem at the end.

yoyo930021 on 8 Nov 2020

Any updates on this issue? I have the same problem here, downgrade to 1.76 still seems not to work.

roland-reed on 20 Nov 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

vsce package : ERROR Command failed: npm list --production --parseable --depth=99999 --loglevel=error

pmahend1 · 4Comments

The specified icon 'extension/icon.png' wasn't found in the extension.

true0r · 3Comments

Revert changes ignore file semantic

joaomoreno · 4Comments

Keyword list limit can be worked around by specifying "one, two, three" tags

joaomoreno · 4Comments

Support for scoped naming, e.g., @myscope/package-name?

vazexqi · 4Comments