Renovate: Support lerna with yarn 2 (berry)

Created on 14 Nov 2020  路  13Comments  路  Source: renovatebot/renovate

What Renovate type, platform and version are you using?

Hosted App, on github

Describe the bug

The yarn install in post-update is broken:
https://github.com/renovatebot/renovate/blob/master/lib/manager/npm/post-update/yarn.ts#L53-L60

as renovate is trying to run: npm i -g yarn@'>= 2.2.0'

https://github.com/christophehurpeau/nightingale/pull/95#issuecomment-727179668

Relevant debug logs

DEBUG: Executing command(branch="renovate/pin-dependencies")
{
  "command": "docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/christophehurpeau/nightingale\":\"/mnt/renovate/gh/christophehurpeau/nightingale\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -w \"/mnt/renovate/gh/christophehurpeau/nightingale\" docker.io/renovate/node:15.2.0 bash -l -c \"npm i -g yarn@'>= 2.2.0' && sed -i 's/ steps,/ steps.slice(0,1),/' /home/ubuntu/.npm-global/lib/node_modules/yarn/lib/cli.js && npm i -g [email protected] && yarn install --ignore-scripts --ignore-engines --ignore-platform && lerna bootstrap --no-ci --ignore-scripts -- --ignore-scripts --ignore-engines --ignore-platform\""
}
DEBUG: lock file error(branch="renovate/pin-dependencies")
{
  "cmd": [
    "yarn install --ignore-scripts --ignore-engines --ignore-platform",
    "lerna bootstrap --no-ci --ignore-scripts -- --ignore-scripts --ignore-engines --ignore-platform"
  ],
  "err": {
    "killed": false,
    "code": 1,
    "signal": null,
    "cmd": "docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/christophehurpeau/nightingale\":\"/mnt/renovate/gh/christophehurpeau/nightingale\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -w \"/mnt/renovate/gh/christophehurpeau/nightingale\" docker.io/renovate/node:15.2.0 bash -l -c \"npm i -g yarn@'>= 2.2.0' && sed -i 's/ steps,/ steps.slice(0,1),/' /home/ubuntu/.npm-global/lib/node_modules/yarn/lib/cli.js && npm i -g [email protected] && yarn install --ignore-scripts --ignore-engines --ignore-platform && lerna bootstrap --no-ci --ignore-scripts -- --ignore-scripts --ignore-engines --ignore-platform\"",
    "stdout": "",
    "stderr": "npm ERR! code ETARGET\nnpm ERR! notarget No matching version found for yarn@>= 2.2.0.\nnpm ERR! notarget In most cases you or one of your dependencies are requesting\nnpm ERR! notarget a package version that doesn't exist.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-11-14T10_25_13_983Z-debug.log\n",
    "message": "Command failed: docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/christophehurpeau/nightingale\":\"/mnt/renovate/gh/christophehurpeau/nightingale\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -w \"/mnt/renovate/gh/christophehurpeau/nightingale\" docker.io/renovate/node:15.2.0 bash -l -c \"npm i -g yarn@'>= 2.2.0' && sed -i 's/ steps,/ steps.slice(0,1),/' /home/ubuntu/.npm-global/lib/node_modules/yarn/lib/cli.js && npm i -g [email protected] && yarn install --ignore-scripts --ignore-engines --ignore-platform && lerna bootstrap --no-ci --ignore-scripts -- --ignore-scripts --ignore-engines --ignore-platform\"\nnpm ERR! code ETARGET\nnpm ERR! notarget No matching version found for yarn@>= 2.2.0.\nnpm ERR! notarget In most cases you or one of your dependencies are requesting\nnpm ERR! notarget a package version that doesn't exist.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-11-14T10_25_13_983Z-debug.log\n",
    "stack": "Error: Command failed: docker run --rm --name=renovate_node --label=renovate_child -v \"/mnt/renovate/gh/christophehurpeau/nightingale\":\"/mnt/renovate/gh/christophehurpeau/nightingale\" -v \"/tmp/renovate-cache\":\"/tmp/renovate-cache\" -v \"/home/ubuntu/.npmrc\":\"/home/ubuntu/.npmrc\" -e NPM_CONFIG_CACHE -e npm_config_store -w \"/mnt/renovate/gh/christophehurpeau/nightingale\" docker.io/renovate/node:15.2.0 bash -l -c \"npm i -g yarn@'>= 2.2.0' && sed -i 's/ steps,/ steps.slice(0,1),/' /home/ubuntu/.npm-global/lib/node_modules/yarn/lib/cli.js && npm i -g [email protected] && yarn install --ignore-scripts --ignore-engines --ignore-platform && lerna bootstrap --no-ci --ignore-scripts -- --ignore-scripts --ignore-engines --ignore-platform\"\nnpm ERR! code ETARGET\nnpm ERR! notarget No matching version found for yarn@>= 2.2.0.\nnpm ERR! notarget In most cases you or one of your dependencies are requesting\nnpm ERR! notarget a package version that doesn't exist.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR!     /tmp/renovate-cache/others/npm/_logs/2020-11-14T10_25_13_983Z-debug.log\n\n    at ChildProcess.exithandler (child_process.js:303:12)\n    at ChildProcess.emit (events.js:311:20)\n    at ChildProcess.EventEmitter.emit (domain.js:482:12)\n    at maybeClose (internal/child_process.js:1021:16)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)"
  },
  "type": "lerna",
  "lernaClient": "yarn"
}

To Reproduce

Additional context

npm feature
Source
picture christophehurpeau

All 13 comments

What in particular is wrong? Eg is it the quote marks?

rarkins picture rarkins on 14 Nov 2020

When minYarnVersion === '>= 2.2.0', it should not enter in the if:

    if (isYarn1 && minYarnVersion) {
      installYarn += `@${quote(yarnCompatibility)}`;
    }

(https://github.com/renovatebot/renovate/blob/master/lib/manager/npm/post-update/yarn.ts#L53-L60)
but it does, as it tries to run npm i -g yarn@'>= 2.2.0' but should run npm i -g yarn instead

christophehurpeau picture christophehurpeau on 14 Nov 2020

So your repo is on yarn 1 and the yarn 2 install is a complete mistake?

rarkins picture rarkins on 14 Nov 2020

in renovate, yarn 1 is only used to read .yarnrc.yml and run yarn 2 (see https://github.com/renovatebot/renovate/pull/7183)

christophehurpeau picture christophehurpeau on 14 Nov 2020

It's because Renovate doesn't have a similar logic for Lerna. In fact, Renovate doesn't support Yarn 2 with Lerna yet:
https://github.com/renovatebot/renovate/blob/93711e1463d22962bf2060c9307277ca3a1b3f85/lib/manager/npm/post-update/lerna.ts#L47-L56

Could you change the title to Support Lerna with Yarn 2?

ylemkimon picture ylemkimon on 14 Nov 2020
👍1

BTW, it seems Lerna is not fully compatible with Yarn 2: https://github.com/lerna/lerna/issues/2449, e.g., passes the removed argument --ignore-scripts if --ignore-scripts is specified with Lerna, so it's impossible to not run lifecycle scripts. If it's fixed, the following patch (licensed under the same license with this repo) would work:

From 69ae4514cbb94d288b0ce62d54821f3fec3fe293 Mon Sep 17 00:00:00 2001
From: ylemkimon
Date: Sun, 15 Nov 2020 03:15:11 +0900
Subject: [PATCH 1/1] fix(npm): support Lerna with Yarn 2

---
 .../__snapshots__/lerna.spec.ts.snap          | 135 +++++++++++++++++-
 lib/manager/npm/post-update/lerna.spec.ts     |  58 ++++----
 lib/manager/npm/post-update/lerna.ts          |  35 +++--
 3 files changed, 188 insertions(+), 40 deletions(-)

diff --git a/lib/manager/npm/post-update/__snapshots__/lerna.spec.ts.snap b/lib/manager/npm/post-update/__snapshots__/lerna.spec.ts.snap
index 9711c9c84..92bb51870 100644
--- a/lib/manager/npm/post-update/__snapshots__/lerna.spec.ts.snap
+++ b/lib/manager/npm/post-update/__snapshots__/lerna.spec.ts.snap
@@ -1,5 +1,6 @@
 // Jest Snapshot v1, https://goo.gl/fbAQLP
-exports[`manager/npm/post-update/lerna generateLockFiles() allows scripts for trust level high 1`] = `
+
+exports[`manager/npm/post-update/lerna generateLockFiles() allows scripts for trust level high with npm v^6.0.0 1`] = `
 Array [
   Object {
     "cmd": "npm install  --no-audit --package-lock-only",
@@ -40,6 +41,88 @@ Array [
 ]
 `;

+exports[`manager/npm/post-update/lerna generateLockFiles() allows scripts for trust level high with yarn v^1.10.0 1`] = `
+Array [
+  Object {
+    "cmd": "yarn install --ignore-engines --ignore-platform",
+    "options": Object {
+      "cwd": "some-dir",
+      "encoding": "utf-8",
+      "env": Object {
+        "HOME": "/home/user",
+        "HTTPS_PROXY": "https://example.com",
+        "HTTP_PROXY": "http://example.com",
+        "LANG": "en_US.UTF-8",
+        "LC_ALL": "en_US",
+        "NO_PROXY": "localhost",
+        "PATH": "/tmp/path",
+      },
+      "maxBuffer": 10485760,
+      "timeout": 900000,
+    },
+  },
+  Object {
+    "cmd": "lerna bootstrap --no-ci -- --ignore-engines --ignore-platform",
+    "options": Object {
+      "cwd": "some-dir",
+      "encoding": "utf-8",
+      "env": Object {
+        "HOME": "/home/user",
+        "HTTPS_PROXY": "https://example.com",
+        "HTTP_PROXY": "http://example.com",
+        "LANG": "en_US.UTF-8",
+        "LC_ALL": "en_US",
+        "NO_PROXY": "localhost",
+        "PATH": "/tmp/path",
+      },
+      "maxBuffer": 10485760,
+      "timeout": 900000,
+    },
+  },
+]
+`;
+
+exports[`manager/npm/post-update/lerna generateLockFiles() allows scripts for trust level high with yarn v^2.0.0 1`] = `
+Array [
+  Object {
+    "cmd": "yarn install",
+    "options": Object {
+      "cwd": "some-dir",
+      "encoding": "utf-8",
+      "env": Object {
+        "HOME": "/home/user",
+        "HTTPS_PROXY": "https://example.com",
+        "HTTP_PROXY": "http://example.com",
+        "LANG": "en_US.UTF-8",
+        "LC_ALL": "en_US",
+        "NO_PROXY": "localhost",
+        "PATH": "/tmp/path",
+      },
+      "maxBuffer": 10485760,
+      "timeout": 900000,
+    },
+  },
+  Object {
+    "cmd": "lerna bootstrap --no-ci --",
+    "options": Object {
+      "cwd": "some-dir",
+      "encoding": "utf-8",
+      "env": Object {
+        "HOME": "/home/user",
+        "HTTPS_PROXY": "https://example.com",
+        "HTTP_PROXY": "http://example.com",
+        "LANG": "en_US.UTF-8",
+        "LC_ALL": "en_US",
+        "NO_PROXY": "localhost",
+        "PATH": "/tmp/path",
+      },
+      "maxBuffer": 10485760,
+      "timeout": 900000,
+    },
+  },
+]
+`;
+
 exports[`manager/npm/post-update/lerna generateLockFiles() defaults to latest if lerna version unspecified 1`] = `
 Array [
   Object {
@@ -122,10 +205,51 @@ Array [
 ]
 `;

-exports[`manager/npm/post-update/lerna generateLockFiles() generates yarn.lock files 1`] = `
+exports[`manager/npm/post-update/lerna generateLockFiles() generates yarn.lock files yarn v^1.10.0 1`] = `
+Array [
+  Object {
+    "cmd": "yarn install --ignore-engines --ignore-platform --ignore-scripts",
+    "options": Object {
+      "cwd": "some-dir",
+      "encoding": "utf-8",
+      "env": Object {
+        "HOME": "/home/user",
+        "HTTPS_PROXY": "https://example.com",
+        "HTTP_PROXY": "http://example.com",
+        "LANG": "en_US.UTF-8",
+        "LC_ALL": "en_US",
+        "NO_PROXY": "localhost",
+        "PATH": "/tmp/path",
+      },
+      "maxBuffer": 10485760,
+      "timeout": 900000,
+    },
+  },
+  Object {
+    "cmd": "lerna bootstrap --no-ci --ignore-scripts -- --ignore-engines --ignore-platform --ignore-scripts",
+    "options": Object {
+      "cwd": "some-dir",
+      "encoding": "utf-8",
+      "env": Object {
+        "HOME": "/home/user",
+        "HTTPS_PROXY": "https://example.com",
+        "HTTP_PROXY": "http://example.com",
+        "LANG": "en_US.UTF-8",
+        "LC_ALL": "en_US",
+        "NO_PROXY": "localhost",
+        "PATH": "/tmp/path",
+      },
+      "maxBuffer": 10485760,
+      "timeout": 900000,
+    },
+  },
+]
+`;
+
+exports[`manager/npm/post-update/lerna generateLockFiles() generates yarn.lock files yarn v^2.0.0 1`] = `
 Array [
   Object {
-    "cmd": "yarn install --ignore-scripts --ignore-engines --ignore-platform",
+    "cmd": "yarn install",
     "options": Object {
       "cwd": "some-dir",
       "encoding": "utf-8",
@@ -137,13 +261,14 @@ Array [
         "LC_ALL": "en_US",
         "NO_PROXY": "localhost",
         "PATH": "/tmp/path",
+        "YARN_ENABLE_SCRIPTS": "0",
       },
       "maxBuffer": 10485760,
       "timeout": 900000,
     },
   },
   Object {
-    "cmd": "lerna bootstrap --no-ci --ignore-scripts -- --ignore-scripts --ignore-engines --ignore-platform",
+    "cmd": "lerna bootstrap --no-ci --ignore-scripts --",
     "options": Object {
       "cwd": "some-dir",
       "encoding": "utf-8",
@@ -155,6 +280,7 @@ Array [
         "LC_ALL": "en_US",
         "NO_PROXY": "localhost",
         "PATH": "/tmp/path",
+        "YARN_ENABLE_SCRIPTS": "0",
       },
       "maxBuffer": 10485760,
       "timeout": 900000,
@@ -244,4 +370,3 @@ Array [
   },
 ]
 `;
-
diff --git a/lib/manager/npm/post-update/lerna.spec.ts b/lib/manager/npm/post-update/lerna.spec.ts
index ba8881a40..6e6e9c72f 100644
--- a/lib/manager/npm/post-update/lerna.spec.ts
+++ b/lib/manager/npm/post-update/lerna.spec.ts
@@ -70,17 +70,20 @@ describe(getName(__filename), () => {
       expect(res.error).toBe(false);
       expect(execSnapshots).toMatchSnapshot();
     });
-    it('generates yarn.lock files', async () => {
-      const execSnapshots = mockExecAll(exec);
-      const res = await lernaHelper.generateLockFiles(
-        lernaPkgFile('yarn'),
-        'some-dir',
-        { constraints: { yarn: '^1.10.0' } },
-        {}
-      );
-      expect(execSnapshots).toMatchSnapshot();
-      expect(res.error).toBe(false);
-    });
+    it.each([['^1.10.0'], ['^2.0.0']])(
+      'generates yarn.lock files yarn v%s',
+      async (yarnVersion) => {
+        const execSnapshots = mockExecAll(exec);
+        const res = await lernaHelper.generateLockFiles(
+          lernaPkgFile('yarn'),
+          'some-dir',
+          { constraints: { yarn: yarnVersion } },
+          {}
+        );
+        expect(execSnapshots).toMatchSnapshot();
+        expect(res.error).toBe(false);
+      }
+    );
     it('defaults to latest if lerna version unspecified', async () => {
       const execSnapshots = mockExecAll(exec);
       const res = await lernaHelper.generateLockFiles(
@@ -106,19 +109,26 @@ describe(getName(__filename), () => {
       expect(res.error).toBe(false);
       expect(execSnapshots).toMatchSnapshot();
     });
-    it('allows scripts for trust level high', async () => {
-      const execSnapshots = mockExecAll(exec);
-      global.trustLevel = 'high';
-      const res = await lernaHelper.generateLockFiles(
-        lernaPkgFile('npm'),
-        'some-dir',
-        {},
-        {}
-      );
-      delete global.trustLevel;
-      expect(res.error).toBe(false);
-      expect(execSnapshots).toMatchSnapshot();
-    });
+    it.each([
+      ['npm', '^6.0.0'],
+      ['yarn', '^1.10.0'],
+      ['yarn', '^2.0.0'],
+    ])(
+      'allows scripts for trust level high with %s v%s',
+      async (client, version) => {
+        const execSnapshots = mockExecAll(exec);
+        global.trustLevel = 'high';
+        const res = await lernaHelper.generateLockFiles(
+          lernaPkgFile(client),
+          'some-dir',
+          { constraints: { [client]: version } },
+          {}
+        );
+        delete global.trustLevel;
+        expect(res.error).toBe(false);
+        expect(execSnapshots).toMatchSnapshot();
+      }
+    );
   });

   describe('getLernaVersion()', () => {
diff --git a/lib/manager/npm/post-update/lerna.ts b/lib/manager/npm/post-update/lerna.ts
index bbb95de91..b27044db6 100644
--- a/lib/manager/npm/post-update/lerna.ts
+++ b/lib/manager/npm/post-update/lerna.ts
@@ -1,4 +1,4 @@
-import semver, { validRange } from 'semver';
+import semver, { minVersion, validRange } from 'semver';
 import { quote } from 'shlex';
 import { join } from 'upath';
 import { logger } from '../../../logger';
@@ -41,19 +41,35 @@ export async function generateLockFiles(
   logger.debug(`Spawning lerna with ${lernaClient} to create lock files`);
   const preCommands = [];
   const cmd = [];
+  const extraEnv: ExecOptions['extraEnv'] = {
+    NPM_CONFIG_CACHE: env.NPM_CONFIG_CACHE,
+    npm_config_store: env.npm_config_store,
+  };
   let cmdOptions = '';
   try {
     if (lernaClient === 'yarn') {
       let installYarn = 'npm i -g yarn';
       const yarnCompatibility = config.constraints?.yarn;
-      if (validRange(yarnCompatibility)) {
+      const minYarnVersion =
+        validRange(yarnCompatibility) && minVersion(yarnCompatibility);
+      const isYarn1 = !minYarnVersion || minYarnVersion.major === 1;
+      if (isYarn1 && validRange(yarnCompatibility)) {
         installYarn += `@${quote(yarnCompatibility)}`;
       }
       preCommands.push(installYarn);
-      if (skipInstalls !== false) {
-        preCommands.push(optimizeCommand);
+      if (isYarn1) {
+        if (skipInstalls !== false) {
+          preCommands.push(optimizeCommand);
+        }
+        cmdOptions = '--ignore-engines --ignore-platform';
+      }
+      if (global.trustLevel !== 'high' || config.ignoreScripts) {
+        if (isYarn1) {
+          cmdOptions += ' --ignore-scripts';
+        } else {
+          extraEnv.YARN_ENABLE_SCRIPTS = '0';
+        }
       }
-      cmdOptions = '--ignore-scripts --ignore-engines --ignore-platform';
     } else if (lernaClient === 'npm') {
       let installNpm = 'npm i -g npm';
       const npmCompatibility = config.constraints?.npm;
@@ -78,10 +94,7 @@ export async function generateLockFiles(
     const tagConstraint = await getNodeConstraint(config);
     const execOptions: ExecOptions = {
       cwd,
-      extraEnv: {
-        NPM_CONFIG_CACHE: env.NPM_CONFIG_CACHE,
-        npm_config_store: env.npm_config_store,
-      },
+      extraEnv,
       docker: {
         image: 'renovate/node',
         tagScheme: 'npm',
@@ -101,11 +114,11 @@ export async function generateLockFiles(
       const homeNpmrc = join(homeDir, '.npmrc');
       execOptions.docker.volumes = [[homeNpmrc, '/home/ubuntu/.npmrc']];
     }
-    cmd.push(`${lernaClient} install ${cmdOptions}`);
+    cmd.push(`${lernaClient} install ${cmdOptions}`.trim());
     const lernaVersion = getLernaVersion(lernaPackageFile);
     logger.debug('Using lerna version ' + lernaVersion);
     preCommands.push(`npm i -g lerna@${quote(lernaVersion)}`);
-    cmd.push(lernaCommand);
+    cmd.push(lernaCommand.trim());
     await exec(cmd, execOptions);
   } catch (err) /* istanbul ignore next */ {
     logger.debug(
-- 
2.29.2

ylemkimon picture ylemkimon on 14 Nov 2020

yes but that's weird, I don't use or need lerna bootstrap, this should be like any other project I think

christophehurpeau picture christophehurpeau on 15 Nov 2020

I found a workaround: I deleted lerna.json and used "lerna" key in package.json. Renovate seems to detect lerna by just looking for lerna.json. As I use only lerna to release and for run command in scripts, this works for me.

christophehurpeau picture christophehurpeau on 15 Nov 2020

We probably need to close that loophole :D

Question is: are there any reasons for running lerna bootstrap when Yarn is used, or should we simplify and just call yarn?

rarkins picture rarkins on 15 Nov 2020
👍1

@rarkins I would say simplify when yarn is used and in lerna's config useWorkspaces is true: https://github.com/lerna/lerna/blob/main/commands/bootstrap/index.js#L181-L183

christophehurpeau picture christophehurpeau on 15 Nov 2020

@christophehurpeau good idea. So in that case if we detect Lerna but useWorkspaces is true then we'll treat it as if it's Yarn and ignore Lerna

rarkins picture rarkins on 15 Nov 2020
👍1

I'm not sure if this should be labeled bug or feature, I'll let @rarkins decide on that.
Also you might want to put it on the Renovate project board. :wink:

HonkingGoose picture HonkingGoose on 15 Nov 2020

:tada: This issue has been resolved in version 23.95.0 :tada:

The release is available on:

Your semantic-release bot :package::rocket:

renovate-release picture renovate-release on 7 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ghost picture ghost  路  3Comments
rarkins picture rarkins  路  3Comments
ikatyang picture ikatyang  路  4Comments
jycouet picture jycouet  路  4Comments
Flydiverny picture Flydiverny  路  4Comments