Renovate: npm v7 support

[JamieMagee]

What would you like Renovate to be able to do?

Support the newly released npm v7. The blog post Presenting v7.0.0 of the npm CLI goes over some of the new features and breaking changes.

I'd like to use this issue as a mega-issue to discuss and co-ordinate npm v7 related changes to Renovate.

Did you already have any implementation ideas?

Some of the key features that will require Renovate changes:

• [ ] package-lock.json v2

  I can't yet find a schema for this, but here's a good example

• [ ] Workspaces

  See RFC 0026

• [ ] npx

  npx has been completely rewritten to use the npm exec command. There are various changes in functionality, most noticeable being a prompt if the module you are trying to run is not yet installed.

  There are a few places we use npx, and we should make sure that we are calling npx non-interatively.

• [ ] Peer dependencies

  Not sure if this one affects us

  Automatically installing peer dependencies (while this feature is something we think is desirable new behavior, it does potentially break certain workflows).

This list may not be complete.

The new library, @npmcli/arborist is also quite interesting, and should make interacting with the node_modules tree easier in future.

Are there any workarounds or alternative ideas you've tried to avoid needing this feature?

Switch to yarn :joy:

Is this a feature you'd be interested in implementing yourself?

Yes, with assistance

[mccxiv]

I'm here to express my confusion because we're already receiving PRs from renovate with "lockfileVersion": 2.
Is this already implemented?

[rarkins]

@mccxiv it's not intended. Can you reproduce the problem in a public repo? Does your engines allow node 15?

[mccxiv]

I don't have a public repro for now

We use a monorepo on Gitlab. The top level package.json has:

"engines": {
    "node": ">= 10.0.0",
    "npm": ">= 5.6.0"
  },

The individual packages do not have an engines: field

The odd thing is some PRs don't use lockfile v2, and some do (see line diffs):

[Quramy]

we're already receiving PRs from renovate with "lockfileVersion": 2

@rarkins I have same issue.

Here is my reproducing repo: https://github.com/Quramy/npm-ts-workspaces-example/pull/3#issuecomment-716925358 (My repo uses NPM v7 workspaces feature too).

[FYI]

I found https://github.com/renovatebot/renovate/blob/master/lib/manager/npm/post-update/npm.ts#L30-L33

And I configured constraints params in order to tell that my NPM version is v7, but the bot still outputs update issue 🤔

{
  "extends": ["config:base"],
  "constraints": {
    "npm": "^7.0.0"
  }
}

[rarkins]

@Quramy what is your issue? You constraints say npm 7 so you're going to get lockfileversion 2.

[rarkins]

I can reproduce it. If you have an engines that allows node 15, and no npm constraint limiting <7, then Renovate will run npm install using Node 15 which defaults to npm 7. Example log:

DEBUG: Found compatible npm version (repository=renovate-tests/npm42, branch=renovate/chalk-2.x)
       "constraint": ">=12",
       "version": "15.0.1"
DEBUG: Resolved tag constraint (repository=renovate-tests/npm42, branch=renovate/chalk-2.x)
       "image": "docker.io/renovate/node",
       "tagConstraint": ">=12",
       "tagVersioning": "npm",
       "tag": "15.0.1"

[rarkins]

I've experimented with using an npm constraint (e.g. npm i npm@^6.0.0) but it doesn't seem to downgrade the npm version in node 15.

[rarkins]

If you add the following to your renovate.json then it succeeds in getting Renovate to use node 14:

  "constraints": {
    "node": "< 15.0.0"
  }

Unfortunately it's only succeeding due to what seems like a bug, but it can work for now and be forwards compatible for anyone who wants to support node 15 in their engines without running it during Renovate PRs.

[Quramy]

Sorry, the bot update failure in my repo was not Renovate issue. Probably because some of the dependencies couldn't be installed in the Node v15 environment. I misunderstood it as package-lock version issue :bow:

[rarkins]

Actually, the required config will be as following once #7561 is merged and live:

{
  "force": {
    "constraints": {
      "node": "< 15.0.0"
    }
  }
}

The force wrapper is necessary because otherwise the detected engines node constraint takes precedence.

[ylemkimon]

@rarkins

I've experimented with using an npm constraint (e.g. npm i npm@^6.0.0) but it doesn't seem to downgrade the npm version in node 15.

Do you mean by setting

{
  "constraints": {
    "npm": "^6.0.0"
  }
}

?

In that case, it would haven't worked due to the cached location of npm after installing npm:
https://github.com/renovatebot/renovate/blob/367e591c66cd071257966b05a1ca9a63849ab8df/lib/manager/npm/post-update/npm.ts#L30-L33

I opened a PR to fix the issue including automatically setting the constraints in #7700.

• [x] package-lock.json v2
  I can't yet find a schema for this, but here's a good example

[ylemkimon]

• [x] Workspaces
  See RFC 0026

Workspaces seem to work out of the box: https://github.com/ylemkimon/npm-workspace.

• [x] npx

  npx has been completely rewritten to use the npm exec command. There are various changes in functionality, most noticeable being a prompt if the module you are trying to run is not yet installed.

  There are a few places we use npx, and we should make sure that we are calling npx non-interatively.

The only place npx is used is: https://github.com/renovatebot/renovate/blob/75737805c804f69f427e0f8147c463836b1af448/lib/manager/npm/post-update/yarn.ts#L145-L156

[ylemkimon]

npx will also work out of the box, i.e., non-interactively, because we set CI=true:
https://github.com/renovatebot/renovate/blob/75737805c804f69f427e0f8147c463836b1af448/lib/manager/npm/post-update/yarn.ts#L64-L67
and npm 7 detects it: https://github.com/npm/cli/blob/a28aff769a77f127f371c31afcb9e9814722e5cd/lib/exec.js#L190-L191

I think we can say Renovate has full support for npm 7.

[stephenwade]

7700 adds support for npm 7 in Docker, but npm 6 is still used when run locally (npm i -g renovate). Is there any way to get Renovate to use npm 7 (or get it to use my globally installed npm instead of the bundled npm) when running locally?

EDIT: npm 6 is only used when in local development. Globally installed renovate will properly use your globally installed npm version.

[viceice]

Looks like this is a technical limitation, as yarn / npm always prefer local bin before global.

Maybe we need to migrate away from referencing the full npm package.

[stephenwade]

Renovate doesn't reference npm directly, but semantic-release does so it still ends up in the local bin folder.

[stephenwade]

Are there any config options to specify the path to npm?

[viceice]

Semantic release is a dev dependency and shouldn't be installed on production. 🤔

[viceice]

Are there any config options to specify the path to npm?

Nope

[stephenwade]

Okay, I found my issue. I was testing with a local dev build which is why I had npm@6 in that folder. When I run the globally installed renovate, it uses my globally installed npm version.

[ylemkimon]

@stephenwade Try using binarySource: 'docker' (--binary-source=docker).