Renovate: Feature request: Rushjs monorepo support

[KevinGrandon]

What would you like Renovate to be able to do?
We have recently moved to a monorepo managed by Rushjs. Rush is similar to yarn workspaces, and projects are defined by entries in a top-level rush.json file. Currently it seems that there's some manual work to update rush lockfiles, and it would be great if renovate did this out of the box.

Describe the solution you'd like
Enable renovate to work with rushjs monorepos. Perhaps custom lockfile logic could occur when detecting a project with a top-level rush.json file and a common/ folder.

Describe alternatives you've considered
We might be able to handle lockfile updating ourselves, with a custom CI runner that builds and updates the lockfile.

Additional context
We experienced this with the Fusion.js codebase after we migrated to a monorepo.
We saw that renovate was opening up Pull Requests, but not updating the common lockfile.

[rarkins]

Are you able to describe the commands you use if you were to manually make the updates? Ie so I can try to replicate them with Renovate.

For lerna for example if we detect a package file is part of the monorepo then we make all changes and then run “lerna bootstrap” after installing the version of lerna specified

[KevinGrandon]

Hello. Sorry, here's a more complete example:

• We clone the repository.
• Then if rush.json is there, we would install rush globally, and run rush install to make the initial installation.
• If we upgrade a dependency there is a rush add command, but it doesn't work with all package managers yet. So instead we typically modify package.json files, then run rush update --full.

[danwbyrne]

I am in a similar situation at https://github.com/neo-one-suite/neo-one, we recently made an effort to migrate to rushjs. Renovate does all of the package.json updates appropriately but we need to be able to apply the changes from rush update --full to the PR or else they won't ever pass a CI check where we use rush install (not allowed to update the shrinkwrap file).

[jlsjonas]

@danwbyrne note that you actually just need rush update, which will reduce the amount of conflicts you get between dependency updates

@rarkins rush update after updating the package.json files will update the applicable lockfile (rush supports different package managers, which one is being used is being abstracted behind rush)
In a CI environment you would use the following command to trigger rush node common/scripts/install-run-rush.js update.

The main caveat here, is the lockfile. This will cause a file conflict in every renovate PR as soon as one is merged; effectively being limited to 1 PR concurrency (in our testing using the above command as postUpgradeTask)

[rarkins]

@jlsjonas thanks for the info.

which one is being used is being abstracted behind rush

We would also need to detect/know that inside Renovate because we'd make sure we install the right manager and version prior to running rush.

In a CI environment you would use the following command to trigger rush node common/scripts/install-run-rush.js update.

Is this as an alternative to rush being globally installed or run via npx?

This will cause a file conflict in every renovate PR as soon as one is merged

I think "might" or "often" is more correct than "will"? i.e. if it's npm or yarn lockfiles then rush itself doesn't increase the chances of a conflict compared to the underlying manager, does it?

[jlsjonas]

We would also need to detect/know that inside Renovate because we'd make sure we install the right manager and version prior to running rush.

This can quite easily be detected based on the rush.json file, see https://rushjs.io/pages/maintainer/package_managers/#specifying-your-package-manager

Is this as an alternative to rush being globally installed or run via npx?

As an recommended alternative to being installed globally. npx is not an option as it would introduce a phantom node_modules folder (see https://rushjs.io/pages/maintainer/enabling_ci_builds/)

I think "might" or "often" is more correct than "will"? i.e. if it's npm or yarn lockfiles then rush itself doesn't increase the chances of a conflict compared to the underlying manager, does it?

It doesn't increase chances of conflicts indeed, however what we're experiencing (at least on Bitbucket Server/Stash) is that PR's already receive a _conflict_ stamp (at least on the lockfile) if it's changed, even if there's no line-level conflicts. This might be an issue with BBS more than anything though.

[rarkins]

This can quite easily be detected based on the rush.json file

What's the fallback if none of the lines are uncommented? Or do you think it's a reasonable requirement that Renovate users must specify one?

BTW is it mandatory that rush projects include common/scripts/install-run-rush.js ?

This might be an issue with BBS more than anything though.

Sounds like it could be. Ideally it would only conflict if there are line-level conflicts.

[jlsjonas]

What's the fallback if none of the lines are uncommented? Or do you think it's a reasonable requirement that Renovate users must specify one?

From their getting started:

Choose a package manager: The template defaults to using PNPM, but you can also use NPM or Yarn. See NPM vs PNPM vs Yarn for guidance.

The rush.json file is generated as part of the rush init process, thus it will already contain one of the lines. Having neither should be considered an invalid configuration.

BTW is it _mandatory_ that rush projects include common/scripts/install-run-rush.js ?

Again from their docs:

All Rush repos come with a script common/scripts/install-run-rush.js

So, yes.

Sounds like it could be. Ideally it would only conflict if there are line-level conflicts.

Definitely

[mmkal]

@rarkins was there any movement on this? Would the team be open to a PR adding basic rush support? I'd be happy to chip in, although I don't have a good sense of how big a lift it would be 🙃

My take on pnpm vs npm vs yarn with rush: I think it would be fine to _only_ support rush_x_pnpm at least initially. The default rush setup is with pnpm, and it looks to be the rush development team's preferred package manager. So I suspect the majority of rush monorepos are using pnpm, and there'd be significant value in having those ones play nice with renovate. In any case, pnpm would make sense to tackle first.

Some more observations that may be relevant to a renovate-rush integration:

• rush validates that exactly one of pnpmVersion, npmVersion and yarnVersion is specified (two => error, zero => error). There is no default.
• Rush projects are bootstrapped to include common/scripts/install-run-rush.js, and if the file is edited or removed, it is regenerated by the next rush install. Any rush repo without these scripts is invalid - and isn't really something renovate needs to worry about - rush will enforce.
• rush.json can be queried for the list of projects. In fact, it looks like @microsoft/rush-lib has a helper for loading a strongly-typed version which knows how to search for the rush.json file:

import {RushConfiguration} from '@microsoft/rush-lib'

const rushConfig = RushConfiguration.loadFromDefaultLocation()
rushConfig.projects.forEach(p => console.log(p.packageName, p.projectFolder)

• The RushConfugration class also has a method for finding the lock file: rushConfig.getCommittedShrinkwrapFilename()
• Editing any of the project's package.json files, then running rush install updates the lockfile automatically, which could make things easier for configurations other than those requesting lockfile-only changes.
• Rush and lerna aren't technically incompatible. The repo I've been testing rush on still has a lerna.json file, mostly for the ability to run lerna exec locally.

CC @octogonz who might correct me on some of the above points. Those are things I've noticed using rush on a public repo: https://github.com/mmkal/ts.

~One other thought: it may be possible to have a very bare-bones implementation which requires rush _and_ lerna. The leaf packages could be bumped in the established way (somehow avoiding lockfile changes), then rush update run at the end to update rush's lockfile.~ (removed - turns out renovate doesn't rely on lerna.json to discover projects)

[rarkins]

I would definitely be open to a PR, as we haven't moved this forward yet. Feel free to ping us here in this issue if/when you uncover any unforeseen decisions needing made.

The big challenge is that npm support is already a bit messy due to npm, yarn, pnpm and lerna all essentially being "package managers" for package.json files. It would be nice if we could classify them as separate managers internally but for now they all fall under lib/manager/npm/.

Would rush be like lerna in the sense that if we detect the presence of Rush then we use it instead of npm/pnpm/yarn?

[Stuk]

Having native support would be great!

I've set up Renovate with Rush+pnpm with a config similar to:

    "postUpgradeTasks": {
        "commands": [
            // ignore post-install scripts that might not work in the Renovate runner
            "echo 'ignore-scripts=true' >> common/config/rush/.npmrc",
            // Playwright caches browsers in the home directory, which doesn't work on Renovate
            "env PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 node common/scripts/install-run-rush.js update"
        ],
        "fileFilters": [
            "common/config/rush/browser-approved-packages.json",
            "common/config/rush/nonbrowser-approved-packages.json",
            "common/config/rush/repo-state.json",
            "common/config/rush/pnpm-lock.yaml"
        ]
    }

and it seems to work so far.

The ignore-scripts and Playwright config is specific to our setup, but I thought I'd document here in case it's useful to others.

[mmkal]

@Stuk this is great, thank you! I didn't know about postUpgradeTasks. Did you have to do anything special to allow renovate to find the package.json files? My assumption was that wouldn't work, since usually they're discovered by reading rush.json.

[viceice]

Renovate searches for all package.json files in the repo. We only filter some test folders with config:base preset

[mmkal]

@rarkins @viceice do postUpgradeTasks run on the renovate app, or only self-hosted? In the docs it says they only run when trustLevel is high, but that's a self-hosted configuration only.

I ask because I'm seeing renovate PRs getting errors along the lines of You need to run "rush update". And rush update is included in postUpgradeTasks. Example config: https://github.com/mmkal/ts/blob/main/renovate.json. Dashboard job logs - I can't find any logs indicating that the node common/scripts/install-run-rush update call was made: https://app.renovatebot.com/dashboard#github/mmkal/ts/235504626

[rarkins]

They don't run in the hosted app

[octogonz]

Renovate searches for all package.json files in the repo. We only filter some test folders with config:base preset

This is probably okay. It's not the Rush way, but it's probably close enough to be accurate.

My take on pnpm vs npm vs yarn with rush: I think it would be fine to _only_ support rush_x_pnpm at least initially.

This is a reasonable bet to make. It might not matter too much however, since the rush update front end abstracts away many of the differences between package managers.

One other thought: it may be possible to have a very bare-bones implementation which requires rush _and_ lerna.

This doesn't make sense to me. It would work for Lerna folder layouts that have been converted to Rush, but it probably would not work for a typical Rush folder layout.

Would rush be like lerna in the sense that if we detect the presence of Rush then we use it instead of npm/pnpm/yarn?

This is the main point. For a single repo, your algorithm might be like this:

1. Edit package.json
2. Run npm install
3. Commit the result

Whereas for a Rush repo, invoking npm install or pnpm install is not going to work. Your bot MUST do it like this:

1. Edit all the package.json files that you find
2. Run ./common/scripts/install-run-rush.js update. This command has no dependencies and should work in pretty much any Node.js environment
3. Commit the result, which may include other Rush config files that got adjusted

[rarkins]

In rush is there the concept of "internal" packages that Renovate should ignore by default? And if so then is it like Lerna and Yarn Workspaces where we should just ignore any internal package dependency where we find the same name in a package.json?

[octogonz]

Architecturally, the design is prescriptive: the Rush projects are enumerated explicitly in Rush's config files (mainly rush.json) and all other package.json files can be assumed to be test assets, or deadwood files from a previous build, or some other role that should probably be ignored. Rush never uses wildcards to discover projects because scanning folders is inefficient, plus a CI machine will often preserve leftover files from a previous build so that caches can be reused.

But this is a technical detail. Renovate would probably do fine to use a simple glob to find package.json files and a simple .gitignore type filter to allow users to manually exclude files that got processed incorrectly. It's not ideal, but it's a much simpler contract to support, particularly if you want to be compatible with many different versions of Rush.

[octogonz]

Ohhh wait a sec... I think you're actually asking about local project references (where the a dependency gets symlinked to another project in the repo, rather than being installed from NPM).

Yes, Rush does this. There are two models. The installation model we're moving towards is PNPM workspaces model, which use the workspace: specifier like in this example:

libraries/stream-collator/package.json

  . . .
  "devDependencies": {
    "@microsoft/rush-stack-compiler-3.5": "workspace:*",
    "@types/heft-jest": "1.0.1",
    "@types/node": "10.17.13",
    "@rushstack/eslint-config": "workspace:*",
    "@rushstack/heft": "workspace:*"
  }
  . . .

These are easy to detect and ignore. This model is opt-in as an experiment currently, and will become the default in Rush 6.

The longstanding older model used normal version ranges, like this example:

tsdoc-config/package.json

  "dependencies": {
    "@microsoft/tsdoc": "0.12.21",  // <--- reference to local project
    "ajv": "~6.12.3",
    "jju": "~1.4.0",
    "resolve": "~1.12.0"
  },

This is harder to detect. The dependency is considered a local reference (i.e. symlinked instead of installed) only if all of these things are true:

• The project "@microsoft/tsdoc" exists in the monorepo (i.e. some other package.json has this as its "name"); AND
• The version range ("0.12.21") is SemVer-compatible with the "version" field of that other project; AND
• The dependency is not exempted via the "cyclicDependencyVersions" option in rush.json

You can use the @microsoft/rush-lib API to answer these questions, but it's probably safe to skip this support, as the older model is going to be deprecated soon. The workspace: specifier is a superior design, and many repos are already opted-in to that mode.

[rarkins]

Thank you everyone for the interest and expert details on Rush.

File matching

We'll need to decide between the "traditional" npm manager approach of matching package.json and then looking for meta files like lerna.json or maybe a new approach of matching rush.json and then considering all the package.json files as "additional files" instead of "package files". The additional file approach seems cleaner although may need some improvements to our "autoReplace" logic first.

Manager conflicts

Assuming we define rush as a standalone "manager" in Renovate, we need to think what to do when Renovate matches a Rush project as both npm/yarn/pnpm (currently bundled into the npm manager) as well as with rush. It might work as-is (both managers update the same package.json files and don't really care if somehow they were already updated) but ideally we wouldn't do duplicate work or risk some type of conflict.

Obviously we could ask users to manually disable npm if you're using Rush, but normally we try to automate that type of logic away. Example logic we could use:

1. If rush returns any files during the extraction phase then ignore/disregard any files that npm also returns, OR

2. Dig into the rush matches to find all package.json files and ignore/disregard only those specific files for npm. If there happens to still be some non-rush package.json files then process them as-is with npm

Do you think (2) is a valid use case, or is (1) simplest and best?

Ignoring internal packages

Generally, if Renovate finds a dependency with a valid version then it tries to update it. For Lerna and Yarn Workspaces, we do a post-processing step to collect all package.json>name fields and then ignore any references to those packages in any other package files. We don't have any fancy logic to check if they're within the valid range or not, etc.

If Rush is moving towards the workspace: approach then that would be very simple because we would naturally ignore a non-semver reference anyway.

Artifact updating

Renovate uses the term "artifact updating" to mean any updating required after we've patched package files. e.g. may be lock files, checksums, vendored dependencies, or whatever Rush needs.

For most package managers we perform the artifact updating immediately after updating a dependency. i.e.

1. Patch package file
2. Run command to update artifacts

For current npm managers, we delay artifact updating until after all dependency updating / package file patching has been done. Which best describes Rush?

1. Best to run the Rush command after each package file patch
2. Need to wait until all patching is done
3. Either works, but doing it once is most efficient

Remember that if we run the Rush command after each patch, it can mean that the same package in multiple package files can have inconsistent versions. e.g. if we want to update a dependency X from 1.0.0 to 1.0.1 in 5 different files then the first four times would mean both 1.0.0 and 1.0.1 are present while only after the 5th time would it be consistently 1.0.1.

Installing tools before updating artifacts

Do I understand correctly that there's no need to do something like npm install -g rush because the logic is self-contained in the committed ./common/scripts/install-run-rush.js? How about npm, yarn or pnpm - does Rush assume that they are installed globally or does it manage the installation of them too?

Commit file patterns

Can we have pre-defined patterns of which "rush files" to look for modifications for? Although the equivalent of git add --all might work most of time, we generally try to be explicit about which files we're committing just so we don't end up adding something we don't want to.

[mmkal]

My as-a-user, not-a-maintainer views (since I said I'd be willing to open a PR - that's looking ambitious now, but can try to help with hopefully-accurate info):

1. If rush returns any files during the extraction phase then ignore/disregard any files that npm also returns, OR
2. Dig into the rush matches to find all package.json files and ignore/disregard only those specific files for npm. If there happens to still be some non-rush package.json files then process them as-is with npm

Do you think (2) is a valid use case, or is (1) simplest and best?

(1) looks simplest and best to me. I suspect the vast majority of rush monorepos just... use rush. If not, the first and best choice for those repos would be to fix that and not have secret projects that their monorepo manager isn't told about. The second choice would probably be to open an issue with rush asking them to fix whatever issue is preventing users from wanting to declare their packages in rush.json. The third option would be to create a new issue here saying "please support my hybrid rush use case" and see how many 👍 s it gets.

For current npm managers, we delay artifact updating until after all dependency updating / package file patching has been done. Which best describes Rush?

1. Best to run the Rush command after each package file patch
2. Need to wait until all patching is done
3. Either works, but doing it once is most efficient

Remember that if we run the Rush command after each patch, it can mean that the same package in multiple package files can have inconsistent versions. e.g. if we want to update a dependency X from 1.0.0 to 1.0.1 in 5 different files then the first four times would mean both 1.0.0 and 1.0.1 are present while only after the 5th time would it be consistently 1.0.1.

For this one, it's pretty important that rush update is called once after all package patches have been applied. Rush has a setting ensureConsistentVersions which would error for repos that use it in the intermediate stages.

Do I understand correctly that there's no need to do something like npm install -g rush because the logic is self-contained in the committed ./common/scripts/install-run-rush.js? How about npm, yarn or pnpm - does Rush assume that they are installed globally or does it manage the installation of them too?

No need to npm install -g rush. But it looks like install-run-rush needs npm to exist globally.

Can we have pre-defined patterns of which "rush files" to look for modifications for? Although the equivalent of git add --all might work most of time, we generally try to be explicit about which files we're committing just so we don't end up adding something we don't want to.

In the repo I've been testing on it's common/config/rush/pnpm-lock.yaml. I suspect a fuller list would be common/config/rush/{pnpm-lock.yaml,shrinkwrap.yaml,yarn.lock,npm-shrinkwrap.json} based on a quick look at the rush code. I don't know if there are others though.

---

Again, take what I say with a pinch of salt, the above is just based on what I've seen as a (fairly recent) user.

[octogonz]

I agree with everything @mmkal said.

In the repo I've been testing on it's common/config/rush/pnpm-lock.yaml. I suspect a fuller list would be common/config/rush/{pnpm-lock.yaml,shrinkwrap.yaml,yarn.lock,npm-shrinkwrap.json} based on a quick look at the rush code. I don't know if there are others though.

It would be safe to match any file under common/config/*. Rush manages this folder very carefully and will report an error if an unrecognized file appears there.

Rush supports many other advanced version-related features that we could throw at you (e.g. autoinstallers, installation variants, allowedAlternativeVersions, preferredVersions, etc.). But 95% of your users won't use those features, or would be fine if Renovate Bot occasionally makes a PR that needs manual assistance to merge. Better to start simple and see what people actually ask about.