Renovate: Support Post Upgrade Task

[ghost]

I've recently been publishing a tslint-config-<name> package containing TSLint rules.

Every time I add new rules, or modify rules in a way that requires code changes on the user's part, I release a new major version.

A new major version causes renovate to submit pull requests to consuming projects. The pull requests can't be accepted automatically because there are linting errors in the build of consuming projects.

I was wondering if there was a way to setup a process that would run tslint --fix on the user's codebase to auto-fix TSLint errors? (This could also apply to eslint --fix)

This could also apply to things like codemods.

[rarkins]

Interesting idea. I reallly wanted to avoid Renovate needing to git clone every project, but it might be necessary for this. Worse still I think we need to npm install too, for getting all the tslint and eslint plugins and configs, right?

Wondering if CI can do this instead (seeing as it's already cloning and installing) but I guess most CI systems are not set up for committing back to a project.

[rarkins]

I'm liking this idea the more I think about it. I'm torn between:
(a) trying to plan a generic approach (e.g. codemods like you pointed to), or
(b) just getting it working with eslint/tslint to start with and learning from there

One challenge can be to detect the right "fix" command because for many projects simply running eslint --fix might impact the wrong files/directories. But we can probably write a good default detector, e.g. searching npm scripts for the repository's npm run eslint command and then appending -- --fix to it. And then allow for users to configure the "fix" command in renovate config if necessary.

Another challenge is knowing for sure when to run it. e.g. if you just searched for eslint in any upgraded package name, that might work well, but it might not catch things if you update prettier, for example. Ideally we don't have to run eslint --fix for every single branch "just in case" so may need a provisioned set of patterns for package names to decide whether to run eslint --fix.

[ghost]

One challenge can be to detect the right "fix" command because for many projects simply running eslint --fix might impact the wrong files/directories.

As an example, projects I maintain may not have the same source code directory naming convention as other projects.

[rarkins]

Example request from https://github.com/algolia/instantsearch.js/pull/2556

In this case it appears they would run yarn run fix:prettier.
The challenge is how to "sandbox" it and/or protect myself from arbitrary commands.
Maybe git clone, yarn, then yarn run fix:prettier done inside a Docker container so it can't "escape" ?

[rarkins]

Here is another use case: https://github.com/angular/devkit/blob/master/docs/specifications/update.md

[mxey]

Another use case: We use Skaffold for Kubernetes pipelines. We have our own Docker image with Skaffold in it that we version and have Renovate bump in .gitlab-ci.yml. Skaffold tends to make incompatible config changes on new releases, but provides a command skaffold fix to update to the new config version.

[rarkins]

Example in repo https://github.com/prisma/nexus-prisma:

• Renovate (already) updates nexus-prisma
• Renovate needs to run yarn nexus-codegen afterwards, and commit all changed files

This requires:

• gitFs to be in use (so that all files can be accessed)
• full install of node_modules (in order to run from node_modules/.bin/nexus-codegen)

And then this new capabilities:

• post upgrade task
• detect file changes using git

Cc @Weakky

[bhutanigautam]

❤️ this thread.
Another use case -

We have a monorepo of packages and deployables/webs. The packages are ever green in a way that the webs consume the packages from within the monorepo instead of fetching the packages from a npm registry. Every time a package dependency is updated, we run a npm install in the web to update the webs package-lock.json file with the new dependency.
Renovate is working out great for us for the external packages but for anything inside the monorepo, we _will_ need to run a task (npm install) after the update so that the entire monorepo is up to date with the latest changes.

[wodenx]

Another use-case. We have shared CI scripts and configuration files which we are managing as an npm package. The CI system requires these to be present in each repo at the root, so we have a small npm script which copies them into place. It would be great to be able to execute this on each update, so that the updated config files were in the correct location within the updating pr.

[ndbroadbent]

Another use-case: My .gitlab-ci.yml uses some private Docker images that I push to the GitLab Docker registry. I was previously using a :latest tag for everything, but then the CI builds would start failing if I updated a language or system dependency version on master, but hadn't rebased a branch yet, etc.

To fix this, I've just started tagging these images with specific version labels: $RUBY_VERSION-$NODE_VERSION-$LIBRARY_VERSION-.... (I also pull in all of these versions from files that are now managed by Renovate Bot. For now I just manually rebuild and push these Docker images for CI, but eventually I want to automate all of this somehow.)

So now my .gitlab-ci.yml looks something like this:

image: registry.gitlab.com/myproject/myapp/ci:ruby2.5.7-node8.11.3
...
image: registry.gitlab.com/myproject/myapp/ci_python:ruby2.5.7-node8.11.3-pythonx.x.x
...
image: registry.gitlab.com/myproject/myapp/ci_php:ruby2.5.7-node8.11.3-phpx.x.x

And I've set up a bash script that uses sed to update the version tags for these images. Example:

#!/bin/bash
set -e
RUBY_VERSION=$(cat .ruby-version)
NODE_VERSION=$(cat package.json | jq -r ".engines.node" | grep -o "\d\+\.\d\+\.\d\+")
IMAGE_LABEL="ruby$RUBY_VERSION-node$NODE_VERSION"

sed -i.bak \
  -e "s/\(image: registry\.gitlab\.com[^:]*\):.*/\\1:$IMAGE_LABEL/g" \
  ".gitlab-ci.yml"
rm -f .gitlab-ci.yml.bak

So I'm looking for some way to automatically run ./scripts/update_gitlab_ci_images whenever Renovate Bot updates my Ruby/Node/Python/PHP/etc. versions, so that my tests are running with the correct versions.

Thanks!

[dominic-jones]

Since you were asking for use cases..

We run a version "pin" script whenever our (maven/gradle) jar files change. That's mainly for PR review, so reviewers can see what transitive dependencies will be modified in a given upgrade. This we would like to run post-renovate run.

[renovate-release]

:tada: This issue has been resolved in version 19.118.0 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket:

[kellyselden]

Is there a ticket I can watch for when this rolls out to the public instance?

[rarkins]

@kellyselden this feature is currently reserved for self-hosted users only, due to the security risks. However if you think there's a command that could be safely run post-upgrade in the app, please raise a request in https://github.com/renovatebot/app-support and we can consider it.

[darl]

Playing with this feature right now and cannot get it working.
It appears that post-upgrade tasks executed before package updates got flushed to disk (in platform.commitFilesToBranch function).
Is it intended?

[daekene]

@darl No, this is not intended and I've recently encountered the issue myself. I've raised a PR to flush changes to disk before the tasks are executed, see #5988