Renovate: Poor man's scalability for self-hosted Renovate

[fgreinacher]

What would you like Renovate to be able to do?

I'd like to use Renovate's autodiscovery feature (for convenience) and at the same time run multiple Renovate instances in parallel (for scalability).

I think a pretty simple solution to achieve this would be to introduce three new settings, like:

• clusterMode: boolean if set to true Renovate is aware that it's running in a cluster where it only needs to take care of a part of the work.
• clusterSize: number defines the number of instances in the cluster.
• clusterIndex: number defines the "instance ID" of the currently running instance in the cluster.

In this mode each instances would split the number of repositories to process in clusterSize groups and pick one based on its clusterIndex.

Did you already have any implementation ideas?

Logic described above, implemented around here:

https://github.com/renovatebot/renovate/blob/71cf56eaf9a8f010d72e4b19647f3cfae3bcdbf1/lib/workers/global/index.ts#L55-L61

Are there any workarounds or alternative ideas you've tried to avoid needing this feature?

Reimplementing auto-discovery outside renovate, spliting the lists there and passing them to Renovate explicitely.

Is this a feature you'd be interested in implementing yourself?

Yes

[rarkins]

autodiscoverFilter was created partly for this purpose, i.e. so that you could approximately split projects between separate instances using filters.

Are you suggesting that e.g. you'd run 3 at a time and they would each pick every third repo, using the order returned by autoDiscover?

[fgreinacher]

Yeah, I already know autodiscoverFilter, but you'd have to hack around with the pattern to find a good distribution. We are operating Renovate in a self-service manner and do want to reduce manual tuning as far as possible.

So yes, what I'd like to do is spin up 3 (or any number) instances that work independently on their part of the work. Each instance would then only process each clusterSizeth repository, starting at its clusterIndex.

[nejch]

Just to give more context, this can then be run, for example, in GitLab CI with parallel jobs, which expose CI_NODE_INDEX and CI_NODE_TOTAL, so each job would fetch only their share of projects. It's usually used for tests but would be nice here 👍

https://docs.gitlab.com/ee/ci/yaml/#parallel

[dlouzan]

Just thinking aloud here: wouldn't this approach introduce race conditions?

As I understand this, the parallel approach in Gitlab works in e.g. a test because the source of "truth" is a single revision of a file and then we can batch runs of those tests since all parallel jobs see the same file on startup; in the case of renovate with auto-discover, each parallel run has to trigger a remote call to Gitlab's API to fetch the list of repos to check, and this could change between two calls, so the indexes would potentially not match.

I remember reading at some point that concurrent runs of renovate of the same repository were not supported, unsure about the actual consequences. But if the jobs are not synced, we'd at least have the potential of missing repos on a run.

[nejch]

Hm that's true, the repo discovery should probably be done in a single call and then split for the run. So this can already be achieved manually by dumping the list into a file and splitting it for the actual run, I guess?

But maybe the CLI could have a convenience "discover"/"run" subcommands/flags to split this, if that makes any sense.

[rarkins]

Yes, this would introduce a race condition, its seriousness would depend on the time gap between starting up the multiple instances. e.g.

Let's say that there's 300 repos and the first node boots, takes repos 1,4,7,10,13....
Now let's say node 2 boots and takes repos 2,5,8,11,14.....
Now say that repo 3 is deleted or uninstalled.
The third node will take 4,7,10,13.....

I don't know if it's practical but remember that Renovate will look for a config.js or config.json file as its config, and a repository list is a valid config option there (as opposed to autodiscover results). So potentially you could query the list once and then dump it to file as you're suggesting.

FYI also you shouldn't see "actual" problems if the two instances run on the same repo at the same time, but there's a chance you'll see errors and it becomes quite noisy if you get errors frequently and don't know if they're false positives or not. And obviously it's defeating the purpose if you accidentally end up with two nodes servicing the same list and ignoring another list completely due to the race condition.

[fgreinacher]

Thanks for all your inputs, I did not think about this race condition!

I now think that this requirement needs to be solved outside Renovate,

• either by finding a good autodiscoverFilter to split the load as necessary
• or by performing repo discovery and generating of the split lists upfront, passing specific repository lists to Renovate

Therefore I'll close this issue. Let's just post an update here @dlouzan @nejch once we found a solution that works for us.