Renovate: Batched updates for noise reduction

[rarkins]

Although Renovate supports schedules, grouping and automerging to reduce "noise", the result is still too noisy for many people who watch renovated projects. Of course, most of that annoyance is because of GitHub Notifications, which GitHub could improve greatly by letting you mute certain conditions.. but we need to work with what we have.

One idea I have had is to improve the way Renovate does scheduled upgrades.

Today, the way weekly updates might work is either:

1. Keep dependencies mostly separate (e.g. other than predefined monorepos) and then you end up with maybe 10+ PRs all at once at the scheduled time. Then you have the choice of either (a) merging them all independently and hoping none conflicted with each other, or (b) merging, waiting (for rebase), merging, waiting, etc. If you have lock files then you may have no choice but to wait for PR regeneration after each merge.

2. Aggressively group together dependencies for the weekly update. In this case if one update is bad, the PR fails tests and probably nothing will get upgraded. And then what? Work out manually which is bad and manually fix the branch to try to pass? i.e. this is great when it works/passes and ugly when it does not.

I am thinking of a third option:

3a) Renovate creates one branch per upgrade (or group) like in option 1 above, but does not create the PR. For most people, this would be "silent" or quiet enough because no GitHub notification as no PR
3b) Once a week (or actually when scheduled..), Renovate bundles together all passing updates into one branch/PR.

When this works well, it means you get one PR per week like in (2) above, with hopefully much reduced chance that this combined PR fails.

Two main failure cases to consider:

i) If an upgrade in a branch fails during the week, what to do then? e.g. PR the failed upgrade immediately for someone to look at? Or ignore the failure and wait x days to see if a fix is issued, i.e. only raising a failed PR if the failed upgrade doesn't appear to be getting fixed, or in case it needs a fix in the current repo?

ii) If the combined weekly branch/PR fails testing. i.e. the individual branches had passed, but when combined they have failed - either because of a conflict between them or perhaps a transient CI problem. Right now I'm thinking that the only option is to still raise the PR and let a human intervene.

[rarkins]

If anyone has the time, I would like to hear feedback and ideas. @vvo @schickling @hbetts @ikatyang @olstenlarck @edmorley @darkbasic

[tunnckoCore]

Yep, 3b sounds good, but as you said it has problems too.

Thinking it... One interesting path may be to raise PR if all tests are passing and if they fail when combined i.e. PR fails, then __automatically close__ the PR and open separate PRs for them.

Why? yes, it still be noise, but it would be better than to have failing PR with all combined deps - how you will now WHY pr fails and which deps/versions are in conflict? I think this shouldn't be done by human, so let the bot send separate PRs.

And good options that user should use are the rebase and automerge options.

[rarkins]

Another idea I’d had was whether to “build them up” as the week goes. But I think the logic for that could be really hard to both implement and understand.

[ghost]

For option (1), my experience has been that the individual PRs are quite time consuming to integrate back into the master branch, but not really annoying (The behavior is driven by my own configuration choices, and I accept the consequences). However, the time needed to accept a pull request, then wait for the others to be updated, is a drag. It's a _lot_ of context switching, between the pull requests, and my other work.

Option (2) seems like a non-starter. I think goes against the original Design Decisions around Renovate; to make it really easy to discover when a dependency update requires manual intervention to resolve (and to make that easy to reason about).

Therefore a compromise seems like the most reasonable way forward to reduce _noise_.

[ghost]

As for option (3), it seems that perhaps it's unnecessary if you're auto-merging. If an update passes, and you have selected the option to merge directly from a branch instead of a pull request, then you would never receive a PR, or notice, for that update. It would simply be merged into your master branch.

So my assumption is that option (3) is for developers that want manual control over accepting dependency updates, but want less _noise_ than option (1).

[edmorley]

For me personally, I don't think I'd ever want updates batched (other than for monorepo packages like react*), so it's hard to predict what may be useful for others :-)

[ghost]

If an upgrade in a branch fails during the week, what to do then?

Would you not just raise the PR for the failed branch at the same time you raised the PR containing all working dependency updates? All PRs would be raised at the scheduled time? This would remain in line with the behavior of option (1) as to timing.

As for failure case (ii), I'd probably raise individual PRs so that the developer can start accepting, one at a time, those that pass until the one, or more, dependencies causing the failure are discovered.

[ghost]

@edmorley personally I wouldn't use a batch feature.

I prefer semantic version ranges for dependencies, auto-merging minor and patch versions (if a dep is pinned), and weekly lockfile maintenance. The combination of those options keeps _noise_ to an acceptable level.

However, I assume there are developers out there that are receiving a large number of PRs from Renovate and want to further lower the _noise_.

[rarkins]

@Osmose I noticed that you have your socorro repository set to batch all updates together, and also that you had some breaking changes that held up the batch recently, so your repo might be a good candidate for this type of feature. If you have time, can you read through the above discussion and let us know your thoughts?

[Osmose]

Context switching between PRs is really disruptive for me, so the proposals that involve extra PRs being made don't seem attractive to me. I much prefer batching all update-related changes into a single PR, because then I don't have to coordinate in my head which update PRs have and have not gotten merged, or which combinations of update PRs result in passing tests.

It's not actually clear to me what's noisy about the current state of things with batching; a single PR gets made, it fails, and then it's human intervention time. What noise are we trying to reduce?

[rarkins]

@Osmose your preferred approach (batching all updates together once a month) is definitely the "quietest" approach.

But you may be surprised to hear that many prefer the almost opposite approach - they want their PRs immediately and they want each dependency in its own PR unless they are very closely related (e.g. published from the same monorepo).

It's not so much a "time" thing - some people also defer separated PRs weekly or monthly for example - but rather about the type of human intervention needed and how to handle it.

In your case, you might have a critical React bugfix being held up in your "all" PR because you need time to fix breaking changes in D3, for example. Whereas by keeping each dependency separate you can merge the passing fixes and features of others immediately and wait as long as you want to fix a breaking change from another dependency. The disadvantage of this separate approach is "noise".

What I am suggesting with this feature discussion is a kind of compromise: Renovate "silently" tests each new update as they arrive, so it knows which ones pass on their own. Then at the scheduled time (e.g. might be weekly or monthly), Renovate then combines all passing ones into a single batch PR that hopefully also passes, and any failing ones should probably be raised into their own PRs for human intervention. This way you would end up with more PRs than the single batch "all", because failing ones get separated, but perhaps the convenience of not holding up the rest of them outweighs this extra "noise".

But in summary I don't think there's right or wrong ways - it depends completely on (a) the person who does most of the work following up on the PRs, and (b) their team who has to listen to the "noise". I'm just wondering if this idea might be attractive to some, but apparently yet to find someone on this thread :)

[Osmose]

Ah. The wording of the initial comment made it seem like you were saying "batching helps with noise but is still too noisy", instead of "some people might want an option with less noise but also less of these problems with batching". That makes sense.

I don't think I can add more besides my vote that monthly batching is exactly what I want already. :D

[vvo]

3a) Renovate creates one branch per upgrade (or group) like in option 1 above, but does not create the PR. For most people, this would be "silent" or quiet enough because no GitHub notification as no PR
3b) Once a week (or actually when scheduled..), Renovate bundles together all passing updates into one branch/PR.

I think those are very interesting ideas yes, the default renovate behaviour should be to create one branch silently, have it OK/NOT OK (open PR if not ok, immediately).

For all OK PRS, when schedule is met then open a single PR with all the updates in it. In most cases this should be completely OK? But it's harder to do maybe.

[rarkins]

In most cases this should be completely OK? But it's harder to do maybe.

I think it's fair to say in most cases it should be OK. The only reasons it would fail are:

• Two or more upgrades that individually passed somehow conflict when upgraded together
• Transient CI failure that should clear if retried

Something else we can probably also do is leave the individual branches still there. This would allow you to create PRs from those branches in 1-2 clicks if you wish to fall back to individual branches.

[vvo]

Two or more upgrades that individually passed somehow conflict when upgraded together

Yep but since you will still create a branch for the batched and only if it fails (for automerge), open a PR, then we should be fine :)

[bhearsum]

I haven't seen it mentioned here yet, but the other reason that batching is useful is to avoid bustage where one package depends on a recent version of another one. This is less likely to happen when you stay on top of updates, but it's something I've seen before when using Pyup.io.

[Leksat]

Option 3 means more work for Renovate and CI. (If I got it right - Renovate first tests updates individually and if everything is good, it bundles them into a single PR)

Wouldn't it make sense to swap 3a and 3b? I mean, for me it would be perfect if:

• Renovate creates a PR with all updates.
• If checks are green, it merges the PR.
• If checks fail, it creates individual PRs for all updates to catch the guilty one.