Activiti: singlePullRequest convergence/gated pipelines unreliable due to versions getting overwritten

[ryandawsonuk]

Seems to be related to use of rebase, which causes the convergence checks to fail. I think it's actually the record of the versions for the chart-level convergence checks which is the problem. The content of the chart requirements file is fine at the point when the issue occurs. See comments from https://github.com/jenkins-x/updatebot/issues/43#issuecomment-466016749 onwards and the linked examples.

[ryandawsonuk]

Tried a couple of experiments above but hit problems with the shell syntax. Getting a closer to an approach.

[ryandawsonuk]

The approach in these PRs is to do a backoff and retry.

This is now running in the example-runtime-bundle, activiti-cloud-query, example-cloud-connector and activiti-cloud-notifications-graphql pipelines.

It simply retries the push-version, it doesn't remove the local .updatebot-repos/. There's a risk that could be needed if updatebot doesn't do a fetch. We should see whether it still happens and that will help us find out.

It would be preferable if we could resolve in updatebot itself but our PRs aren't getting reviewed there (I've had one open for months) and it feels like a fairly big change because the updatebot whose change was overwritten thinks it got the change through. We'd have to get updatebot to wait and re-refetch and run again, which is what we're doing with this change anyway.

[ryandawsonuk]

Just seen a case that initially looked like an instance of this https://github.com/Activiti/activiti-cloud-application-chart/pull/139#issuecomment-469334644 but actually turned out to be due to a github failure - so there can be different causes

[ryandawsonuk]

OK it still happens. Again http://jenkins.jx.35.228.195.195.nip.io/job/Activiti/job/activiti-cloud-query/job/master/81/console pushed v7.1.20 but the chart PR doesn't have the commit https://github.com/Activiti/activiti-cloud-application-chart/pull/142/commits

Probably this is because it doesn't remove the local .updatebot-repos, as noted above https://github.com/Activiti/Activiti/issues/2587#issuecomment-469255585

So either we remove those between pushes or we try to get it to merge without the rebase and force-push

[ryandawsonuk]

So this is/was most pronounced with charts but turns out it also affects singlePR convergence for activiti-cloud-dependencies. The conversation comments in https://github.com/Activiti/activiti-cloud-dependencies/pull/327 reveal that activiti-cloud-acceptance-tests did push a version to it (as its pipeline also says http://jenkins.jx.35.242.205.159.nip.io/job/Activiti/job/activiti-cloud-acceptance-tests/job/develop/88/console) but then service-common pushed its version with a force-push and overwrote it.

[ryandawsonuk]

Build 103 of service-common produced v7.1.16 and that has now propagated through cloud-dependencies , producing v7.1.22 there and then this was propagated through the examples and into application chart and example-chart - all without any manual intervention.

As a further test build 199 of cloud-dependencies produced v7.1.23 and that has gone through smoothly too

[ryandawsonuk]

To be sure I wasn't missing something I triggered another run and v7.1.25 also propagated through without any manual intervention