Flux: Parallel releases gives git conflicts

[guzmo]

Describe the bug
When we do parallel releases in different pipelines we see the error below quiet often. It seems that if another pipeline pushes a release while another release has started, the local git repository will be behind remote and flux doesn't re-pull the branch, but rather throws this exception.

fluxctl --k8s-fwd-ns=flux release  --workload=deployment/some-service     --watch     --timeout=5m0s     --update-image=ourimage:tag
Submitting release ...
Error: git push [email protected]:a-repository.git [dev refs/notes/flux]: failed to push some refs to '[email protected]:a-repository.git', full output:
 To [email protected]:a-repository.git
   2b99c5ba..c8b828ae  dev -> dev
 ! [rejected]          refs/notes/flux -> refs/notes/flux (fetch first)
error: failed to push some refs to '[email protected]:a-repository.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
Run 'fluxctl release --help' for usage.
    at ChildProcess.exithandler (child_process.js:294:12)
    at ChildProcess.emit (events.js:198:13)
    at ChildProcess.EventEmitter.emit (domain.js:448:20)
    at maybeClose (internal/child_process.js:982:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)

To Reproduce
Steps to reproduce the behaviour:

1. Multi environment repository with overlays/ with a shared base. Each environment in a separate branch.
2. Create two services that exists both in the same environment/branch
3. Start two fluxctl release.... at almost the same time
4. One of them fails with above error

Expected behavior
Fluxctl should pull if it sees this error and try to automerge, if not possible, sure throw this exception.

Additional context
Add any other context about the problem here, e.g

• Flux version: 1.14.1
• Helm Operator version: 0.10.1
• Kubernetes version: 1.13.7-gke.19
• Git provider: Bitbucket

[max-lobur]

We're having similar concern since we put couple apps/envs in a single gitops repo, and they race for it.
Can we have git rebase & retry couple times with random timeout before failing? This will help to linearize all changes

[max-lobur]

Looks like this function: https://github.com/fluxcd/flux/blob/aef27b00febc7a0ea4f501e0ff83b88fe6ee5eab/git/working.go#L142
I can make a pull request if core team likes the idea

[avielb]

hey, any news on this?

[luislhl]

[rejected] refs/notes/flux -> refs/notes/flux (fetch first)

The error occurs not when pushing the branch, but when pushing the notesRef.

So the solution would be to always refetch the notes ref before trying to push the new notes.

Is there some way to control the interval Flux fetches notes from the repository? Decreasing it could help decrease how often this problem occurs. Does --git-poll-interval also control this?