Flux: Last synced comparison can break with manifest generation

[hamishforbes]

Describe the bug

I happened to refactor the directory layout of my config repo and upgrade to 1.20 at the same time.
This involved changing the git.path configuration to a new path which didn't exist in the previously sync'd revision.

It looks like when 1.20.0 tries to sync the new revision it first checks out the old revision and tries to generate the manifests in order to compare with the latest.

But the (new) path in the config doesn't exist in the old revision, so manifests can't be generated.

Downgraded to 1.19.0 and applied the config post-refactor and upgraded to 1.20.0 again, all working fine.

Expected behavior

Failing to generate old manifests shouldn't cause the whole sync to fail.
Flux should continue and apply the new revision.

Logs

Logs from 1.20.0 sync

ts=2020-07-09T12:57:21.217162206Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://[email protected]/****/k8s-flux-config branch=master HEAD=8be24a3797c8fa13916230beeffed7a739caffcc
ts=2020-07-09T12:57:21.361643463Z caller=loop.go:108 component=sync-loop err="loading last-synced resources: reading the repository checkout: error finding a config file starting at path \"environments/core-staging-euw1\": stat /tmp/flux-working720488700/environments/core-staging-euw1: no such file or directory"

I exec'd into the container and found the orphaned working directory was on the previously sync'd commit

/tmp/flux-working720488700/> git branch -av
* (HEAD detached at 6eda3f4) 6eda3f4 Add...

[squaremo]

@ordovicia What are your thoughts on this problem related to #3039?

The problem is that if you change your --git-path argument, then the generation (on first sync) of the previous resources will be generated using that git-path, and will either include things that are actually new, or break if the new path wasn't there before. If you have your flux configuration checked into git, and you change the argument there, you will probably hit this, because syncing will restart flux with the new path, but with the last-synced revision before the change.

[ordovicia]

Yes, I also think this issue comes from changes introduced in #3039.
One mitigation I came up is that when Daemon.getLastResources() failed to load resources from repo, Daemon.Sync() reports the error and continues processing, instead of immediately returning the error.
This might produce an inaccurate sync event, but syncing a cluster with a repo should be done well.

[monotek]

forced resync option via fluxctl would be nice...

[BarrieShieh]

I got this issue too. I changed the git path in config. And even i reinstalled the fluxcd 1.20.0, flux seems still pull the old commit...

[monotek]

Workaround that worked for me now was to delete fluxcd-sync & flux-write-check-c4d08f673a remote git tags and restart the flux pod...

[squaremo]

The general problem of not being able to reconstitute the previous state is addressed in #3223.

[gaffneyd4]

Just ran into this issue.

I added a new path in our git repo clusters/blue/foo, and updated the flux spec to sync the new path in a new commit (--git-path=clusters/blue/foo, but during flux's sync loop it was pointing at the previous commit when that path in the directory did not exist.

This caused flux to get stuck and throw this error:
flux-6855495dc8-nf5gj flux ts=2020-09-25T21:32:15.144567503Z caller=loop.go:108 component=sync-loop err="loading last-synced resources: reading the repository checkout: error finding a config file starting at path \"clusters/blue/foo\": stat /tmp/flux-working938582672/clusters/blue/foo: no such file or directory"

I tried deleting the flux pod a few times but it remained stuck after bootstrapping.

To get out of this state, I repointed the git tag that the flux instance was using to store its state manually:

# where d86d73cf9a2142ad3f8cfb26f694c0931b5e3e11 is the next commit that created the missing path
git tag -af flux-blue d86d73cf9a2142ad3f8cfb26f694c0931b5e3e11 -m "Sync pointer"
git push -f origin flux-blue

Then after rolling the flux pod again, it completed a sync successfully.

Leaving this here in case anyone else has a similar issue.