Flux: Trying to sync errors out with a nil error

[Skarlso]

Describe the bug

I have a flux install like this:

fluxctl install \
--git-user=gaia-bot \
[email protected] \
[email protected]:gaia-pipeline/gaia-infrastructure \
--git-path=namespaces,workloads \
--namespace=flux | kubectl apply -f -

Note the that user where the repo is and the user who would do the pushing is different.
git-user is gaia-bot but the repo is gaia-pipeline/gaia-infrastructure. I don't think this should be a problem?

EDIT: tried with gaia-bot as gaia-bot/gaia-infrastructure too. Same result. Something is clearly missing...

To Reproduce

Steps to reproduce the behaviour:

The repo of the manifests: https://github.com/gaia-pipeline/gaia-infrastructure

It has the deployment key installed.

Logs

If applicable, please provide logs of fluxd. In a standard stand-alone installation of Flux, you'd get this by running kubectl logs deploy/flux -n flux.

ts=2020-07-19T13:09:34.376395611Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2020-07-19T13:09:34.376946965Z caller=images.go:27 component=sync-loop msg="no automated workloads"
ts=2020-07-19T13:09:34.402757981Z caller=loop.go:108 component=sync-loop err="loading last-synced resources: git repo not ready: git repo has not been cloned yet"
ts=2020-07-19T13:09:34.38026621Z caller=main.go:795 addr=:3030
ts=2020-07-19T13:09:35.027662793Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=1.20.0
ts=2020-07-19T13:09:37.126746601Z caller=warming.go:198 component=warmer info="refreshing image" image=docker.io/cilium/cilium tag_count=216 to_update=1 of_which_refresh=0 of_which_missing=1
ts=2020-07-19T13:09:37.267764819Z caller=repocachemanager.go:223 component=warmer canonical_name=index.docker.io/cilium/cilium auth={map[]} warn="manifest for tag svc-v2-fix-1.5.0 missing in repository docker.io/cilium/cilium" impact="flux will fail to auto-release workloads with matching images, ask the repository administrator to fix the inconsistency"
ts=2020-07-19T13:09:37.267883007Z caller=warming.go:206 component=warmer updated=docker.io/cilium/cilium successful=0 attempted=1
ts=2020-07-19T13:09:37.279356646Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2020-07-19T13:09:38.306194128Z caller=images.go:27 component=sync-loop msg="no automated workloads"
ts=2020-07-19T13:09:45.793616555Z caller=loop.go:134 component=sync-loop event=refreshed url=ssh://[email protected]/gaia-pipeline/gaia-infrastructure branch=master HEAD=e1f7dbc720307ced0d1155063409a7e13214fa47
ts=2020-07-19T13:09:45.803811327Z caller=sync.go:60 component=daemon info="trying to sync git changes to the cluster" old= new=e1f7dbc720307ced0d1155063409a7e13214fa47
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x176c94f]

goroutine 63 [running]:
github.com/fluxcd/flux/pkg/cluster/kubernetes/resource.Load.func1(0xc002813ec0, 0x25, 0x0, 0x0, 0x20f5120, 0xc00065d620, 0x0, 0xc00065cc30)
    /home/circleci/go/src/github.com/fluxcd/flux/pkg/cluster/kubernetes/resource/load.go:32 +0x6f
path/filepath.Walk(0xc002813ec0, 0x25, 0xc00068b618, 0x0, 0x0)
    /usr/local/go/src/path/filepath/path.go:402 +0x6a
github.com/fluxcd/flux/pkg/cluster/kubernetes/resource.Load(0xc001e48600, 0x1a, 0xc000b13100, 0x2, 0x2, 0xc001e48d00, 0x20, 0x20, 0x40c1d6)
    /home/circleci/go/src/github.com/fluxcd/flux/pkg/cluster/kubernetes/resource/load.go:31 +0x19b
github.com/fluxcd/flux/pkg/cluster/kubernetes.(*manifests).LoadManifests(0xc000a797a0, 0xc001e48600, 0x1a, 0xc000b13100, 0x2, 0x2, 0xc002813f50, 0x0, 0x20)
    /home/circleci/go/src/github.com/fluxcd/flux/pkg/cluster/kubernetes/manifests.go:122 +0x67
github.com/fluxcd/flux/pkg/manifests.(*rawFiles).GetAllResourcesByID(0xc002665d40, 0x213dda0, 0xc000044070, 0x22d943563f27e439, 0x81b22800cef83b71, 0xc00068b760)
    /home/circleci/go/src/github.com/fluxcd/flux/pkg/manifests/rawfiles.go:96 +0x60
github.com/fluxcd/flux/pkg/daemon.doSync(0x213dda0, 0xc000044070, 0x21257a0, 0xc002665d40, 0x2158440, 0xc0000d8800, 0xc002813f50, 0x2b, 0x20f2f20, 0xc000466750, ...)
    /home/circleci/go/src/github.com/fluxcd/flux/pkg/daemon/sync.go:221 +0x66
github.com/fluxcd/flux/pkg/daemon.(*Daemon).Sync(0xc0003d6900, 0x213dda0, 0xc000044070, 0x2f56e37c, 0xed6a63d19, 0x0, 0xc002813b60, 0x28, 0x21256a0, 0xc0005aa040, ...)
    /home/circleci/go/src/github.com/fluxcd/flux/pkg/daemon/sync.go:71 +0x404
github.com/fluxcd/flux/pkg/daemon.(*Daemon).Loop(0xc0003d6900, 0xc000086660, 0xc000284430, 0x20f2f20, 0xc0004667b0)
    /home/circleci/go/src/github.com/fluxcd/flux/pkg/daemon/loop.go:103 +0x525
created by main.main
    /home/circleci/go/src/github.com/fluxcd/flux/cmd/fluxd/main.go:777 +0x5990

The pod crashes horribly.

Versions

Kube: kubernetes-v1.17.6
Flux: version=1.20.0

[Skarlso]

Tried 1.19 and I've got this error:

ts=2020-07-19T16:37:25.620077511Z caller=images.go:23 component=sync-loop error="getting unlocked automated resources: walking \"/tmp/flux-working710212861/namespaces\" for yaml files: lstat /tmp/flux-working710212861/namespaces: no such file or directory"           │
│ ts=2020-07-19T16:37:26.521572058Z caller=warming.go:198 component=warmer info="refreshing image" image=postgres tag_count=318 to_update=318 of_which_refresh=0 of_which_missing=318

This is clear. So there is a regression in 1.20 not handling this. BTW. The doc said it's not important where the files are, so why is it looking for a namespaces folder?

[rvarunikaa]

I get a similar error ..

`panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x176c7af]

goroutine 64 [running]:
github.com/fluxcd/flux/pkg/cluster/kubernetes/resource.Load.func1(0xc0005f21b0, 0x2a, 0x0, 0x0, 0x20f50e0, 0xc0006ca450, 0x0, 0xc0006092f0)
/home/circleci/go/src/github.com/fluxcd/flux/pkg/cluster/kubernetes/resource/load.go:32 +0x6f
path/filepath.Walk(0xc0005f21b0, 0x2a, 0xc00080f6b0, 0x0, 0x0)
/usr/local/go/src/path/filepath/path.go:402 +0x6a
github.com/fluxcd/flux/pkg/cluster/kubernetes/resource.Load(0xc00081fb00, 0x1a, 0xc0002ccbd0, 0x1, 0x1, 0xc0002ccb00, 0xc000828ba0, 0xc00081fb00, 0xc00080f790)
/home/circleci/go/src/github.com/fluxcd/flux/pkg/cluster/kubernetes/resource/load.go:31 +0x19b
github.com/fluxcd/flux/pkg/cluster/kubernetes.(manifests).LoadManifests(0xc000828ba0, 0xc00081fb00, 0x1a, 0xc0002ccbd0, 0x1, 0x1, 0x42e0aa, 0x0, 0xc000820160)
/home/circleci/go/src/github.com/fluxcd/flux/pkg/cluster/kubernetes/manifests.go:122 +0x67
github.com/fluxcd/flux/pkg/manifests.(rawFiles).GetAllResourcesByID(0xc0000d9bc0, 0x213dda0, 0xc0000e8048, 0xc001eedad0, 0x28, 0x21257a0)
/home/circleci/go/src/github.com/fluxcd/flux/pkg/manifests/rawfiles.go:96 +0x60
github.com/fluxcd/flux/pkg/daemon.(Daemon).getLastResources(0xc00025b320, 0x213dda0, 0xc0000e8048, 0x21256a0, 0xc00036e100, 0x0, 0x0, 0x0)
/home/circleci/go/src/github.com/fluxcd/flux/pkg/daemon/sync.go:140 +0x176
github.com/fluxcd/flux/pkg/daemon.(Daemon).Sync(0xc00025b320, 0x213dda0, 0xc0000e8048, 0x28a09ff7, 0xed6a7a270, 0x0, 0xc001eed9e0, 0x28, 0x21256a0, 0xc00036e100, ...)
/home/circleci/go/src/github.com/fluxcd/flux/pkg/daemon/sync.go:49 +0x82
github.com/fluxcd/flux/pkg/daemon.(*Daemon).Loop(0xc00025b320, 0xc000192120, 0xc00030e710, 0x20f2ee0, 0xc0004b2510)
/home/circleci/go/src/github.com/fluxcd/flux/pkg/daemon/loop.go:103 +0x525
created by main.main
/home/circleci/go/src/github.com/fluxcd/flux/cmd/fluxd/main.go:777 +0x5990`

[Skarlso]

@rvarunikaa most likely it's the same. You need to have a namespaces and a workloads folder. And in there you'll have your yaml files.

[rvarunikaa]

Thanks @Skarlso . I looked at your comment from yesterday with respect to version 1.19. My issue was resolved without needing to add any folders but by just switching to v1.19 for fluxcd.

[tshak]

This very likely a regression in 1.20. As part of an automated test suite I install Flux and the git folders are not yet created. This puts Flux in a CrashLoopBackoff state. The logs were much more clear in 1.19 and would retry.