Flux: problems combining generated manifests via .flux.yaml with raw manifests in repo

[alexhumphreys]

Describe the bug
I have a raw yaml workload in a repo, and it's running in the cluster started by flux. When I add a .flux.yaml file, and edit the raw yaml workload and commit that, flux correctly starts the templated manifest, but removes the previous raw manifest.

I have made a minimal repo https://github.com/alexhumphreys/flux-yaml-test to demonstrate. The commit https://github.com/alexhumphreys/flux-yaml-test/commit/1caaa6a5f19dba2328094e14d7962d4b46f54b4e adds a raw yaml workload that is a deployment named nginx-workload. The commit https://github.com/alexhumphreys/flux-yaml-test/commit/dd6d69f93b2e0819faf075164f6c96e926da5777 adds a .flux.yaml file which generates a deployment named nginx-template.

Flux is pretty new to me so there's a good chance I'm doing something pretty basic wrong, but any help would be appreciated.

To Reproduce
Steps to reproduce the behaviour:

1. go to https://github.com/alexhumphreys/flux-yaml-test/tree/1caaa6a5f19dba2328094e14d7962d4b46f54b4e
2. deploy that to your environment
3. go to https://github.com/alexhumphreys/flux-yaml-test/commit/dd6d69f93b2e0819faf075164f6c96e926da5777 (child commit of the previous 1caaa6a)
4. deploy that to your environment
5. observer that templated nginx appears, and the raw nginx workload is removed.

Expected behavior
I expected both nginx-template and nginx-workload to both remain after that dd6d69f is applied. The docs here have the line "Flux supports both generated manifests and raw manifests tracked in the same repository", so I would assume what I'm attempting should be possible.

Logs

{"caller":"main.go:224","ts":"2019-07-08T10:16:55.797222605Z","version":"1.13.1"}
{"caller":"main.go:316","msg":"using in cluster config to connect to the cluster","ts":"2019-07-08T10:16:55.797290029Z"}
{"caller":"main.go:395","component":"cluster","identity":"/var/fluxd/keygen/identity","ts":"2019-07-08T10:16:55.867183097Z"}
{"caller":"main.go:396","component":"cluster","identity.pub":"-----REDACTED-----","ts":"2019-07-08T10:16:55.86722387Z"}
{"caller":"main.go:401","host":"https://10.96.0.1:443","ts":"2019-07-08T10:16:55.867246425Z","version":"kubernetes-v1.13.5"}
{"caller":"main.go:413","kubectl":"/usr/local/bin/kubectl","ts":"2019-07-08T10:16:55.867286975Z"}
{"caller":"main.go:425","ping":true,"ts":"2019-07-08T10:16:55.867842005Z"}
{"caller":"main.go:558","email":"[email protected]","notes-ref":"flux","set-author":false,"signing-key":"","sync-tag":"flux-sync","ts":"2019-07-08T10:16:55.86938521Z","url":"[email protected]:alexhumphreys/flux-yaml-test","user":"Weave Flux","verify-signatures":false}
{"caller":"main.go:619","ts":"2019-07-08T10:16:55.869450979Z","upstream":"no upstream URL given"}
{"caller":"loop.go:85","component":"sync-loop","err":"git repo not ready: git repo has not been cloned yet","ts":"2019-07-08T10:16:55.870853199Z"}
{"caller":"images.go:18","component":"sync-loop","msg":"polling images","ts":"2019-07-08T10:16:55.870920859Z"}
{"caller":"images.go:28","component":"sync-loop","msg":"no automated workloads","ts":"2019-07-08T10:16:55.870945348Z"}
{"addr":":3030","caller":"main.go:640","ts":"2019-07-08T10:16:55.871516005Z"}
{"caller":"checkpoint.go:24","component":"checkpoint","latest":"1.13.1","msg":"up to date","ts":"2019-07-08T10:16:56.35147338Z"}
{"caller":"warming.go:198","component":"warmer","image":"memcached","info":"refreshing image","of_which_missing":72,"of_which_refresh":0,"tag_count":72,"to_update":72,"ts":"2019-07-08T10:16:57.073837225Z"}
{"attempted":72,"caller":"warming.go:206","component":"warmer","successful":72,"ts":"2019-07-08T10:16:58.110585929Z","updated":"memcached"}
{"caller":"images.go:18","component":"sync-loop","msg":"polling images","ts":"2019-07-08T10:16:58.110728576Z"}
{"caller":"images.go:28","component":"sync-loop","msg":"no automated workloads","ts":"2019-07-08T10:16:58.11076126Z"}
{"caller":"warming.go:198","component":"warmer","image":"vault","info":"refreshing image","of_which_missing":51,"of_which_refresh":0,"tag_count":51,"to_update":51,"ts":"2019-07-08T10:16:58.830092877Z"}
{"attempted":51,"caller":"warming.go:206","component":"warmer","successful":51,"ts":"2019-07-08T10:16:59.582517498Z","updated":"vault"}
{"caller":"images.go:18","component":"sync-loop","msg":"polling images","ts":"2019-07-08T10:16:59.583779236Z"}
{"caller":"images.go:28","component":"sync-loop","msg":"no automated workloads","ts":"2019-07-08T10:16:59.583856351Z"}
{"caller":"warming.go:198","component":"warmer","image":"gcr.io/google_containers/defaultbackend","info":"refreshing image","of_which_missing":5,"of_which_refresh":0,"tag_count":5,"to_update":5,"ts":"2019-07-08T10:17:00.281339039Z"}
{"attempted":5,"caller":"warming.go:206","component":"warmer","successful":5,"ts":"2019-07-08T10:17:01.163260052Z","updated":"gcr.io/google_containers/defaultbackend"}
{"caller":"images.go:18","component":"sync-loop","msg":"polling images","ts":"2019-07-08T10:17:01.163450888Z"}
{"caller":"images.go:28","component":"sync-loop","msg":"no automated workloads","ts":"2019-07-08T10:17:01.163478411Z"}
{"caller":"warming.go:198","component":"warmer","image":"quay.io/kubernetes-ingress-controller/nginx-ingress-controller","info":"refreshing image","of_which_missing":53,"of_which_refresh":0,"tag_count":53,"to_update":53,"ts":"2019-07-08T10:17:04.69912131Z"}
{"attempted":53,"caller":"warming.go:206","component":"warmer","successful":53,"ts":"2019-07-08T10:17:05.334066946Z","updated":"quay.io/kubernetes-ingress-controller/nginx-ingress-controller"}
{"caller":"images.go:18","component":"sync-loop","msg":"polling images","ts":"2019-07-08T10:17:05.334316522Z"}
{"caller":"images.go:28","component":"sync-loop","msg":"no automated workloads","ts":"2019-07-08T10:17:05.33435533Z"}
{"caller":"warming.go:198","component":"warmer","image":"gcr.io/kubernetes-helm/tiller","info":"refreshing image","of_which_missing":73,"of_which_refresh":0,"tag_count":73,"to_update":73,"ts":"2019-07-08T10:17:05.948637609Z"}
{"attempted":73,"caller":"warming.go:206","component":"warmer","successful":73,"ts":"2019-07-08T10:17:07.048638704Z","updated":"gcr.io/kubernetes-helm/tiller"}
{"caller":"images.go:18","component":"sync-loop","msg":"polling images","ts":"2019-07-08T10:17:07.048892127Z"}
{"caller":"images.go:28","component":"sync-loop","msg":"no automated workloads","ts":"2019-07-08T10:17:07.048927305Z"}
{"caller":"warming.go:198","component":"warmer","image":"docker.io/weaveworks/weave-kube","info":"refreshing image","of_which_missing":378,"of_which_refresh":0,"tag_count":378,"to_update":378,"ts":"2019-07-08T10:17:07.801657753Z"}
{"attempted":378,"caller":"warming.go:206","component":"warmer","successful":378,"ts":"2019-07-08T10:17:10.542703951Z","updated":"docker.io/weaveworks/weave-kube"}
{"caller":"images.go:18","component":"sync-loop","msg":"polling images","ts":"2019-07-08T10:17:10.542946799Z"}
{"caller":"images.go:28","component":"sync-loop","msg":"no automated workloads","ts":"2019-07-08T10:17:10.542981822Z"}
{"caller":"warming.go:198","component":"warmer","image":"docker.io/weaveworks/weave-npc","info":"refreshing image","of_which_missing":360,"of_which_refresh":0,"tag_count":360,"to_update":360,"ts":"2019-07-08T10:17:11.241032248Z"}
{"attempted":360,"caller":"warming.go:206","component":"warmer","successful":360,"ts":"2019-07-08T10:17:13.892936094Z","updated":"docker.io/weaveworks/weave-npc"}
{"caller":"images.go:18","component":"sync-loop","msg":"polling images","ts":"2019-07-08T10:17:13.893182405Z"}
{"caller":"images.go:28","component":"sync-loop","msg":"no automated workloads","ts":"2019-07-08T10:17:13.89322312Z"}
{"caller":"warming.go:198","component":"warmer","image":"docker.io/weaveworks/flux","info":"refreshing image","of_which_missing":33,"of_which_refresh":0,"tag_count":33,"to_update":33,"ts":"2019-07-08T10:17:14.529466743Z"}
{"attempted":33,"caller":"warming.go:206","component":"warmer","successful":33,"ts":"2019-07-08T10:17:15.226749091Z","updated":"docker.io/weaveworks/flux"}
{"caller":"images.go:18","component":"sync-loop","msg":"polling images","ts":"2019-07-08T10:17:15.226985318Z"}
{"caller":"images.go:28","component":"sync-loop","msg":"no automated workloads","ts":"2019-07-08T10:17:15.227018979Z"}
{"caller":"loop.go:85","component":"sync-loop","err":"git repo not ready: git clone --mirror: fatal: Could not read from remote repository., full output:\n Cloning into bare repository '/tmp/flux-gitclone174512794'...\[email protected]: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","ts":"2019-07-08T10:18:55.871120895Z"}
{"HEAD":"1caaa6a5f19dba2328094e14d7962d4b46f54b4e","branch":"master","caller":"loop.go:111","component":"sync-loop","event":"refreshed","ts":"2019-07-08T10:19:08.795991056Z","url":"[email protected]:alexhumphreys/flux-yaml-test"}
{"args":"","caller":"sync.go:480","cmd":"apply","count":1,"method":"Sync","ts":"2019-07-08T10:19:10.785076202Z"}
{"caller":"sync.go:546","cmd":"kubectl apply -f -","err":null,"method":"Sync","output":"deployment.apps/nginx-workload created","took":"330.090451ms","ts":"2019-07-08T10:19:11.115310698Z"}
{"caller":"loop.go:206","component":"sync-loop","new":"1caaa6a5f19dba2328094e14d7962d4b46f54b4e","old":"fb32e8ba67166977830f6ed501db64382b91ac9e","tag":"flux-sync","ts":"2019-07-08T10:19:17.868297451Z"}
{"HEAD":"1caaa6a5f19dba2328094e14d7962d4b46f54b4e","branch":"master","caller":"loop.go:111","component":"sync-loop","event":"refreshed","ts":"2019-07-08T10:19:18.984485343Z","url":"[email protected]:alexhumphreys/flux-yaml-test"}
{"caller":"warming.go:198","component":"warmer","image":"nginx","info":"refreshing image","of_which_missing":245,"of_which_refresh":0,"tag_count":245,"to_update":245,"ts":"2019-07-08T10:19:56.703110498Z"}
{"attempted":245,"caller":"warming.go:206","component":"warmer","successful":245,"ts":"2019-07-08T10:20:00.313421213Z","updated":"nginx"}
{"caller":"images.go:18","component":"sync-loop","msg":"polling images","ts":"2019-07-08T10:20:00.313684555Z"}
{"caller":"images.go:112","component":"sync-loop","container":"nginx-workload","current":"nginx:1.16.0","info":"added update to automation run","new":"nginx:1-alpine-perl","pattern":"glob:*","reason":"latest 1-alpine-perl (2019-07-01 21:26:42.594359557 +0000 UTC) \u003e current 1.16.0 (2019-06-11 00:04:48.728051038 +0000 UTC)","repo":"nginx","ts":"2019-07-08T10:20:00.351570357Z","workload":"default:deployment/nginx-workload"}
{"caller":"loop.go:119","component":"sync-loop","jobID":"9adf1535-c680-ab5f-6ea0-f1a4c81b8843","state":"in-progress","ts":"2019-07-08T10:20:00.35168049Z"}
{"caller":"releaser.go:59","component":"sync-loop","jobID":"9adf1535-c680-ab5f-6ea0-f1a4c81b8843","ts":"2019-07-08T10:20:00.377868829Z","type":"release","updates":1}
{"caller":"loop.go:129","component":"sync-loop","err":{"type":"user","help":"Problem committing and pushing to git repository.\n\nThere was a problem with committing changes and pushing to the git\nrepository.\n\nIf this has worked before, it most likely means a fast-forward push\nwas not possible. It is safe to try again.\n\nIf it has not worked before, this probably means that the repository\nexists but the SSH (deploy) key provided doesn't have write\npermission.\n\nIn GitHub, please check via the repository settings that the deploy\nkey is \"Read/write\". You can cross-check the fingerprint with that\ngiven by\n\n    fluxctl identity\n\nIf the key is present but read-only, you will need to delete it and\ncreate a new deploy key. To create a new one, use\n\n    fluxctl identity --regenerate\n\nThe public key this outputs can then be given to GitHub; make sure you\ncheck the box to allow write access.\n\n","error":"git push [email protected]:alexhumphreys/flux-yaml-test [master refs/notes/flux]: failed to push some refs to '[email protected]:alexhumphreys/flux-yaml-test', full output:\n To github.com:alexhumphreys/flux-yaml-test\n   649425b..2438569  refs/notes/flux -\u003e refs/notes/flux\n ! [rejected]        master -\u003e master (fetch first)\nerror: failed to push some refs to '[email protected]:alexhumphreys/flux-yaml-test'\nhint: Updates were rejected because the remote contains work that you do\nhint: not have locally. This is usually caused by another repository pushing\nhint: to the same ref. You may want to first integrate the remote changes\nhint: (e.g., 'git pull ...') before pushing again.\nhint: See the 'Note about fast-forwards' in 'git push --help' for details.\n"},"jobID":"9adf1535-c680-ab5f-6ea0-f1a4c81b8843","state":"done","success":"false","ts":"2019-07-08T10:20:04.229398468Z"}
{"HEAD":"dd6d69f93b2e0819faf075164f6c96e926da5777","branch":"master","caller":"loop.go:111","component":"sync-loop","event":"refreshed","ts":"2019-07-08T10:20:07.97022118Z","url":"[email protected]:alexhumphreys/flux-yaml-test"}
{"args":"","caller":"sync.go:480","cmd":"apply","count":1,"method":"Sync","ts":"2019-07-08T10:20:09.941963206Z"}
{"caller":"sync.go:546","cmd":"kubectl apply -f -","err":null,"method":"Sync","output":"deployment.apps/nginx-template created","took":"151.313709ms","ts":"2019-07-08T10:20:10.093344664Z"}
{"caller":"sync.go:149","dry-run":false,"info":"cluster resource not in resources to be synced; deleting","resource":"default:deployment/nginx-workload","ts":"2019-07-08T10:20:13.881471326Z"}
{"args":"","caller":"sync.go:480","cmd":"delete","count":1,"method":"Sync","ts":"2019-07-08T10:20:13.88153072Z"}
{"caller":"sync.go:546","cmd":"kubectl delete -f -","err":null,"method":"Sync","output":"deployment.apps \"nginx-workload\" deleted","took":"71.114982ms","ts":"2019-07-08T10:20:13.952686315Z"}
{"caller":"daemon.go:652","component":"daemon","event":"Sync: dd6d69f, default:deployment/nginx-template","logupstream":"false","ts":"2019-07-08T10:20:13.955543471Z"}
{"caller":"loop.go:206","component":"sync-loop","new":"dd6d69f93b2e0819faf075164f6c96e926da5777","old":"1caaa6a5f19dba2328094e14d7962d4b46f54b4e","tag":"flux-sync","ts":"2019-07-08T10:20:17.05171854Z"}
{"HEAD":"dd6d69f93b2e0819faf075164f6c96e926da5777","branch":"master","caller":"loop.go:111","component":"sync-loop","event":"refreshed","ts":"2019-07-08T10:20:18.169072078Z","url":"[email protected]:alexhumphreys/flux-yaml-test"}
{"HEAD":"dd6d69f93b2e0819faf075164f6c96e926da5777","branch":"master","caller":"loop.go:111","component":"sync-loop","event":"refreshed","ts":"2019-07-08T10:22:09.169909711Z","url":"[email protected]:alexhumphreys/flux-yaml-test"}
{"args":"","caller":"sync.go:480","cmd":"apply","count":1,"method":"Sync","ts":"2019-07-08T10:22:20.138582227Z"}
{"caller":"sync.go:546","cmd":"kubectl apply -f -","err":null,"method":"Sync","output":"deployment.apps/nginx-template unchanged","took":"147.36829ms","ts":"2019-07-08T10:22:20.286009484Z"}

There is an error in that pile:

{"caller":"loop.go:129","component":"sync-loop","err":{"type":"user","help":"Problem committing and pushing to git repository.\n\nThere was a problem with committing changes and pushing to the git\nrepository.\n\nIf this has worked before, it most likely means a fast-forward push\nwas not possible. It is safe to try again.\n\nIf it has not worked before, this probably means that the repository\nexists but the SSH (deploy) key provided doesn't have write\npermission.\n\nIn GitHub, please check via the repository settings that the deploy\nkey is \"Read/write\". You can cross-check the fingerprint with that\ngiven by\n\n    fluxctl identity\n\nIf the key is present but read-only, you will need to delete it and\ncreate a new deploy key. To create a new one, use\n\n    fluxctl identity --regenerate\n\nThe public key this outputs can then be given to GitHub; make sure you\ncheck the box to allow write access.\n\n","error":"git push [email protected]:alexhumphreys/flux-yaml-test [master refs/notes/flux]: failed to push some refs to '[email protected]:alexhumphreys/flux-yaml-test', full output:\n To github.com:alexhumphreys/flux-yaml-test\n   649425b..2438569  refs/notes/flux -\u003e refs/notes/flux\n ! [rejected]        master -\u003e master (fetch first)\nerror: failed to push some refs to '[email protected]:alexhumphreys/flux-yaml-test'\nhint: Updates were rejected because the remote contains work that you do\nhint: not have locally. This is usually caused by another repository pushing\nhint: to the same ref. You may want to first integrate the remote changes\nhint: (e.g., 'git pull ...') before pushing again.\nhint: See the 'Note about fast-forwards' in 'git push --help' for details.\n"},"jobID":"9adf1535-c680-ab5f-6ea0-f1a4c81b8843","state":"done","success":"false","ts":"2019-07-08T10:20:04.229398468Z"}

I did the double check it recommended, and the fluxctl identity fingerprint matches the deploy key on github, and has read/write access.

Additional context

• flux version docker.io/weaveworks/flux:1.13.1
• Helm Operator version: not using helm operator
• Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
• Git provider: github
• dockerhub, nginx official image.
• that template command I'm using is pretty goofy, just cating a text file, but it was the simplest i could think of to make a failing case. But I did run into a similar issue using the more complicated .flux.yaml here https://github.com/alexhumphreys/flux-test-drive/blob/master/.flux.yaml

[squaremo]

Thank you for the great bug report!

From looking at your example repo, I think the problem is that fluxd defaults to examining one path (./) in the repo, and the .flux.yaml there is taken as defining the whole of the configuration -- fluxd will assume the only thing it needs to do is run what's in it.

To mix raw config and generated config, you'll need to provide two paths, one with the .flux.yaml and one containing the raw YAMLs. So you might have

-+ templates
 +-- .flux.yaml
 `-- nginx.txt
-+ workloads
 +-- nginx.yaml

then supply --git-path=templates,workloads as an argument to fluxd.

It sounds like the docs were not very clear on this point, so I'll have a look and see if they can be improved.

[alexhumphreys]

Thanks for your fast reply! Just tested out moving the .flux.yaml and the --git-path flag and it did the trick, thanks!

Just testing there so I understand the directory structure, it seems when you pass --git-path=templates,workloads the following raw yaml workload files will/won't be found and deployed by flux:

project-root
│   not-found-raw.yaml    
│
└───template
│   │   .flux.yaml
│   │   nginx.txt
│   └───subfolder1
│       │   not-found-raw.yaml
│   
└───workloads
│   │   found-raw.yaml
│   └───subfolder2
│       │   found-raw.yaml

[squaremo]

Yes that's correct. The algorithm is basically this:

• for each --git-path
  
  
  
  • if .flux.yaml exists then follow what it says there and ignore any other files
  
  
  • if no .flux.yaml then look for YAML files here and below (except for directories that look like Helm charts)
  
  

[alexhumphreys]

Ah, good to know, thanks! I'll close this issue so.

Also, thanks for your work on flux, it's got some great ideas, hopefully we can add it to our stack and simplify a bunch of stuff :+1: