Argo-cd: App of Apps not working when using kustomize applications

[ageekymonk]

Checklist:

• [ x] I've searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

When creating an Apps of Apps, where each app is a kustomize app, apps donot get created. They throw the following error. one or more synchronization tasks are not valid. If i apply manually the app alone it works.

To Reproduce
Create a kustomize app which points to another kustomize app.

Expected behavior

Apps of Apps should be created.

Screenshots

Version
argocd: v1.4.2+48cced9
BuildDate: 2020-01-24T01:07:43Z
GitCommit: 48cced9d925b5bc94f6aa9fa4a8a19b2a59e128a
GitTreeState: clean
GoVersion: go1.12.6
Compiler: gc
Platform: darwin/amd64
argocd-server: v1.4.2+48cced9
BuildDate: 2020-01-24T01:07:03Z
GitCommit: 48cced9d925b5bc94f6aa9fa4a8a19b2a59e128a
GitTreeState: clean
GoVersion: go1.12.6
Compiler: gc
Platform: linux/amd64
Ksonnet Version: v0.13.1
Kustomize Version: Version: {Version:kustomize/v3.2.1 GitCommit:d89b448c745937f0cf1936162f26a5aac688f840 BuildDate:2019-09-27T00:10:52Z GoOs:linux GoArch:amd64}
Helm Version: v2.15.2
Kubectl Version: v1.14.0

[shabbir-taulia]

Sorry for commenting on this issue, but I seem to be running into the same using helm app of apps too. Running a sync results in the following

argoproj.io  Application  argocd     test-app                      OutOfSync  Missing        the server could not find the requested resource

And looking at the the application object i see

status:
  health:
    status: Missing
  observedAt: "2020-05-05T00:55:28Z"
  operationState:
    finishedAt: "2020-05-05T00:55:28Z"
    message: one or more synchronization tasks are not valid
    operation:
      initiatedBy:
        username: admin

Whereas adding the applications individually via the UI seems to work without any issues.

---

argo version
```
➜ (⎈ gke_poc_us-central1_argo:argocd) ~ ✗ argocd version
argocd: v1.5.2+c2c19f4
BuildDate: 2020-04-15T16:43:41Z
GitCommit: c2c19f42ad78ed7a6fb70e86aed117be484feb50
GitTreeState: clean
GoVersion: go1.14
Compiler: gc
Platform: darwin/amd64
argocd-server: v1.5.3+095c5d6
BuildDate: 2020-05-02T04:21:47Z
GitCommit: 095c5d616b1cb39f87e8f7a54cabea2643e1c99a
GitTreeState: clean
GoVersion: go1.14.1
Compiler: gc
Platform: linux/amd64
Ksonnet Version: v0.13.1
Kustomize Version: {Version:kustomize/v3.5.4 GitCommit:3af514fa9f85430f0c1557c4a0291e62112ab026 BuildDate:2020-01-11T03:12:59Z GoOs:linux GoArch:amd64}
Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
Kubectl Version: v1.14.0

[Woland2k]

Same problem as above with argo 1.5.4 and kustomize templates. However, mine says sync is successful, however no application pods are created. Works fine if i create app directly. I have a full repository available if someone can double check:

https://github.com/VirtoCommerce/vc-deployments/tree/3e831b31db71fe8f9867bd6257ea69c0560c9471

After more experiements, I was able to fix the issue by adding namespace under yaml definition in the apps folder

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: webstore-demo
  **namespace: argocd**

It has to have argocd namespace, anything else and it doesn't work. In my deployment I add everything into "demo" namespace and argocd runs under default argocd namespace.

[AustinSmart]

I have the same error w/ a Helm App of Apps. All application manifests specify namespace: argocd.

edit: for anyone who found themselves here while trying out the App of App patterns. If your child apps are deploying to a different cluster, make sure your _parent_ app has destination server: https://kubernetes.default.svc as that is where the child ArgoCD Application resources are created and tracked.

[FearTheBadger]

This is still a problem in 1.6.1

argocd version --client
  argocd: v1.6.1+159674e
  BuildDate: 2020-06-19T00:39:46Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
argocd-server version
  argocd-server: v1.6.1+159674e
  BuildDate: 2020-06-19T00:41:05Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
Kustomize version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
Kubectl Version:
  Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", 
  GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2019-10-31T21:06:55Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

The error from the application controller:

level=info msg="adding resource result, status: 'SyncFailed', phase: '', message: 'the server could not find the requested resource'" application=test kind=Kustomization name= namespace=test phase=Sync syncId=00008-PFwkm
level=info msg="Updating operation state. phase: Running -> Failed, message: '' -> 'one or more synchronization tasks are not valid'" application=test syncId=00009-cRIAm
level=info msg="sync/terminate complete" application=test duration=7.003573ms syncId=00009-cRIAm
level=info msg="updated 'test' operation (phase: Failed)"
level=info msg="Sync operation to 056d4f65824e28726b07ca417ed9452cdbd30db5 failed: one or more synchronization tasks are not valid" application=test dest-namespace=test dest-server="https://kubernetes.default.svc" reason=OperationCompleted type=Warning

[flickerfly]

One thing I've discovered in my research is the API endpoint yourargocd.com/api/v1/repositories/{repo}/apps. It appears that if if hit, with the problematic {repo} name, it would result in some context for where the issue manifests. It is document on your ArgoCD UI at the path yourargocd.com/swagger-ui#operation/ListApps.

I'm wondering, is it that ArgoCD isn't recognizing Kustomize is involved so trying to actually add the Kustomization type to the cluster rather than processing it? If so, how does ArgoCD get triggered on this?

In the interest of discovering that information, but not being a Golang guy, I found these hopefully relevant bits. There is apparently two ways this is supposed to be triggered. There is some documentation in the Tool Detection Doc, but it left me with some questions and clearly it isn't working exactly as documented.

_First,_ auto-detection based on the contents of the repo.

So what does the code use to determine if a directory of code should be handled as a Kustomize appType?
The answer appears to be on line 168 of kustomize.go.

var KustomizationNames = []string{"kustomization.yaml", "kustomization.yml", "Kustomization"}

This is then used on line 181 in the isKustomization function. It appears very consistent with the docs.

This info is used by reposerver/repository/repository.go on line 100 in the ListApps function. There is also a ListApps function in server/repository/repository.go on line 155, but this one doesn't use the discovery.Discover function, but instead calling the API ListApps endpoint.

_Second_, explicit identification in the Application resource as spec.source.kustomize. I'm not clear on what this should be if I just want to explicitly say this should use kustomize, but not change other characteristics. Maybe spec.source.kustomize: []? pkg/apis/application/v1alpha1/types.go line 2260 seems to be involved and best I can tell anything non-nil would be acceptable.

The two methods of detection appear to come together in reposerver/repository/repository.go function GetAppSourceType where it first checks ExplicitType, then if that comes back blank, uses discovery.AppType to tell GenerateManifests which type of application to output manifests for and hopefully hits line 378 where it gets processed by Kustomize.

My suspicion is that we aren't making it this far, but I don't know how to watch the code at work.

[FearTheBadger]

I think I have finally found the trigger point.

All kustomization.yaml files will be ignored, or kind: Kustomization configs will throw the error above, if the application source directory recurse: true is set. recurse: false or omitting recurse works without issue.

Failing scenario:

kind: Application
spec:
  source:
    directory:
      recurse: true

Successful scenario:

kind: Application
spec:
  source:
    directory:
      recurse: false

[dpennell]

edit: for anyone who found themselves here while trying out the App of App patterns. If your child apps are deploying to a different cluster, make sure your _parent_ app has destination server: https://kubernetes.default.svc as that is where the child ArgoCD Application resources are created and tracked.

This works for app of apps where all of the children are apps. If any of the children are simple resources (e.g. Secret), the end up being created in the local cluster and not the external one.