Argo-cd: Unable to delete application in K8s v1.12

[alexmojo]

Whem I'm trying to delete application I get following error from application-controller:

time="2018-10-24T13:51:28Z" level=info msg="Unable to delete application resources: invalid resource \"pods/proxy\": [may not contain '/']" application=jsreporter reason=StatusRefreshed type=Warning
time="2018-10-24T13:51:28Z" level=info msg="Deleting resources" application=jsreporter
time="2018-10-24T13:51:29Z" level=info msg="Unable to delete application resources: invalid resource \"services/proxy\": [may not contain '/']" application=jsreporter reason=StatusRefreshed type=Warning
time="2018-10-24T13:51:29Z" level=info msg="Deleting resources" application=jsreporter
time="2018-10-24T13:51:31Z" level=info msg="Unable to delete application resources: invalid resource \"services/proxy\": [may not contain '/']" application=jsreporter reason=StatusRefreshed type=Warning
time="2018-10-24T13:52:53Z" level=info msg="Deleting resources" application=jsreporter
time="2018-10-24T13:52:53Z" level=info msg="cache miss: apires|https://10.96.0.1:443"
time="2018-10-24T13:52:55Z" level=info msg="Unable to delete application resources: invalid resource \"nodes/proxy\": [may not contain '/']" application=jsreporter reason=StatusRefreshed type=Warning
time="2018-10-24T13:52:55Z" level=info msg="Deleting resources" application=jsreporter
time="2018-10-24T13:52:56Z" level=info msg="Unable to delete application resources: invalid resource \"nodes/proxy\": [may not contain '/']" application=jsreporter reason=StatusRefreshed type=Warning

Argocd version: v0.10.0
Kubernetes version: v.1.12

[alexmojo]

Little update. After quick research, I've found that everything works fine on kubernetes cluster version 1.10.4. Did I miss anything about kubernetes version compatibility in the docs?

[jessesuen]

It seems it may be related to v1.12 which we haven't tested with yet. Will investigate

[jessesuen]

I'm able to reproduce on v1.12. Should be able to figure out what's going on now.

[jessesuen]

Issue appears in the dynamic client. I'm attempting to update go-client to see if things help.
Related: https://github.com/argoproj/argo-cd/issues/353

[jessesuen]

I updated to client-go 9.0 and k8s APIs v1.12, however updating the client libraries did not improve anything.

So it appears in v1.12, the K8s API resource discovery API is starting to returning resources such as nodes/proxy and pods/proxy which ArgoCD cannot handle properly.

Notice that these resources do not support the list verb. What is happening is that we are blindly attempting to list resources of type pods/proxy, even though list is not supported on them. Either list was supported on them before, but not anymore, or these are new resources which we never handled properly. Whatever the case, we need to check if the list verb is supported on the resource before attempting to list+delete.

[kuznero]

This is not fixed yet, right? I see the same issue with latest argocd from master. (Still running pre-pulled server though). Is there any workaround to it at the moment?

[kuznero]

If someone can point me into the direction where to check for lists in the code I will be volunteering to producing a PR with the fix shortly. This is rather critical for us as we planned to start using ArgoCD in my team.Ping, @jessesuen

[jessesuen]

@kuznero this should be fixed in this commit:

https://github.com/argoproj/argo-cd/commit/2498f60c57b705cfe27105b832bba8ea210e03c3

However, this fix is only in master. I checked and it looks like the commit can be cherry-picked to release-0.10 without conflicts. Do you need a v0.10.5 release with this?

[jessesuen]

I think we can push out v0.10.5 patch with this fix, since v0.11 has some ways to go. There's also a UI bug that I think we should get out along with this.

[kuznero]

@jessesuen it would be great to get it into v0.10.5 if possible! Huge thanks!

[jessesuen]

Ok released v0.10.5 with the fix:

https://raw.githubusercontent.com/argoproj/argo-cd/release-0.10/manifests/install.yaml

[kuznero]

Ok released v0.10.5 with the fix:

https://raw.githubusercontent.com/argoproj/argo-cd/release-0.10/manifests/install.yaml

Great stuff! Thanks!

[kuznero]

@jessesuen not sure if this is related to v0.10.5, but here is what I get trying to deploy most trivial things (in kube-system ns, but still):

> argocd app sync admin
KIND  NAME  STATUS  HEALTH  HOOK  OPERATIONMSG

Application:        admin
Operation:          Sync
Phase:              Failed
Start:              2018-11-14 09:35:24 +0100 CET
Finished:           2018-11-14 09:35:36 +0100 CET
Duration:           12s
Message:            one or more objects failed to apply (dry run)

KIND            NAME        STATUS     HEALTH   HOOK  OPERATIONMSG
ServiceAccount  admin-user  OutOfSync  Missing        Error from server (Forbidden): error when retrieving current configuration of:
Resource: "/v1, Resource=serviceaccounts", GroupVersionKind: "/v1, Kind=ServiceAccount"
Name: "admin-user", Namespace: "kube-system"
Object: &{map["apiVersion":"v1" "kind":"ServiceAccount" "metadata":map["labels":map["applications.argoproj.io/app-name":"admin"] "name":"admin-user" "namespace":"kube-system" "annotations":map["kubectl.kubernetes.io/last-applied-configuration":""]]]}
from server for: "STDIN": serviceaccounts "admin-user" is forbidden: User "system:anonymous" cannot get resource "serviceaccounts" in API group "" in the namespace "kube-system"
ClusterRoleBinding  admin-user  OutOfSync  Missing       Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io "admin-user" is forbidden: User "system:anonymous" cannot get resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope

The requested helm chart only has service account and a role binding (to cluster-admin).

Am I missing something obvious?

[kuznero]

Just downgraded to v0.10.4 and deployments work fine there.