Argo-cd: Sync hangs with cert-manager on latest RC

Created on 16 Aug 2020  路  14Comments  路  Source: argoproj/argo-cd

Describe the bug
Trying to sync cert-manager on 1.7 RC causes ArgoCD to just hang without any progress being made. Even --loglevel debug shows no output.

To Reproduce

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.16.1/cert-manager.crds.yaml
kubectl create namespace cert-manager
argocd repo add https://charts.jetstack.io --type helm --name jetpack
argocd app create cert-manager --repo https://charts.jetstack.io --helm-chart cert-manager --revision v0.16.1 --dest-namespace cert-manager --dest-server https://kubernetes.default.svc
argocd app sync cert-manager

Expected behavior
Argocd should successfully sync.

Screenshots

If applicable, add screenshots to help explain your problem.

Version

argocd: v1.6.2+3d1f37b.dirty
  BuildDate: 2020-08-01T07:00:10Z
  GitCommit: 3d1f37b0c53f4c75864dc7339e2831c6e6a947e0
  GitTreeState: dirty
  GoVersion: go1.14.6
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.7.0-rc1+4728412
  BuildDate: 2020-08-15T19:31:00Z
  GitCommit: 4728412cc315437c08c660ea1e58d6de432a6e4a
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.17.8

Logs

 argocd app sync cert-manager --loglevel debug
TIMESTAMP  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE
2020-08-16T19:02:17+01:00                                ServiceAccount                  cert-manager  cert-manager-cainjector                  Running   Synced              serviceaccount/cert-manager-cainjector unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRole                     cert-manager  cert-manager-controller-clusterissuers   Running   Synced              clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
2020-08-16T19:02:17+01:00  admissionregistration.k8s.io  MutatingWebhookConfiguration    cert-manager  cert-manager-webhook                     Running   Synced              mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
2020-08-16T19:02:17+01:00   apps                         Deployment                      cert-manager  cert-manager-webhook                     Running   Synced              deployment.apps/cert-manager-webhook unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRoleBinding              cert-manager  cert-manager-controller-orders           Running   Synced              clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRoleBinding              cert-manager  cert-manager-controller-certificates     Running   Synced              clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
2020-08-16T19:02:17+01:00                                   Service                      cert-manager          cert-manager                     Running   Synced              service/cert-manager unchanged
2020-08-16T19:02:17+01:00   apps                         Deployment                      cert-manager  cert-manager-cainjector                  Running   Synced              deployment.apps/cert-manager-cainjector unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRole                     cert-manager  cert-manager-controller-challenges       Running   Synced              clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRole                     cert-manager  cert-manager-controller-ingress-shim     Running   Synced              clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
2020-08-16T19:02:17+01:00   apps                         Deployment                      cert-manager          cert-manager                     Running   Synced              deployment.apps/cert-manager unchanged
2020-08-16T19:02:17+01:00                                ServiceAccount                  cert-manager  cert-manager-webhook                     Running   Synced              serviceaccount/cert-manager-webhook unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRole                     cert-manager  cert-manager-controller-certificates     Running   Synced              clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRole                     cert-manager  cert-manager-cainjector                  Running   Synced              clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io           Role                      cert-manager  cert-manager-webhook:dynamic-serving     Running   Synced              role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRoleBinding              cert-manager  cert-manager-controller-issuers          Running   Synced              clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRoleBinding              cert-manager  cert-manager-controller-challenges       Running   Synced              clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRoleBinding              cert-manager  cert-manager-controller-clusterissuers   Running   Synced              clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     RoleBinding                     kube-system   cert-manager:leaderelection              Running   Synced              rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection configured
2020-08-16T19:02:17+01:00  admissionregistration.k8s.io  ValidatingWebhookConfiguration  cert-manager  cert-manager-webhook                     Running   Synced              validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook configured
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRole                     cert-manager  cert-manager-controller-orders           Running   Synced              clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRole                     cert-manager     cert-manager-edit                     Running   Synced              clusterrole.rbac.authorization.k8s.io/cert-manager-edit reconciled. clusterrole.rbac.authorization.k8s.io/cert-manager-edit unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRoleBinding              cert-manager  cert-manager-controller-ingress-shim     Running   Synced              clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io           Role                      kube-system   cert-manager:leaderelection              Running   Synced              role.rbac.authorization.k8s.io/cert-manager:leaderelection unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io           Role                      kube-system   cert-manager-cainjector:leaderelection   Running   Synced              role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     RoleBinding                     kube-system   cert-manager-cainjector:leaderelection   Running   Synced              rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection unchanged
2020-08-16T19:02:17+01:00                                ServiceAccount                  cert-manager          cert-manager                     Running   Synced              serviceaccount/cert-manager unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRole                     cert-manager  cert-manager-controller-issuers          Running   Synced              clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRole                     cert-manager     cert-manager-view                     Running   Synced              clusterrole.rbac.authorization.k8s.io/cert-manager-view reconciled. clusterrole.rbac.authorization.k8s.io/cert-manager-view unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     ClusterRoleBinding              cert-manager  cert-manager-cainjector                  Running   Synced              clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector unchanged
2020-08-16T19:02:17+01:00  rbac.authorization.k8s.io     RoleBinding                     cert-manager  cert-manager-webhook:dynamic-serving     Running   Synced              rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving configured
2020-08-16T19:02:17+01:00                                   Service                      cert-manager  cert-manager-webhook                     Running   Synced              service/cert-manager-webhook unchanged
**hangs**
bug cherry-pic1.7 core
Source
picture LalitMaganti
👍3

All 14 comments

We have the same issue since upgrading cert-manager to v0.16.1 (previous functioning cert-manager version was v0.13.1).
ArgoCD Version:

argocd version
argocd: v1.6.1+159674e
  BuildDate: 2020-06-19T00:39:46Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.6.1+159674e
  BuildDate: 2020-06-19T00:41:05Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.14.0

The App contains:

The App is in Auto-Sync (including prune and heal) mode.

The kustomize source can be applied to the cluster via _kubectl_ without problems.

However, the App in ArgoCD shows this:
Screenshot from 2020-08-18 15-13-13

Tried deleting the App (without Cascade) and apply it again. No change.

Executing

> argocd app sync core-cert-manager --loglevel debug
TIMESTAMP  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE

yields nothing (hangs).

Let me know when more information could be helpful.

Timoses picture Timoses on 18 Aug 2020

I'm fairly sure that the issue is the same that is mentioned in the cert-manager upgrade notes here.

kubectl versions with patch versions lower than v1.18.8 v1.17.11 or v1.16.14 have issues updating the v0.16 CRD files, due to a bug when handling deeply nested CRDs. This bug will make kubectl apply -f [...] hang.

The good news is that it should be solved simply by updating some dependencies since this is already solved in the newest patch versions of kubectl. There is no fix for helm yet though that I am aware of.

lentzi90 picture lentzi90 on 18 Aug 2020

Thank you @lentzi90 , we are using v1.18.6. Working on PR that upgrades the client to v1.18.8

alexmt picture alexmt on 18 Aug 2020
❤1

FWIW, my observation is the same as @Timoses. Interestingly, the kustomize version I'm using here is not reporting correctly below. The container image I'm using is cyrilix/argocd:v1.6.2 and the architecture is arm64 and not amd64, despite the fact that it does run perfectly fine on my raspi4.

Additionally, the cert-manager installation instructions suggest v1.19.0-rc.1 of kubectl is required.

argocd: v1.6.2+unknown
  BuildDate: 2020-08-01T04:24:49Z
  GitCommit:
  GitTreeState:
  GitTag: 1.6.2
  GoVersion: go1.14.6
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.6.2+dc2b9d0
  BuildDate: 2020-08-02T11:25:44Z
  GitCommit: dc2b9d0c96896f64a1d9a1db19a2fac78b6cce9e
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/arm64
  Ksonnet Version: vdev-2020-08-02T11:40:34+0000
  Kustomize Version: {Version:unknown GitCommit:$Format:%H$ BuildDate:1970-01-01T00:00:00Z GoOs:linux GoArch:arm64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.14.0

I also found that running:
kubectl -n argocd logs argocd-application-controller-6996f99c6c-d7w8z -f |grep warning yielded this:

time="2020-08-18T22:57:59Z" level=warning msg="Failed to convert resource: no kind \"CustomResourceDefinition\" is registered for version \"apiextensions.k8s.io/v1\" in scheme \"pkg/runtime/scheme.go:101\""
time="2020-08-18T22:58:00Z" level=warning msg="Failed to convert resource: no kind \"CustomResourceDefinition\" is registered for version \"apiextensions.k8s.io/v1\" in scheme \"pkg/runtime/scheme.go:101\""
time="2020-08-18T22:58:00Z" level=warning msg="Failed to convert resource: no kind \"CustomResourceDefinition\" is registered for version \"apiextensions.k8s.io/v1\" in scheme \"pkg/runtime/scheme.go:101\""

which also suggested that I need to add:

  ignoreDifferences:
  - group: apiextensions.k8s.io
    kind: CustomResourceDefinition
    jsonPointers:
    - /status

which I did, to no avail :frowning:

bradwood picture bradwood on 19 Aug 2020

For us the issue is not resolved.

Applied with apply only

> argocd app get core-cert-manager
Name:               core-cert-manager
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          argocd
URL:                http://localhost:8000/applications/core-cert-manager
Repo:               https://git-repository.int/kube.git
Target:             development
Path:               core/cert-manager/base
SyncWindow:         Sync Allowed
Sync Policy:        <none>
Sync Status:
Health Status:

> argocd app resources core-cert-manager --loglevel debug
FATA[0110] rpc error: code = Unavailable desc = transport is closing

> argocd app sync core-cert-manager --loglevel debug
TIMESTAMP  GROUP        KIND   NAMESPACE                  NAME    STATUS   HEALTH        HOOK  MESSAGE
time="2020-08-27T12:49:30+02:00" level=warning msg="watch err: rpc error: code = Unavailable desc = transport is closing"
time="2020-08-27T12:50:52+02:00" level=warning msg="watch err: rpc error: code = Unavailable desc = transport is closing"
time="2020-08-27T12:52:13+02:00" level=warning msg="watch err: rpc error: code = Unavailable desc = transport is closing"
time="2020-08-27T12:53:35+02:00" level=warning msg="watch err: rpc error: code = Unavailable desc = transport is closing"
time="2020-08-27T12:54:56+02:00" level=warning msg="watch err: rpc error: code = Unavailable desc = transport is closing"
...

When I apply the app without apply only it is stuck:
Screenshot from 2020-08-27 13-57-28

Trying to sync now:

> argocd app sync core-cert-manager --loglevel debug
FATA[0001] rpc error: code = FailedPrecondition desc = another operation is already in progress
Timoses picture Timoses on 27 Aug 2020
👍1

Still seeing it i v1.7.1 unfortunately. :disappointed:
I also realized that the application controller Pod is completely hogging the CPU while this is happening.

These issues seem very similar/related:

lentzi90 picture lentzi90 on 27 Aug 2020
👍1

Thank you for verifying it @lentzi90 , @Timoses ! Just published v1.7.2 that fixes the bug (this time for sure fix it). Can you please check if issue is resolved for you ?

alexmt picture alexmt on 28 Aug 2020

Cert-manager itself works now, the problem is I now get missing health issues with ClusterIssuers and Certificates.

mhaddon picture mhaddon on 29 Aug 2020

Working for me now! Thanks for fixing!

lentzi90 picture lentzi90 on 30 Aug 2020

and for me! :beer:

bradwood picture bradwood on 30 Aug 2020

Great, it does fix the issue. The app is now displayed as all green.

It displays some warnings though:
Screenshot from 2020-09-01 12-35-43

Timoses picture Timoses on 1 Sep 2020

Note to my previous comment:
I upgraded cert-manager while Auto-Sync was off.
After upgrade I did notice that ArgoCD said "Out of Sync" and that the diff displayed the following label to be different:

- app.kubernetes.io/instance: cert-manager
+ app.kubernetes.io/instance: core-cert-manager

My cert-manager application in ArgoCD itself is named core-cert-manager.
There CRDs themselves still have the cert-manager instance name, the others (as far as I can tell) all have core-cert-manager now.

Timoses picture Timoses on 1 Sep 2020

I had a similar issue for the last day or two. I was passing extra config to cert-manager to install into the cert-manager namespace. I was constantly get errors about cluster resources not being installable.

Screenshot from 2020-10-06 09-58-05

Resulting in this:
Screenshot from 2020-10-06 09-57-53

The issue was that on that particular cluster, RBAC was correct in argocd, but I had not given cluster level permissions. I had scoped only to specific namespaces.
I had to re-add the cluster without the namespace limitation for this to work. It had worked in that setup in our staging environment.. but I had not noted to difference of the cluster being added in argocd. in case anyone happens across this limitation.

Not sure if that's a bug or intended? ArgoCD not being able to install cluster level resources with namespaced setup.

hawksight picture hawksight on 6 Oct 2020

The namespaced mode feature was designed to enable managing resources with namespace permissions only. If namespaces are specified in cluster configuration then Argo CD assumes it cannot manage cluster level resources.

However, it might be useful from a performance perspective to connect several namespaces and cluster level resources. We would need to add a cluster setting that enables/disables cluster level resources. @hawksight could you please create feature request to support it?

alexmt picture alexmt on 7 Oct 2020
👀1 👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

rosscdh picture rosscdh  路  3Comments
ksaito1125 picture ksaito1125  路  3Comments
travis-sobeck picture travis-sobeck  路  3Comments
KarstenSiemer picture KarstenSiemer  路  3Comments
turbotankist picture turbotankist  路  3Comments