Argo-cd: Prometheus Operator Custom Resource Definitions won't sync correctly

[cookandy]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Prometheus Operator custom resource definitions will not sync. I have resynced the CRDs many times, but they still have differences in the status field, causing the application to show OutOfSync. I have tried to remove and reinstall the application, but the issue remains.

To Reproduce

Install the Prometheus Operator 8.12.1 Helm chart.

Expected behavior

All resources including the CRDs should be synced. Instead, there are differences in the status field and the CRDs show unsynced.

Screenshots

Applicaiton status:

Differences in CRD:

Version

argocd: v1.3.6+89be1c9
  BuildDate: 2019-12-10T22:48:19Z
  GitCommit: 89be1c9ce6db0f727c81277c1cfdfb1e385bf248
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.4.2+48cced9
  BuildDate: 2020-01-24T01:07:03Z
  GitCommit: 48cced9d925b5bc94f6aa9fa4a8a19b2a59e128a
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: Version: {Version:kustomize/v3.2.1 GitCommit:d89b448c745937f0cf1936162f26a5aac688f840 BuildDate:2019-09-27T00:10:52Z GoOs:linux GoArch:amd64}
  Helm Version: v2.15.2
  Kubectl Version: v1.14.0

Logs

argocd-application-controller logs:

time="2020-03-16T19:05:41Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=apps
time="2020-03-16T19:05:41Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=apps
time="2020-03-16T19:05:41Z" level=info msg="updated 'prometheus-operator' operation (phase: Running)"
time="2020-03-16T19:05:41Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:a5775288ce0e5824ed61f33c7f6aec77f5829f8e,Prune:false,DryRun:false,SyncStrategy:&SyncStrategy{Apply:nil,Hook:&SyncStrategyHook{SyncStrategyApply:SyncStrategyApply{Force:false,},},},Resources:[]SyncOperationResource{SyncOperationResource{Group:admissionregistration.k8s.io,Kind:MutatingWebhookConfiguration,Name:prometheus-operator-admission,},SyncOperationResource{Group:admissionregistration.k8s.io,Kind:ValidatingWebhookConfiguration,Name:prometheus-operator-admission,},SyncOperationResource{Group:apiextensions.k8s.io,Kind:CustomResourceDefinition,Name:alertmanagers.monitoring.coreos.com,},SyncOperationResource{Group:apiextensions.k8s.io,Kind:CustomResourceDefinition,Name:podmonitors.monitoring.coreos.com,},SyncOperationResource{Group:apiextensions.k8s.io,Kind:CustomResourceDefinition,Name:prometheuses.monitoring.coreos.com,},SyncOperationResource{Group:apiextensions.k8s.io,Kind:CustomResourceDefinition,Name:prometheusrules.monitoring.coreos.com,},SyncOperationResource{Group:apiextensions.k8s.io,Kind:CustomResourceDefinition,Name:servicemonitors.monitoring.coreos.com,},SyncOperationResource{Group:apiextensions.k8s.io,Kind:CustomResourceDefinition,Name:thanosrulers.monitoring.coreos.com,},},Source:nil,Manifests:[],}}" application=prometheus-operator
time="2020-03-16T19:05:41Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: monitoring)" application=prometheus-operator
time="2020-03-16T19:05:41Z" level=info msg="Skipping auto-sync: application status is Synced" application=apps
time="2020-03-16T19:05:41Z" level=info msg="Update successful" application=apps
time="2020-03-16T19:05:41Z" level=info msg="Reconciliation completed" application=apps dest-namespace=default dest-server="https://kubernetes.default.svc" fields.level=1 time_ms=142.389228
time="2020-03-16T19:05:41Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=apps
time="2020-03-16T19:05:41Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=apps
time="2020-03-16T19:05:41Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=master-app
time="2020-03-16T19:05:41Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=master-app
time="2020-03-16T19:05:41Z" level=info msg="Skipping auto-sync: application status is Synced" application=master-app
time="2020-03-16T19:05:41Z" level=info msg="Update successful" application=master-app
time="2020-03-16T19:05:42Z" level=info msg="Reconciliation completed" application=master-app dest-namespace=default dest-server="https://kubernetes.default.svc" fields.level=1 time_ms=70.10682999999999
time="2020-03-16T19:05:42Z" level=info msg="Skipping auto-sync: application status is Synced" application=apps
time="2020-03-16T19:05:42Z" level=info msg="Update successful" application=apps
time="2020-03-16T19:05:42Z" level=info msg="Reconciliation completed" application=apps dest-namespace=default dest-server="https://kubernetes.default.svc" fields.level=1 time_ms=208.769155
time="2020-03-16T19:05:42Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=master-app
time="2020-03-16T19:05:42Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=master-app
time="2020-03-16T19:05:42Z" level=info msg="Skipping auto-sync: application status is Synced" application=master-app
time="2020-03-16T19:05:42Z" level=info msg="Update successful" application=master-app
time="2020-03-16T19:05:42Z" level=info msg="Reconciliation completed" application=master-app dest-namespace=default dest-server="https://kubernetes.default.svc" fields.level=1 time_ms=60.814357
time="2020-03-16T19:05:42Z" level=info msg=syncing application=prometheus-operator isSelectiveSync=true skipHooks=true started=false syncId=00019-AtiWv
time="2020-03-16T19:05:42Z" level=info msg=tasks application=prometheus-operator isSelectiveSync=true syncId=00019-AtiWv tasks="[Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:monitoring/alertmanagers.monitoring.coreos.com obj->obj (,,), Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:monitoring/podmonitors.monitoring.coreos.com obj->obj (,,), Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:monitoring/prometheuses.monitoring.coreos.com obj->obj (,,), Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:monitoring/prometheusrules.monitoring.coreos.com obj->obj (,,), Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:monitoring/servicemonitors.monitoring.coreos.com obj->obj (,,), Sync/0 resource apiextensions.k8s.io/CustomResourceDefinition:monitoring/thanosrulers.monitoring.coreos.com obj->obj (,,), Sync/0 resource admissionregistration.k8s.io/ValidatingWebhookConfiguration:monitoring/prometheus-operator-admission obj->obj (,,), Sync/0 resource admissionregistration.k8s.io/MutatingWebhookConfiguration:monitoring/prometheus-operator-admission obj->obj (,,)]"
time="2020-03-16T19:05:42Z" level=info msg="Applying resource CustomResourceDefinition/thanosrulers.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:42Z" level=info msg="Applying resource CustomResourceDefinition/alertmanagers.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:42Z" level=info msg="Applying resource CustomResourceDefinition/prometheuses.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:42Z" level=info msg="Applying resource CustomResourceDefinition/podmonitors.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:42Z" level=info msg="Applying resource CustomResourceDefinition/prometheusrules.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:42Z" level=info msg="Applying resource CustomResourceDefinition/servicemonitors.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:43Z" level=info msg="Applying resource ValidatingWebhookConfiguration/prometheus-operator-admission in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:43Z" level=info msg="Applying resource MutatingWebhookConfiguration/prometheus-operator-admission in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:43Z" level=info msg="Updating operation state. phase: Running -> Running, message: '' -> 'one or more tasks are running'" application=prometheus-operator syncId=00019-AtiWv
time="2020-03-16T19:05:43Z" level=info msg="Applying resource CustomResourceDefinition/thanosrulers.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:43Z" level=info msg="Applying resource CustomResourceDefinition/alertmanagers.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:43Z" level=info msg="Applying resource CustomResourceDefinition/podmonitors.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:43Z" level=info msg="Applying resource CustomResourceDefinition/servicemonitors.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:43Z" level=info msg="Applying resource CustomResourceDefinition/prometheusrules.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:43Z" level=info msg="Applying resource CustomResourceDefinition/prometheuses.monitoring.coreos.com in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:44Z" level=info msg="adding resource result, status: 'Synced', phase: 'Running', message: 'customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com configured'" application=prometheus-operator kind=CustomResourceDefinition name=prometheusrules.monitoring.coreos.com namespace=monitoring phase=Sync syncId=00019-AtiWv
time="2020-03-16T19:05:44Z" level=info msg="adding resource result, status: 'Synced', phase: 'Running', message: 'customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com configured'" application=prometheus-operator kind=CustomResourceDefinition name=servicemonitors.monitoring.coreos.com namespace=monitoring phase=Sync syncId=00019-AtiWv
time="2020-03-16T19:05:44Z" level=info msg="adding resource result, status: 'Synced', phase: 'Running', message: 'customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com configured'" application=prometheus-operator kind=CustomResourceDefinition name=podmonitors.monitoring.coreos.com namespace=monitoring phase=Sync syncId=00019-AtiWv
time="2020-03-16T19:05:44Z" level=info msg="adding resource result, status: 'Synced', phase: 'Running', message: 'customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com configured'" application=prometheus-operator kind=CustomResourceDefinition name=alertmanagers.monitoring.coreos.com namespace=monitoring phase=Sync syncId=00019-AtiWv
time="2020-03-16T19:05:44Z" level=info msg="adding resource result, status: 'Synced', phase: 'Running', message: 'customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com configured'" application=prometheus-operator kind=CustomResourceDefinition name=thanosrulers.monitoring.coreos.com namespace=monitoring phase=Sync syncId=00019-AtiWv
time="2020-03-16T19:05:44Z" level=info msg="adding resource result, status: 'Synced', phase: 'Running', message: 'customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com configured'" application=prometheus-operator kind=CustomResourceDefinition name=prometheuses.monitoring.coreos.com namespace=monitoring phase=Sync syncId=00019-AtiWv
time="2020-03-16T19:05:44Z" level=info msg="Applying resource ValidatingWebhookConfiguration/prometheus-operator-admission in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:44Z" level=info msg="adding resource result, status: 'Synced', phase: 'Running', message: 'validatingwebhookconfiguration.admissionregistration.k8s.io/prometheus-operator-admission configured'" application=prometheus-operator kind=ValidatingWebhookConfiguration name=prometheus-operator-admission namespace=monitoring phase=Sync syncId=00019-AtiWv
time="2020-03-16T19:05:44Z" level=info msg="Applying resource MutatingWebhookConfiguration/prometheus-operator-admission in cluster: https://172.20.0.1:443, namespace: monitoring"
time="2020-03-16T19:05:44Z" level=info msg="adding resource result, status: 'Synced', phase: 'Running', message: 'mutatingwebhookconfiguration.admissionregistration.k8s.io/prometheus-operator-admission configured'" application=prometheus-operator kind=MutatingWebhookConfiguration name=prometheus-operator-admission namespace=monitoring phase=Sync syncId=00019-AtiWv
time="2020-03-16T19:05:44Z" level=info msg="Updating operation state. phase: Running -> Succeeded, message: 'one or more tasks are running' -> 'successfully synced (all tasks run)'" application=prometheus-operator syncId=00019-AtiWv
time="2020-03-16T19:05:44Z" level=info msg="sync/terminate complete" application=prometheus-operator duration=1.923969873s syncId=00019-AtiWv
time="2020-03-16T19:05:44Z" level=info msg="updated 'prometheus-operator' operation (phase: Succeeded)"
time="2020-03-16T19:05:44Z" level=info msg="Partial sync operation to a5775288ce0e5824ed61f33c7f6aec77f5829f8e succeeded" application=prometheus-operator dest-namespace=monitoring dest-server="https://kubernetes.default.svc" reason=OperationCompleted type=Normal
time="2020-03-16T19:05:44Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=apps
time="2020-03-16T19:05:44Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=apps
time="2020-03-16T19:05:44Z" level=info msg="Refreshing app status (controller refresh requested), level (2)" application=prometheus-operator
time="2020-03-16T19:05:44Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: monitoring)" application=prometheus-operator
time="2020-03-16T19:05:44Z" level=info msg="Skipping auto-sync: application status is Synced" application=apps
time="2020-03-16T19:05:44Z" level=info msg="Update successful" application=apps
time="2020-03-16T19:05:44Z" level=info msg="Reconciliation completed" application=apps dest-namespace=default dest-server="https://kubernetes.default.svc" fields.level=1 time_ms=90.013518
time="2020-03-16T19:05:44Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=master-app
time="2020-03-16T19:05:44Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=master-app
time="2020-03-16T19:05:44Z" level=info msg="Skipping auto-sync: application status is Synced" application=master-app
time="2020-03-16T19:05:44Z" level=info msg="Update successful" application=master-app
time="2020-03-16T19:05:44Z" level=info msg="Reconciliation completed" application=master-app dest-namespace=default dest-server="https://kubernetes.default.svc" fields.level=1 time_ms=40.681895000000004
time="2020-03-16T19:05:46Z" level=info msg="Skipping auto-sync: most recent sync already to a5775288ce0e5824ed61f33c7f6aec77f5829f8e" application=prometheus-operator
time="2020-03-16T19:05:46Z" level=info msg="Update successful" application=prometheus-operator
time="2020-03-16T19:05:46Z" level=info msg="Reconciliation completed" application=prometheus-operator dest-namespace=monitoring dest-server="https://kubernetes.default.svc" fields.level=2 time_ms=1832.6944720000001
time="2020-03-16T19:05:46Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=apps
time="2020-03-16T19:05:46Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=apps
time="2020-03-16T19:05:46Z" level=info msg="Skipping auto-sync: application status is Synced" application=apps
time="2020-03-16T19:05:46Z" level=info msg="Refreshing app status (controller refresh requested), level (1)" application=master-app
time="2020-03-16T19:05:46Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=master-app
time="2020-03-16T19:05:46Z" level=info msg="Update successful" application=apps
time="2020-03-16T19:05:46Z" level=info msg="Reconciliation completed" application=apps dest-namespace=default dest-server="https://kubernetes.default.svc" fields.level=1 time_ms=119.16211899999999
time="2020-03-16T19:05:46Z" level=info msg="Skipping auto-sync: application status is Synced" application=master-app
time="2020-03-16T19:05:46Z" level=info msg="Update successful" application=master-app
time="2020-03-16T19:05:46Z" level=info msg="Reconciliation completed" application=master-app dest-namespace=default dest-server="https://kubernetes.default.svc" fields.level=1 time_ms=44.645311

argocd-server logs:

time="2020-03-16T19:02:43Z" level=info msg="received unary call /application.ApplicationService/Sync" grpc.method=Sync grpc.request.claims="{\"at_hash\":\"7sjBYoFGbUri3YRzSNVWoQ\",\"aud\":\"argo-cd\",\"email\":\"xxx\",\"email_verified\":true,\"exp\":1584470406,\"groups\":[\"xxx:team-Operations\"],\"iat\":1584384006,\"iss\":\"https://xxxx/api/dex\",\"name\":\"user_xx\",\"preferred_username\":\"user_xx\",\"sub\":\"xxx\"}" grpc.request.content="name:\"prometheus-operator\" revision:\"develop\" dryRun:false prune:false strategy:<hook:<syncStrategyApply:<force:false > > > resources:<group:\"admissionregistration.k8s.io\" kind:\"MutatingWebhookConfiguration\" name:\"prometheus-operator-admission\" > resources:<group:\"admissionregistration.k8s.io\" kind:\"ValidatingWebhookConfiguration\" name:\"prometheus-operator-admission\" > resources:<group:\"apiextensions.k8s.io\" kind:\"CustomResourceDefinition\" name:\"alertmanagers.monitoring.coreos.com\" > resources:<group:\"apiextensions.k8s.io\" kind:\"CustomResourceDefinition\" name:\"podmonitors.monitoring.coreos.com\" > resources:<group:\"apiextensions.k8s.io\" kind:\"CustomResourceDefinition\" name:\"prometheuses.monitoring.coreos.com\" > resources:<group:\"apiextensions.k8s.io\" kind:\"CustomResourceDefinition\" name:\"prometheusrules.monitoring.coreos.com\" > resources:<group:\"apiextensions.k8s.io\" kind:\"CustomResourceDefinition\" name:\"servicemonitors.monitoring.coreos.com\" > resources:<group:\"apiextensions.k8s.io\" kind:\"CustomResourceDefinition\" name:\"thanosrulers.monitoring.coreos.com\" > " grpc.service=application.ApplicationService grpc.start_time="2020-03-16T19:02:43Z" span.kind=server system=grpc
time="2020-03-16T19:02:43Z" level=info msg="xxx initiated partial sync to develop (a5775288ce0e5824ed61f33c7f6aec77f5829f8e)" application=prometheus-operator dest-namespace=monitoring dest-server="https://kubernetes.default.svc" reason=OperationStarted type=Normal
time="2020-03-16T19:02:43Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Sync grpc.service=application.ApplicationService grpc.start_time="2020-03-16T19:02:43Z" grpc.time_ms=144.115 span.kind=server system=grpc

[jessesuen]

This appears to be a problem in the chart. The chart seems to be applying empty values in the status, and Argo CD is correctly identifying a configuration difference. I find it very odd that the prometheus chart would even include status as part of the deployed yaml.

[cookandy]

Hi @jessesuen — thanks for the reply. You appear to be correct. Here's the related GHI https://github.com/coreos/prometheus-operator/issues/3022

Closing!

[OKyHb]

Can you suggest any workarounds? How "prometheus-operator" can be installed via argocd?

[rdelpret]

I'm seeing this with the jaeger operator and the Prometheus operator as well. (I am not using helm for either)

[rdelpret]

Note: this used to work and started in v1.5.2

[nipunchawla-bc]

@rdelpret just remove the status field from the crd, it would sync