Argo-cd: Still display OutOfSync in the UI when the changes are applied in argocd

[anggao]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a
question in argocd slack channel.

Checklist:

• [ *] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ *] I've included steps to reproduce the bug.
• [ ] I've pasted the output of argocd version.

Describe the bug

We upgraded to argo-cd v1.7 yesterday, I added a new entry to a configmap and committed the changes to a branch.
Then I switched argo-cd app to use that branch. I saw configmap got updated. but the status of the app still in OutOfSync.

To Reproduce

1. Add a new entry to an existing configmap
2. Commit changes to a branch
3. Change argo-cd app to use the branch
4. Check live manifest confirm changes are applied.
5. But still have OutOfSync in the UI.

Expected behavior
After changes applied, the app should be in sync status.

Version
v1.7

[alexmt]

Hello @anggao , Can you please provide more information to help to reproduce it? If you attach the diff it would help.

[anggao]

@alexmt The change is really simple, like the following, I just added a new entry in the configmap for one of our existing app.

  apiVersion: kustomize.config.k8s.io/v1beta1
  kind: Kustomization

  bases:
    - ../../base

  generatorOptions:
    disableNameSuffixHash: true
  configMapGenerator:
    - name: default-config
      literals:
+     - egress.endpoint="http://v-seservice.XXX.XXX.com"
      - model.log.prediction.percent="0.0"
      - model.log.feature.percent="0.0"

[alexmt]

Thank you! Can you please share which diff Argo CD has detected in config map as well ? Meanwhile trying to reproduce the bug using your kustomization.yaml.

[anggao]

This is what I got from the argo-cd diff

[anggao]

I can see both live manifest and desired manifest looks right and same, but somehow the diff contains above changes

[thomasrepnik]

We discovered the same Problem with docker image versions:

Image Version on Cluster 1.0.64
Image Version in GIT Repository 1.0.64
==> Everything was synced correctly

But ArgoCD is Out of Sync, because ArgoCD somehow thinks that the deployed image Version on the cluster is still 1.0.63 (which is definitely not the case)

Live Manifests and Desired Manifests both show Version 1.0.64 (which is correct)

[whitleykeith]

I've been seeing the same issue. It's usually resolved by restarting the application-controller

[ghost]

Seeing this too on simple helm chart update. Live and Desired labels are ok and updated but ArgoCD claims there is a diff:

Restart of the application controller did 'fix' it, but only for that one sync after restart.
We are on 1.7.2

[whitleykeith]

We're on 1.7.2 as well. We're also seeing a similar issue with the Health status of certain resources with custom healthchecks. Even when a resource's manifest should render a Healthy state, Argo leaves it in whatever state is was in before (Progressing, Missing, etc.).

Similarly, we can solve this issue by restarting the application controller. That makes me think it has something to do with how the application controller stores the app's state. Looking into the history I do see a lot of changes around the app controller so I'm thinking the error is somewhere within the controller logic.

I'm definitely not an expert on the inner workings of the Argo app controller so take my insight with a hefty pile of salt :)

[AndreiBamboi]

We're on 1.7.2 as well. We're also seeing a similar issue with the Health status of certain resources with custom healthchecks. Even when a resource's manifest _should_ render a Healthy state, Argo leaves it in whatever state is was in before (Progressing, Missing, etc.).

Similarly, we can solve this issue by restarting the application controller. That makes me think it has something to do with how the application controller stores the app's state. Looking into the history I do see a lot of changes around the app controller so I'm thinking the error is somewhere within the controller logic.

I'm definitely not an expert on the inner workings of the Argo app controller so take my insight with a hefty pile of salt :)

I do have the same issue on my project. Deleted the app from GUI/kubectl delete apps and recreating it with the same state was in before deletion. Also some resources appear healthy but they dont exist anymore. Restarting the controller fixed it.

[posquit0]

I have the same issue after updating to v1.7.2.

[mhaddon]

I too have the same issue, but after ~8 hours or something everything is fine.
Obviously it would be nice if things were fine instantly.

This is on 1.7.2.

[hamza-tumturk]

I’m facing the same issue after upgrading from v1.6.0 to v1.7.2.
Using the Invalidate cache under Settings/Cluster/ClusterName seems sort out the OutOfSync issue for a while. Also, after doing that, it is still leaving dead bodies behind from previous deployment revisions.

[jessesuen]

Lets dup this to https://github.com/argoproj/argo-cd/issues/4053 where the attempted fix was applied.

[thomasrepnik]

Will we have to wait for the fix until version 1.8.0 ist released?
Because we had no problems prior to 1.7.0 it would be nice to see it in a 1.7.x bugfix release.

[whitleykeith]

@thomasrepnik looks like they just released 1.7.3 a few hours ago which has this fix in it.

[mark-rushakoff]

We are still hitting this on 1.7.3 with a proprietary CRD. It worked fine when we are on Argo 1.4, 1.5, and 1.6.

Is there any debugging information I can share to help out in getting a proper fix @jessesuen @alexmt?