Argo-cd: Application is incorrectly reporting a diff

[jutley]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

From the ArgoCD web console, some of our Applications (same service, many clusters) are reporting being out of sync. The diff in the details shows that a couple values for container env vars need to be added, but the live state shows that they are already there.

When I uncheck "Hide default fields", the full diff disappears, even without selecting the compact view.

When I use the argo cli, it doesn't show any diff, even though the exit code indicates there is one. I'm removing specifics about our deployment in this example.

➜  argocd --grpc-web --server <server> app diff <app>
===== extensions/Deployment <namespace>/<deployment> ======

➜  echo $?
1

To Reproduce

Create an application that points to a path containing this manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        env:
        - name: VAR1
          value: var1
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
      restartPolicy: Always

Make sure the application is synced. Then, manually edit the deployment using kubectl, adding this as the _first_ environment variable:

        - name: VAR2
          value: var2

The Application should now show to be out of sync.

From what I can tell, there _must_ be an environment variable that uses valueFrom. I've done this with both the downward api and a configmap. Additionally, the new environment variable _must_ be the first variable.

Expected behavior

There should be no diff, and the app should report as synced.

Version

➜  argocd --grpc-web --server <server> version
argocd: v1.3.5+3e150df
  BuildDate: 2019-12-09T21:14:37Z
  GitCommit: 3e150df0ede862c476d2ad0b90717ae00a890551
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.3.5+3e150df
  BuildDate: 2019-12-09T21:14:05Z
  GitCommit: 3e150df0ede862c476d2ad0b90717ae00a890551
  GitTreeState: clean
  GoVersion: go1.12.6
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: Version: {Version:kustomize/v3.2.1 GitCommit:d89b448c745937f0cf1936162f26a5aac688f840 BuildDate:2019-09-27T00:10:52Z GoOs:linux GoArch:amd64}
  Helm Version: v2.15.2
  Kubectl Version: v1.14.0

[alexmt]

Thanks for bug report! I'm fixing it right now a part of https://github.com/argoproj/argo-cd/issues/2863 . Expect to get fix by end of day.

[jutley]

Ah, that's great news! I was working on isolating the bug, but if you think you already know the issue then I'll hold off.

[jutley]

@alexmt I've updated the ticket with steps to repro. It turns out, I think the Application _is_ out of sync. There are environment variables in the live Deployment that are not in source.

In most cases, this seems to not make the Application out of sync, but I think it should!

If the intended environment variables include one that uses valueFrom, and the live deployment has a different environment variable as the _first_ variable, then we get a really bizarre diff that does not make any sense.

Does this line up with what you've been looking into for #2863?

[alexmt]

Thank you for the steps to reproduce @jutley !

The diff indeed might be incorrect in this case. The logic which detects the change is working correctly but diff rendering is not right.

As usual, fix requires a little more work than I though :) . Still working on the fix.

[alexmt]

Implemented required changes but had to split fix into two parts:

• Part 1 closes https://github.com/argoproj/argo-cd/issues/2863
• To fix this issue we need to delete this one line: https://github.com/argoproj/argo-cd/pull/2869/files#diff-b9dc50535e8efccaea01a35f0bc05422R108 . Unfortunately, I've encountered an incorrect StatefulSet controller behavior. StatefulSet controller modifies volumeClaimTemplates field but that that field does not support strategic merge patch. So StatefulSet immediately became OutOfSync after running kubectl apply.

Going to create separate PR with part 2 and StatefulSet special handling on top and test it internally to make sure nothing else changes.

[jutley]

@alexmt Any update on this? Our organization is using ArgoCD significantly more, which is fantastic, but this bug is proving to be pretty disorienting for a lot of our engineers.

[alexmt]

Hello @jutley,

There was an unexpected delay (we had to do some internal support work). The fix of that bug is definitely important. We will work on it as soon as 1.5 is release is out. Will try to publish the fix as part of v1.5.1 patch release to get it out sooner.

[k]

I am trying to rely on sync status to determine whether I can cut a release or not (if the application is synced that means the integration test ran with all the correct configuration). However, this is preventing me from doing that in a reliable way. For now I have some hacky diffing customization but that's extremely brittle.

Any update on the timeline for this?

[cehoffman]

Encountering this issue with environment variable diffs as well. For my use case the deployment manifests, provided by another party, shows as synced and no diff after a normal deployment. In order to get visibility into the application and have operations support, we utilize appdynamics cluster operator to inject a agent into the deployment configuration. For most deployments this works fine and shows now diff from argocd, but in one case it shows a persistent diff for the environment variables.

In this case the diff for variables disappears when changing the _order_ of the variables from:

~yaml
- env:
- name: APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY
valueFrom:
secretKeyRef:
key: controller-key
name: cluster-agent-secret
- name: JAVA_TOOL_OPTIONS
value: '-Xmx3750m'
- name: SPRING_PROFILES_ACTIVE
value: basicAuth
- name: SPRING_RABBITMQ_USERNAME
value: guest
- name: SPRING_RABBITMQ_PASSWORD
value: guest
- name: SPRING_RABBITMQ_HOST
value: rabbitmq
- name: SPRING_DATA_MONGODB_HOST
value: mongo
- name: SPRING_DATA_MONGODB_USERNAME
valueFrom:
secretKeyRef:
key: mongodb-dbos-user
name: mongo
- name: SPRING_DATA_MONGODB_PASSWORD
valueFrom:
secretKeyRef:
key: mongodb-dbos-password
name: mongo
- name: SPRING_DATA_MONGODB_ADMIN_USER
valueFrom:
secretKeyRef:
key: mongo-initdb-root-username
name: mongo
- name: SPRING_DATA_MONGODB_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: mongo-initdb-root-password
name: mongo
- name: _JAVA_OPTIONS
value: ' -Dappdynamics.agent.accountAccessKey=$(APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY) -Dappdynamics.controller.hostName=carhartt.saas.appdynamics.com -Dappdynamics.controller.port=443 -Dappdynamics.controller.ssl.enabled=true -Dappdynamics.agent.accountName=carhartt -Dappdynamics.agent.applicationName=MSO-Prod -Dappdynamics.agent.tierName=dbos-master -Dappdynamics.agent.reuse.nodeName=true -Dappdynamics.agent.reuse.nodeName.prefix=dbos-master -Dappdynamics.socket.collection.bci.enable=true -javaagent:/opt/appdynamics-java/javaagent.jar'
- name: APPDYNAMICS_NETVIZ_AGENT_HOST
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: APPDYNAMICS_NETVIZ_AGENT_PORT
value: '3892'
~

to

~yaml
- env:
- name: JAVA_TOOL_OPTIONS
value: '-Xmx3750m'
- name: SPRING_PROFILES_ACTIVE
value: basicAuth
- name: SPRING_RABBITMQ_USERNAME
value: guest
- name: SPRING_RABBITMQ_PASSWORD
value: guest
- name: SPRING_RABBITMQ_HOST
value: rabbitmq
- name: SPRING_DATA_MONGODB_HOST
value: mongo
- name: SPRING_DATA_MONGODB_USERNAME
valueFrom:
secretKeyRef:
key: mongodb-dbos-user
name: mongo
- name: SPRING_DATA_MONGODB_PASSWORD
valueFrom:
secretKeyRef:
key: mongodb-dbos-password
name: mongo
- name: SPRING_DATA_MONGODB_ADMIN_USER
valueFrom:
secretKeyRef:
key: mongo-initdb-root-username
name: mongo
- name: SPRING_DATA_MONGODB_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: mongo-initdb-root-password
name: mongo
- name: APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY
valueFrom:
secretKeyRef:
key: controller-key
name: cluster-agent-secret
- name: _JAVA_OPTIONS
value: ' -Dappdynamics.agent.accountAccessKey=$(APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY) -Dappdynamics.controller.hostName=carhartt.saas.appdynamics.com -Dappdynamics.controller.port=443 -Dappdynamics.controller.ssl.enabled=true -Dappdynamics.agent.accountName=carhartt -Dappdynamics.agent.applicationName=MSO-Prod -Dappdynamics.agent.tierName=dbos-master -Dappdynamics.agent.reuse.nodeName=true -Dappdynamics.agent.reuse.nodeName.prefix=dbos-master -Dappdynamics.socket.collection.bci.enable=true -javaagent:/opt/appdynamics-java/javaagent.jar'
- name: APPDYNAMICS_NETVIZ_AGENT_HOST
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: APPDYNAMICS_NETVIZ_AGENT_PORT
value: '3892'
~

The change here is moving APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY from the first variable to later in the list after the item constant showing it is missing. The compact diff is:

There are several other deployments that have the same environment variables injected and only this one deployment shows an erroneous persistent diff. I see this issue in 1.5.7 and 1.6.0-rc2.