Pulumi-kubernetes: Pulumi preview/up --diff does not display diffs of properties in the same level of nesting if one references a fully-encrypted resource by its name

[bob-bins]

Pulumi preview/up --diff does not display diffs of properties in the same level of nesting if one references a fully-encrypted resource by its name (or likely any other of its properties).

Expected behavior

If I change a deployment image I expect to see:

      ~ spec: {
          ~ template: {
              ~ spec: {
                  ~ containers: [
                      ~ [0]: {
                            ~ image: "oldimagename" => "newimagename"
                            }
                    ]
                }
            }
        }

Current behavior

If I change a deployment image I actually see:

    ~ kubernetes:apps/v1:Deployment: (update)
      ~ spec: {
          ~ template: {
              ~ spec: {
                  ~ containers: [
                      ~ [0]: {
                            }
                    ]
                }
            }
        }

Steps to reproduce

Create these resources

const secret = new Secret("secret", {
  stringData: {
    LALALA: pulumi.secret("lalalala"),
  },
})

const deployment = new Deployment("nginx", {
  metadata: {
    name: "nginx",
  },
  spec: {
    replicas: 1,
    selector: {
      matchLabels: { app: "nginx" },
    },
    template: {
      metadata: {
        labels: { app: "nginx" },
      },
      spec: {
        containers: [
          {
            envFrom: [
              {
                secretRef: { name: secret.metadata.name },
              },
            ],
            env: [
              {
                name: "LOLOLO",
                value: "lolololo",
              },
            ],
            name: "nginx",
            image: "nginx",
          },
        ],
      },
    },
  },
})

Then change "lolololo" to any other string. See that the preview diff does not contain useful info.

Then remove secretRef: { name: secret.metadata.name },. Apply the change. Then change the env var in the deployment again and see that the diff now displays correctly

[lblackstone]

I think this issue may be specific to the k8s provider diffing, and could also be related to client-side preview.

@bob-bins Do you see the same behavior if you use the enableDryRun Provider flag?

[bob-bins]

same output when I enable that flag and run a preview after it is enabled

[bob-bins]

Final findings (I also updated the ticket for proper steps to reproduce):

• If a Deployment references a Secret (e.g. in envFrom) that is fully encrypted in the statefile (does not have the kubectl.kubernetes.io/last-applied-configuration annotation in plaintext), the Deployment's state at that level of nesting will also be encrypted even though it contains NO secret values. This is what causes the preview diff to be useless.
  
  
  
  • There is no good workaround for this. If you instead reference the secret by a hardcoded string, the statefile and diff will be correct, but then the deployment won't automatically replace the pods when values in the secret change
  
  
• This doesn't contribute toward the bug in this ticket but is a residual bug I came across: If you create a k8s resource without marking a value as a secret, and then later change it to a secret without changing any values in that resource, the statefile will correctly encrypt its value BUT it will NOT remove or encrypt kubectl.kubernetes.io/last-applied-configuration which will have it in plaintext. The user will need to make some change to the resource to remove that annotation from the statefile.

[lblackstone]

Ah, thanks for narrowing that down. I'm hoping that we will be able to fix this as a part of https://github.com/pulumi/pulumi-kubernetes/issues/1556