Pulumi-kubernetes: Update fails when no changes are really required (kube stripping empty array?)

[oliverholliday]

A recent behaviour change (possibly related to updating kube?) is that pulumi keeps wanting to re-deploy a service.

If I run a preview --diff I can see that pulumi wants to add some empty arrays for volumeMounts and volumes to the deployment. I suspect kubernetes is then stripping these out and on pass number two pulumi sees a change and wants to redeploy.

We use a single block of code to generate the deployment spec for all our apps and this particular one doesn't need volumes so the array is empty.

The main bug here is that kube doesn't seem to agree that there has even been a change and so the deployment fails after 10 minutes with:

    error: Plan apply failed: 2 errors occurred:
        * Timeout occurred for 'app-ip262zey'
        * Attempted to roll forward to new ReplicaSet, but minimum number of Pods did not become live

Snipped diff here:

   ~ kubernetes:apps/v1:Deployment: (update)
      ~ spec: {
                               SNIPPED
          ~ template: {
                metadata: {
                               SNIPPED
                }
              ~ spec    : {
                  ~ containers                   : [
                      ~ [0]: {
                               SNIPPED
                              + volumeMounts: []
                            }
                    ]
                  + volumes                      : []
                }
            }
        }

[lblackstone]

@oliverholliday Can you provide more details on the resource definition you're passing to Pulumi? I can't tell what's causing the problem from the diff.

[oliverholliday]

import * as pulumi from "@pulumi/pulumi";
import * as kubernetes from "@pulumi/kubernetes";
import * as kubernetesInput from "@pulumi/kubernetes/types/input";

export type VolumeMount = {
    volume: kubernetesInput.core.v1.Volume,
    volumeMount: kubernetesInput.core.v1.VolumeMount
};

export type Config = Partial<AppConfig>;

export type AppConfig = {
    InternalPort: number,
    ExternalPort: number,
    Replicas: number,
    TerminationGracePeriodSeconds: number,
    LivenessProbe: pulumi.Input<kubernetesInput.core.v1.Probe> | undefined
    ReadinessProbe: pulumi.Input<kubernetesInput.core.v1.Probe> | undefined,
    Env: pulumi.Input<kubernetesInput.core.v1.EnvVar>[] | undefined,
    Mounts: VolumeMount[]
};

export const AppDefaults: AppConfig = {
    InternalPort: 80,
    ExternalPort: 80,
    Replicas: 1,
    TerminationGracePeriodSeconds: 300,
    LivenessProbe: undefined,
    ReadinessProbe: undefined,
    Env: undefined,
    Mounts: []
};

export const installApp = (
    namespace: pulumi.Input<string>,
    name: string,
    image: pulumi.Input<string>,
    configuration: Config | undefined = undefined
): AppResources => {

    let config = { ...AppDefaults, ...configuration };

    const labels = { name: name };

    const deployment = new kubernetes.apps.v1.Deployment(name, {
        metadata: { namespace: namespace, labels: labels, },
        spec: {
            selector: { matchLabels: labels },
            replicas: config.Replicas,
            template: {
                metadata: { labels: labels },
                spec: {
                    terminationGracePeriodSeconds: config.TerminationGracePeriodSeconds,
                    containers: [{
                        name: name,
                        image: image,
                        ports: [{ containerPort: config.InternalPort }],
                        volumeMounts: config.Mounts.map(x => x.volumeMount),
                        livenessProbe: config.LivenessProbe,
                        readinessProbe: config.ReadinessProbe,
                        env: config.Env
                    }],
                    volumes: config.Mounts.map(x => x.volume)
                }
            }
        }
    });

    return deployment;
}

installApp(
    namespace: "default",
    name: "app",
    image: "app:latest"
):

[oliverholliday]

If I delete the deployment using kubectl then re-run pulumi, the resource is deployed correctly. When I re-run pulumi the second pass wants to redeploy it with the diff above, and it fails.

Cheers

[oliverholliday]

FYI in case it helps:

I worked around it by changing the config.Mounts.map for config.Mounts === 0 ? undefined : config.Mounts.map in both the above places. This worked as expected and repeated deployments proceed correctly now.

What's interesting is that on the first run the preview diff said it was going to replace the resource, showed no differences at all in the diff output and then it didn't actually do anything during the update.

[lblackstone]

I was able to reproduce on the 0.25.1 release, but it appears to be fixed on the 0.25.2 release.

new k8s.apps.v1.Deployment("foo", {
    spec: {
        selector: {matchLabels: appLabels},
        replicas: 1,
        template: {
            metadata: {labels: appLabels},
            spec: {
                containers: [
                    {name: "nginx", image: "nginx", ports: [{containerPort: 80}]}
                ],
                volumes: []
            }
        }
    }
});

0.25.1

      ~ spec: {
          ~ template: {
              ~ spec    : {
                  + volumes   : []
                }
            }
        }

0.25.2

Resources:
    2 unchanged

Can you try with 0.25.2 and see if that fixes it for you?

[oliverholliday]

Yeah works for me, great!

Thanks Levi

[oliverholliday]

I'm getting this same issue again with using [] for volumes now but only on Windows! Using the same stack state file on linux and it shows no changes.

I've also started using resourcequotas and kube rewrites 1000m to 1 which causes a similar issue.

[lblackstone]

We've got a couple diff fixes in master that will be released early this week. You can try with @pulumi/kubernetes: dev and see if that fixes it for you in the meantime.

[lblackstone]

https://github.com/pulumi/pulumi-kubernetes/releases/tag/v0.25.3 is out now