Pulumi-kubernetes: Changing a ConfigMap's data causes a replacement

[pierlucg-xs]

Changing a ConfigMap's data causes a replacement of that resource, causing other resources that depend on it to be replaced as well (e.g. Deployment using envFrom), causing downtime.

Expected behavior

The ConfigMap should be updated in place.

Steps to reproduce

Running pulumi up twice with the following code:

const testConfigMap = new k8s.core.v1.ConfigMap(
  'test',
  {
    data: {foo: `${Date.now()}`},
  }
);

will result in a replacement of that configMap
+- └─ kubernetes:core/v1:ConfigMap test replace [diff: ~data]

[lblackstone]

@pierlucg-xs Is the Deployment being replaced, or just updated?

Pulumi's k8s provider intentionally treats ConfigMap resources as immutable. I would expect this to trigger a rollout (update) in dependent Deployments rather than a replacement.

[pierlucg-xs]

Yes, here's how I reproduced it with K3S:

```ts
const provider = new k8s.Provider("k3s", {
kubeconfig,
enableDryRun: false,
});

const configMap = new k8s.core.v1.ConfigMap(
"configmap",
{
data: { foo: ${Date.now()} },
metadata: {
name: "configmap",
},
},
{ provider: provider }
);

const deployment = new k8s.apps.v1.Deployment(
"deployment",
{
apiVersion: "apps/v1",
kind: "Deployment",
metadata: {
labels: {
app: "bar",
},
name: "bar",
namespace: "default",
},
spec: {
replicas: 1,
selector: {
matchLabels: {
app: "bar",
},
},
template: {
metadata: {
labels: {
app: "bar",
},
},
spec: {
containers: [
{
envFrom: [{ configMapRef: { name: configMap.metadata.name } }],
image: "nginxdemos/hello",
name: "demo-service",
ports: [{ containerPort: 8080 }],
},
],
},
},
},
},
{ provider: provider }
);
```

Second pulumi up:
Type Name Plan Info pulumi:pulumi:Stack k3s-configmaptest +- ├─ kubernetes:apps/v1:Deployment deployment replace +- └─ kubernetes:core/v1:ConfigMap configmap replace [diff: ~data]

---

Pulumi's k8s provider intentionally treats ConfigMap resources as immutable

Any ideas why? AFAIK ConfigMaps have a PATCH api

[lblackstone]

Ah, ok. The reason you're seeing the replacement is because you are manually specifying the ConfigMap name rather than using auto-naming:

metadata: {
  name: "configmap",
},

If you remove that and let Pulumi auto-name the ConfigMap, the Deployment will update rather than replace.

---

Our k8s provider intentionally treats ConfigMap and Secret resources as immutable rather than using the PATCH API for a couple reasons:

1. Pods referencing the ConfigMap/Secret have to be restarted for changes to take effect. This is a common source of subtle errors where a subset of Pods are referencing the updated ConfigMap/Secret values, while others are still using stale data.
2. With autonamed ConfigMap/Secret resources, Pulumi will create a new instance, update dependents to refer to the new instance, and only remove the old instance once that update is successful. This makes it easier to rollback errors because the original data is still present.

I'd be interested to hear more about the use case if you are intentionally trying to reuse the same ConfigMap.

[pierlucg-xs]

I was not using auto-naming because the cluster was managed with kubectl before and I imported those resources into pulumi as-is, but it's not a hard requirement and I'll definitely change it to use auto-naming.

---

I've definitely been bitten by number 1. in the past! I'd argue that by trying to alleviate a Kubernetes issue, Pulumi is acting in a non "idiomatic K8S" way.

All in all, the benefits of auto-naming are clear and will resolve my issue, thanks a lot for your quick response!

[lblackstone]

I'd argue that by trying to alleviate a Kubernetes issue, Pulumi is acting in a non "idiomatic K8S" way.

Right, sorry for the confusion here, and thanks for linking that upstream issue! I agree that this behavior could be a surprise to some k8s users.

I do think this is the behavior that most users actually want, so I'll open a work item to make this clearer in our docs.