Pulumi-kubernetes: Pulumi can't adhere to `pod.spec.containers.imagePullPolicy` when it is set to "AlwaysPull," as no spec changes occur on future updates w/ `pulumi up`

[metral]

For pods with imagePolicy: AlwaysPull set in their container's spec, Pulumi will not detect any changes after the initial deployment. On a future update, because the spec does not change, Pulumi does not force the kubelet to do a rolling update, as the the node does not force pull down the image and invalidate its Docker image cache of the given container tag.

This can be an issue during dev if you're using the same container tag, and not leveraging hashing in tags or semantic version patching: e.g. "dev," "latest", "0.0.1" etc.

[hausdorff]

Ok, so for example: if you say latest and have the container set to always pull, pulumi will actually never do an update because even though latest has changed, the spec has not.

This is a really important point that I'm not sure what to do about yet. Our Docker package resolves image tags like latest to SHAs at deployment time, so that if the underlying image has changed, you get a rollout. The problem with this is that if you're building the container using the builder API, you can't know whether the tag changed until it's been pushed to the registry -- the registry has a SHA that refers basically to a push, while the local Docker SHA refers to all sorts of local-specific state.

Anyway, we'll have to solve this as part of that workstream anyway.

[pbzdyl]

I am not sure I fully understand the issue but doesn't Kubernetes already handles it by itself? https://kubernetes.io/docs/concepts/containers/images/#updating-images

[metral]

To elaborate: the core of this issue is that when AlwaysPull is set, this is an implicit action that only takes place in k8s when the existing Pod has stopped, or crashed, and restarted; at which point, the kubelet will force a pull down of the image. If this were to happen, Pulumi has no context of this change as it happens outside of Pulumi's control, in k8s only.

Pulumi acts on spec changes, and when there is one it follows a create-before-delete approach. If I push out my current version of the app to the latest tag, roll that out with Pulumi and it runs fine, we're good. But, if at some time later I introduce a breaking change and this new commit gets re-tagged as latest:

1) Pulumi will not actually be aware that I made a change to the app given that the spec does not change due to using the latest tag - this is a primary reason to never use fixed tags in the first place with k8s, but nevertheless it does happen in development

2) If the current working Pod were to go down, crash, get migrated over to another Node etc, and the kubelet force-pulled down the latest tag that is broken as AlwaysPull states it should do, the Pod will enter a CrashLoop on k8s that Pulumi isn't currently aware of, and has no way of informing the user, let alone resolve it.

[hausdorff]

@lblackstone Since this is a "serious bug" and I don't have time to fix it, I'm proposing to bounce it to you. LMK if that does not work.

[lblackstone]

I'm not clear what the solution is here. This is basically a case of "user error", and I don't see an easy answer. A couple possible solutions:

1. Don't allow AlwaysPull
2. Track the image sha somehow and flag discrepancies.

The first option is probably a nonstarter, and I'm not sure how doable the second is.

Any other ideas?

[lblackstone]

@hausdorff Thoughts on this?

[hausdorff]

What does kubectl apply do here? I thought it would trigger a rollout. If it doesn't, I think we should leave the issue alone. If it does, then I think we should fix it.

In ksonnet we had an "image tag resolver" abstraction, which in Pulumi would be a library that resolves a tag to a specific SHA. I think we can do that here and get readability of tags, but with the specificity of SHAs.

[lblackstone]

I tested this out just to be sure:

$ kubectl apply -f pod.yaml
pod/nginx unchanged

I pushed two different versions of the image using the latest tag, and k8s does not automatically pull the image upon rerunning kubectl apply. According to the docs, k8s will only pull the image when the Pod starts. In this case, the Pod spec doesn't change, so kubectl apply is a no-op.

I do think kubectl rollout would trigger a pull, but that's not what we're using to deploy.

[hausdorff]

Alright, then I say close the issue and fix this when we build the image resolver for kubex.

[danvatca]

Just chiming in for anybody looking for a solution to this.
While kubectl rollout would solve the issue, it requires breaking the pulumi workflow in ways that are not very palatable.
What I ended up doing, was to implement the 'image resolver' in the infrastructure code, such that pulumi's Deployment has the image set to the repo/img@sha256:ID instead of repo/img:latest.
A crude version is this:

import docker

def docker_image_resolve(image: str):
    (image_name, tag) = image.split(':')
    client = docker.from_env()
    image_id = client.images.get_registry_data(f"{image}").id
    return f"{image_name}@{image_id}"

....
tool_deployment = Deployment(
...
            "spec": {
                "containers": [
                    {
                        "name": "tool",
                        "image": docker_image_resolve("registry.local/library/tool:latest"),