kubeadm pulls pause on update and setup ignoring imageRepository: in config file

[mruepp]

What keywords did you search in kubeadm issues before filing this one?

pause
imageRepository

Versions

kubeadm version (use kubeadm version): 1.11.0

Environment:

• Kubernetes version (use kubectl version): 1.11
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Others:

What happened?

Setting up the cluster with config file and the following tag:
imageRepository: privateRegistry:5000/google-containers
worked except the pause image is always pulled from k8s.gcr.io/pause, ignoring the setting.
We set up the system by manually pulling pause (1.10 needs the pause-amd64:3.1) and retag it acoordingly.

We upgraded the cluster from 1.10 to 1.11 and had the same issue. The upgrade fails because pause can not be pulled. After pulling manually from privateRegistry:5000/pause:3.1 and retagging it to k8s.gcr.io:pause3.1, the controlplane comes up again.

So kubeadm does not respect the imageRepository, using private or proxy registries, regarding the pause image.

What you expected to happen?

Using the proper configured image registry

How to reproduce it (as minimally and precisely as possible)?

Try to set up k8s with kubeadm from private registry, using the imageRepository Tag

Anything else we need to know?

[rosti]

kubeadm actually respects that image and if you pre-pull your images with kubeadm it should be able to fetch it from your private registry.

The thing, however, is that kubelet appears to be hardwired to several different versions of it (k8s.gcr.io/pause-amd64:latest, k8s.gcr.io/pause:3.1, etc.). So I think, that the problem is caused by kubelet.

We may attempt to pre-pull and tag the image in kubeadm, but I really think, that kubelet should be able to fetch the exact image from its config and use that instead. There is a --pod-infra-container-image flag, but I am not sure if it works at all (have to test it).

[neolit123]

/help
/kind bug
/cc @kubernetes/sig-node-bugs ^ PTAL

[k8s-ci-robot]

@neolit123:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help
/kind bug
/cc @kubernetes/sig-node-bugs ^ PTAL

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[luxas]

Idea, kubeadm could/should set --pod-infra-container-image in the kubelet config file automatically based on the image repository, if the docker CRI runtime is used (the flag is only used for dockershim).
This would be an ok improvement to kubeadm that would miligate this.
@neolit123 WDYT?

[neolit123]

hmm, so pod-infra-container-image is only used in the docker case.
looks like the other CRIs have their own ways of setting the pause image. :\

i guess we should do this workaround for docker in kubeadm for now. :+1:

[neolit123]

PR was sent for this:
https://github.com/kubernetes/kubernetes/pull/66798

it's to be determined if this is going to be cherry picked for 1.11 and how we proceed for 1.12.

until then a workaround for docker and 1.11 is to pass:

nodeRegistration:
  kubeletExtraArgs:
    pod-infra-container-image: <custom>

for other CRIs have a look at this comment:
https://github.com/kubernetes/kubernetes/issues/62732#issuecomment-382269288

[timothysc]

/assign @timothysc

[neolit123]

i need to revisit that PR:
https://github.com/kubernetes/kubernetes/pull/66798

there is one use case that cannot be solved because image repository is part of cluster configuration.
so we might have to compromise.

[mruepp]

Thx guys