Kubeadm: support imagePullPolicy when pulling kubeadm images
Is this a request for help?
Bug / Help and possibly implement a fix for the bug
similar closed issue : https://github.com/kubernetes/kubeadm/issues/34
Versions
kubeadm version (use kubeadm version):
kubeadm version = kubeadm_1.8.2-00
ubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:38:10Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Environment:
- Kubernetes version (use
kubectl version):
kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:48:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration: AWS in isolated vpc.
- OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
Kernel (e.g.
uname -a):
Linux ip-10-205-78-6 4.4.0-1039-aws #48-Ubuntu SMP Wed Oct 11 15:15:01 UTC 2017 x86_64 x86_64 x86_64 GNU/LinuxOthers:
What happened?
kubeadm init doesn't support arg = imagePullPolicy=never or local. This impose a problem when using kubeadm with no internet access since, it defaults trying to pull images from gcr.
kubeadm errors:
command : kubeadm init --kubernetes-version v1.8.2 --pod-network-cidr=10.244.0.0/16
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
What i have done . Downloaded all google_containers related to kubernetes 1.8.2 and then scp to the host and used docker to load the containers . kubeadm default policy imagePullPolicy=always . Therefore it was failing
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
gcr.io/google_containers/kube-apiserver-amd64 v1.8.2 6278a1092d08 12 days ago 194 MB
gcr.io/google_containers/kube-controller-manager-amd64 v1.8.2 5eabb0eae58b 12 days ago 129.2 MB
gcr.io/google_containers/kube-scheduler-amd64 v1.8.2 b48970f8473e 12 days ago 54.9 MB
gcr.io/google_containers/kube-proxy-amd64 v1.8.2 88e2c85d3d02 12 days ago 93.13 MB
gcr.io/google_containers/k8s-dns-sidecar-amd64 1.14.4 38bac66034a6 4 months ago 41.82 MB
gcr.io/google_containers/k8s-dns-kube-dns-amd64 1.14.4 a8e00546bcf3 4 months ago 49.39 MB
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64 1.14.4 f7f45b9cb733 4 months ago 41.42 MB
gcr.io/google_containers/etcd-amd64 3.0.17 243830dae7dd 8 months ago 168.9 MB
gcr.io/google_containers/pause-amd64 3.0
To work around it . I have added imagePullPolicy=never to manifests and used kubelet to start the the pods .
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3469f8d57f49 gcr.io/google_containers/kube-controller-manager-amd64@sha256:c2cd4acd4238b2f2526abf5ba546d4e6f4a46618ad5747a539e8a72c294a7482 "kube-controller-mana" 5 minutes ago Up 5 minutes k8s_kube-controller-manager_kube-controller-manager-ip-10-205-78-6_kube-system_a2384e51c277f0dc61222c242361a42d_0
9233a987f5ac gcr.io/google_containers/kube-apiserver-amd64@sha256:3e980f4b57292568ea8c87be462cf0583e40bbc2dbfff71d0d9e19beda3cb74b "kube-apiserver --sec" 5 minutes ago Up 5 minutes k8s_kube-apiserver_kube-apiserver-ip-10-205-78-6_kube-system_52115e8757d49c532b9f9253a995f4c6_0
50574a1badf2 gcr.io/google_containers/kube-scheduler-amd64@sha256:7c920b718509e8cf811c69178526d84ebfab2bdbb95949f6e82eb5233e7b5f0e "kube-scheduler --kub" 5 minutes ago Up 5 minutes k8s_kube-scheduler_kube-scheduler-ip-10-205-78-6_kube-system_c277372d3697e2f5d4038d02914e31d8_0
aa7eb7294220 gcr.io/google_containers/etcd-amd64@sha256:d83d3545e06fb035db8512e33bd44afb55dea007a3abd7b17742d3ac6d235940 "etcd --listen-client" 5 minutes ago Up 5 minutes k8s_etcd_etcd-ip-10-205-78-6_kube-system_07f2b34fc77ee86c936d12f9da37f985_0
5a7251468f60 gcr.io/google_containers/pause-amd64:3.0 "/pause" 5 minutes ago Up 5 minutes k8s_POD_kube-scheduler-ip-10-205-78-6_kube-system_c277372d3697e2f5d4038d02914e31d8_0
9ec41787b709 gcr.io/google_containers/pause-amd64:3.0 "/pause" 5 minutes ago Up 5 minutes k8s_POD_kube-controller-manager-ip-10-205-78-6_kube-system_a2384e51c277f0dc61222c242361a42d_0
7e8ceedab55c gcr.io/google_containers/pause-amd64:3.0 "/pause" 5 minutes ago Up 5 minutes k8s_POD_kube-apiserver-ip-10-205-78-6_kube-system_52115e8757d49c532b9f9253a995f4c6_0
7d1ee4d69ca6 gcr.io/google_containers/pause-amd64:3.0 "/pause" 5 minutes ago Up 5 minutes k8s_POD_etcd-ip-10-205-78-6_kube-system_07f2b34fc77ee86c936d12f9da37f985_0
master was up after that
kubectl cluster-info
Kubernetes master is running at https://10.205.78.6:6443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
etcd responding
curl 127.0.0.1:2379/version
{"etcdserver":"3.0.17","etcdcluster":"3.0.0"}
API
curl https://10.205.78.6:6443/api/ -k
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "10.205.78.6:6443"
}
]
What you expected to happen?
expected kueadm to work if container-images are present locally.
How to reproduce it (as minimally and precisely as possible)?
reproduce by following the steps above.
Anything else we need to know?
not sure if it would it make sense to add hook kubeadm to talk to a private registry instead !
sefm
All 17 comments
kubeadm default policy imagePullPolicy=always
This is not the case. The default imagePullPolicy is IfNotPresent, so this should work.
Can you post the relevant manifest yamls and the kubelet log when it doesn't work for you?
luxas
on 9 Nov 2017
@luxas : Thanks for your response . Correct . The acceptable image imagePullPolicy values = Always , IfNotPresent and Never . After digging more the issue seems not related to imagePullPolicy . The issue was with not advertising the api server address :
Once I added --apiserver-advertise-address=10.205.78.6 it worked fine
kubeadm init --kubernetes-version=v1.8.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.205.78.6
Your Kubernetes master has initialized successfully!
Closing the issue.
sefm
on 12 Nov 2017
re-opening as we hit the same problem recently.
kubeadm might have to support image pull policy for it's CP images on the API side.
cc @bart0sh
neolit123
on 5 Dec 2019
@neolit123 is this the same issue? Do we want imagePullPolicy specifically for the Never case or for Always?
rosti
on 10 Dec 2019
re-using this issue so that we don't open a new one. a bit messy, but renamed it with a better title at least.
we need to support different policies.
neolit123
on 10 Dec 2019
The Never case is correct for kind FWIW, though I think we're going to just start skipping preflight entirely (we typically fail and ignore most of the checks anyhow... 🙃 e.g. swap).
BenTheElder
on 26 Apr 2020
@neolit123 is this still a problem?
AFAIK kubeadm works in air gapped environment if the image exists locally; users also have options to influence image name (via image repository/image tag fields)
As far as I understand the problem, image pull policy should not add much to what described above..
fabriziopandini
on 26 Apr 2020
It doesn't work when e.g. it's trying to pull an unnecessary pause image
and failing 🙃
On Sun, Apr 26, 2020, 10:24 Fabrizio Pandini notifications@github.com
wrote:
@neolit123 https://github.com/neolit123 is this still a problem?
AFAIK kubeadm works in air gapped environment if the image exists locally;
users also have options to influence image name (via image repository/image
tag fields)As far as I understand the problem, image pull policy should not add much
to what described above..—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubeadm/issues/524#issuecomment-619589637,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHADKY7PWRRNAAS5VCVM2TRORU4JANCNFSM4ECQSPSQ
.
BenTheElder
on 26 Apr 2020
@fabriziopandini
@neolit123 is this still a problem?
it was recently requested by @ncdc for CAPI use too.
our kubeadm logic currently always does a "IfNotPresent":
https://github.com/kubernetes/kubernetes/blob/0acf2f0983d1491caf60367f12c1bd76651209cc/cmd/kubeadm/app/preflight/checks.go#L835-L851
and with a policy of "Never" it will never pull and "Always" would always pull, which will actually run faster than "IfNotPresent" if the image is present locally.
neolit123
on 27 Apr 2020
@neolit123 I think you're referring to the regression in kubeadm that always pulled instead of doing IfNotPresent? I don't think CAPI needs anything now that the regression has been fixed.
ncdc
on 27 Apr 2020
ok, so i had to refresh my memory about that bug:
https://kubernetes.slack.com/archives/C2P1JHS2E/p1575565526161900
we did fix a regression.
there is still the use case for pullPolicy = Never or allowing users skip prepull if we exposed it as a sub-phase of preflight.
neolit123
on 27 Apr 2020
+1 @neolit123 xref https://github.com/kubernetes/kubernetes/issues/90326
Elaborating a little ...
AFAIK kubeadm works in air gapped environment if the image exists locally; users also have options to influence image name (via image repository/image tag fields)
This is not sufficient, you cannot currently configure the pause image to match your CRI. In an airgapped environment this leads to: https://github.com/kubernetes/kubernetes/issues/90326
https://github.com/kubernetes/kubeadm/issues/2020 would be one way to fix that, but alternatively in an airgapped environment I just don't want kubeadm trying to pull images at all, full stop.
If the cluster fails to come up due to missing images that should be relatively easy to diagnose. Pulling was never going to help. I know that I'm not going to pull in an airgapped env so I'd prefer to be able to tell kubeadm exactly that and skip this entirely.
I might not want to skip all preflight checks on a serious cluster though.
(btw though, there's another major potential issue in airgapped env currently that the already existing images can be evicted :/ I'm going to try to revive https://github.com/kubernetes/enhancements/pull/1007)
EDIT: see https://github.com/kubernetes/enhancements/pull/1717
BenTheElder
on 28 Apr 2020
@neolit123 shall add a flag (maybe pullPolicy) to control the behavior of prepulls in pre-flight? os, users can skip the prepulls in flight, instead of just skipping the pre-flight.
and I am happy to help with this.
xlgao-zju
on 18 Jun 2020
@xlgao-zju thanks for your help!
This is on hold now. No new command line flags should be added. The change needs to be part of a new kubeadm config version if done.
But, frankly, we haven't reached a decision if this actually needs to be done or not.
rosti
on 18 Jun 2020
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
on 16 Sep 2020
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
fejta-bot
on 16 Oct 2020
/remove-lifecycle rotten
neolit123
on 16 Oct 2020
Related issues
helphi
·
3Comments
bruceauyeung
·
4Comments
andersla
·
4Comments
ggee
·
4Comments
jbrandes
·
4Comments