K3s: How to pull image from private registry?

[hillbun]

Thanks for helping us to improve k3s! We welcome all bug reports. Please fill out each area of the template so we can better help you. You can delete this message portion of the bug report.

Version:
Provide the output from k3s -v and provide the flags used to install or run k3s server.

k3s version v0.10.0 (f9888ca3)

Describe the bug
A clear and concise description of what the bug is.

I config /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl adding

[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."docker.xxx.cn"]
endpoint = ["http://docker.xxx.cn:5000"]

[plugins.cri.registry.configs."docker.xxx.cn".auth]
username = "xxxx"
password = "xxxxxxx"

when I pull

k3s crictl pull docker.xxx.cn:5000/maxfaith/miop_ui:development

error returned:

FATA[2019-10-30T14:23:39.122757012+08:00] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to resolve reference "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to do request: Head https://docker.xxx.cn:5000/v2/maxfaith/miop_ui/manifests/development: x509: certificate has expired or is not yet valid

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Actual behavior
A clear and concise description of what actually happened.

Additional context
Add any other context about the problem here.

[xiaods]

hi, have a try this: k3s crictl pull --creds USERNAME[:PASSWORD]

[carlosrmendes]

hi configured my private registry on /etc/rancher/k3s/registries.yaml file and I cannot pull my image.
I'm getting this error running version 0.10.2:
Failed to pull image "repo_address/image_name:tag": rpc error: code = NotFound desc = failed to pull and unpack image "repo_address/image_name:tag": failed to resolve reference "repo_address/image_name:tag": repo_address/image_name:tag: not found

[xiaods]

@carlosmkb i suggest use cmd to pull image again.

[hillbun]

hi, have a try this: k3s crictl pull --creds USERNAME[:PASSWORD]

I tried this, same error

FATA[2019-10-30T14:23:39.122757012+08:00] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to resolve reference "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to do request: Head https://docker.xxx.cn:5000/v2/maxfaith/miop_ui/manifests/development: x509: certificate has expired or is not yet valid

[hillbun]

@xiaods

it seems this config not effected

[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."docker.xxx.cn"]
endpoint = ["http://docker.xxx.cn:5000"]

[xiaods]

x509: certificate has expired or is not yet valid， it maybe your ntp time is not sync on master, and client server. please double check the situation.

[xiaods]

@hillbun have you try it

[hillbun]

@xiaods

mostly it is not time problem.

My private registry is insecure.

any similar configureation in docker to solve this problem?
"insecure-registries": ["docker.xxx.cn:5000"]

[hillbun]

[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."docker.xxx.cn:5000"]
endpoint = ["http://docker.xxx.cn:5000"]

[plugins.cri.registry.configs."docker.xxx.cn:5000".auth]
username = "myusername"
password = "mypassword"

crictl pull docker.xxx.cn:5000/maxfaith/miop_ui:development
FATA[2019-11-07T15:29:08.826492009+08:00] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to resolve reference "docker.xxx.cn:5000/maxfaith/miop_ui:development": failed to do request: Head http://docker.xxx.cn:5000/v2/maxfaith/miop_ui/manifests/development: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

the errir occupied

[hillbun]

also, when k3s yalm define imagePullSecrets, it can not pull image either.

[xiaods]

@hillbun sorry for your waiting. i am on a busy case , i will go to setup a environment go test on your case asap.

[xiaods]

@hillbun go through k3s issue list. found a potential issue: https://github.com/rancher/k3s/issues/145 don't know it cause the bug. it should be check.

[xiaods]

https://www.civo.com/learn/set-up-a-private-docker-registry-with-tls-on-kubernetes FYI @hillbun

[hillbun]

@xiaods

As I mentioned, my registry is NON tls

[ysolis]

is this a duplicate of https://github.com/rancher/k3s/issues/502 ?

well, in any case i don't have any problems to use a private registry
https://github.com/rancher/k3s/issues/502#issuecomment-548602687

[stone-wlg]

my k3s version is v1.0.0. i have setup docker registry without https and i use --private-registry registries.yaml.
if i setup k3s with --docker, k3s can use private registry.
if i setup k3s without --docker, k3s never use private registry.

[xiaods]

the --docker options is what purpose? @stone-wlg please give more insight to me. curious for the situation that.

[stone-wlg]

the --docker options is what purpose? @stone-wlg please give more insight to me. curious for the situation that.

https://rancher.com/docs/k3s/latest/en/installation/install-options/
--docker (agent/runtime) Use docker instead of containerd

[riker09]

I can confirm that the default implementation with containerd does not support GitHub Package Registry (GPR) with imagePullSecrets. It seems that imagePullSecrets are not supported by k3s, see this thread on StackOverflow.

However, when running the k3s cluster with the --docker option everything works fine. So the issue must be in containerd, not k3s. :thinking: :man_shrugging:

[ysolis]

I can confirm that the default implementation with containerd does not support GitHub Package Registry (GPR) with imagePullSecrets. It seems that imagePullSecrets are not supported by k3s, see this thread on StackOverflow.

However, when running the k3s cluster with the --docker option everything works fine. So the issue must be in containerd, not k3s. thinking man_shrugging

@riker09 @stone-wlg the problem is apparently only related to Github registry. I have an image in a Gitlab Registry in a personal private repository in gitlab.com, I have deployed k3s 1.0 with default options (servicelb, traefik and containerd), created the secret with kubectl create secret ..., defined the deploy token needed in gitlab to access the private repo/registry ... and the pod was deployed. Please confirm if this is the case. If you need the info, I follow the steps described in [1] and [2] using the imagePullSecrets option in the deployment yaml file.

[1] https://dimsolution.com/blog/2018-07-06/kubernetes-using-a-private-registry-like-gitlab/
[2] https://blog.zedroot.org/2019/01/21/gitlab-ci-kubernetes-pull-a-private-image-from-a-k8s-pod/

[erikwilson]

It looks like an issue with the Github Package Registry or containerd (or both) https://github.com/containerd/containerd/issues/3291

[riker09]

Hello @ysolis and thank you for your kind answer. As I have stated I'm trying to pull from a private GitHub Registry. This fails with containerd, but succeeds when I use Docker. I have commented on the issue provided by @erikwilson as well.

I would love to help to solve this for everybody so please let know if I can provide you with additional information.

Regarding the two links you provided. The first one currently gives me a 502 HTTP error. I read the second blog post. I have created my pullSecret without an email. But I don't suspect that this is of importance, is it? Aside from that, I'm adding the secret to my pods via imagePullSecrets and not with the service account.

[davidnuzik]

@erikwilson is there a bug that needs to be addressed here or is this covered (or needs to be covered) by docs for Private Registry?

[cjellick]

.

[erikwilson]

The linked issue, which is still open, has an update from 30 days ago which says a fix is in progress. @davidnuzik

[erikwilson]

I think some private registries work, but gcr may also be a problem from other issues (#1610). This issue is probably old enough that it can be closed tho, the registries.yaml file is the preferred way to configuration auth (see https://rancher.com/docs/k3s/latest/en/installation/private-registry/). We should probably close this issue and open a new one specific for github registry.

[webees]

Similar help is needed.

https://github.com/containerd/containerd/issues/4452

[ramran-r]

I am trying to pull images from local docker registry by keeping 'pullSecrets:' in deployment yaml, however it is always referring to 'registries.yaml' ( which has TLS Cert & auth details ) . I see some of them succeeded to pull the images using 'pullSecrets'. Could you pls. share the steps followed or is it still known issues with K3S.