K3s: imagePullSecrets support

[simmessa]

The use of imagePullSecrets should be supported in the same way as in Kubernetes, but it seems this is not the case, this is not only limited to initial setup as suggested by #99 but should be available even after booting up the cluster.

Is your feature request related to a problem? Please describe.
I'm not 100% sure this is a bug, but I'm pretty convinced anybody using k8s expects to be able to use imagePullSecrets for deployments and this is not possible at this point. Since this is not officially communicated as a feature I will say it's a feature request, but for a pretty basic feature, from my point of view.

Describe the solution you'd like
I'd like to be able to use imagePullSecrets in deployments for pulling container images from private registries

Describe alternatives you've considered
I know there's another way to achieve this, via document image preloading, but it's clearly a workaround and I believe this kind of 'basic functionality' should be made available as part of a larger k8s standard

Additional context
Here's a link to the situation described pretty well on stackoverflow: https://stackoverflow.com/questions/55457376/imagepullsecrets-not-working-when-using-k3s

[ndarilek]

I wouldn't even call image preloading a workaround. I had a deployment that builds a custom Caddy server, packaged as a Docker image with a static website and additional configuration. I push it to my private GitLab registry, and redeploy it on any code changes. I could preload an initial image, but I don't know how I'd push new changes into the cluster later.

If this isn't supported, I'm hoping there's another solution.

[simmessa]

@ndarilek I second that, we need private registries for most cases, hopefully this issue will go further.

Thanks,

Simone.

[erikwilson]

There may be some underlying issue with containerd/cri pull secrets support where it only works with gcr registry containerd/cri/issues/668, it may be possible we need to update our version of cri, or could be just a config issue.

It may help to try creating a file /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl based off the current template at https://github.com/rancher/k3s/blob/master/pkg/agent/templates/templates.go with your added registry information (see https://github.com/containerd/cri/blob/master/docs/registry.md)

[simmessa]

@erikwilson Sure I'll give it a try and let you know if it works with the additional configuration, many thanks!

[digger18]

imagePullSecrets works for me with Docker runtime (using --docker option).

[ndarilek]

Hmm, it occurred to me that this might be an issue with the container
subsystem. Has anyone tried using Docker? Does the Docker subsystem
support this?

I don't mind using Docker instead of Containerd, but I do need this
feature. Is that something I can do after I've already launched my
agent? I'm just using local paths for storage, so in theory everything
would just re-instantiate with Docker and not Containerd, but...

[ndarilek]

Great news. Can I just add the --docker option to a systemd unit
override and restart k3s to switch engines? Or am I out of luck now that
I'm using containerd?

Thanks.

[ndarilek]

I appended --docker to the command line, and things seem to still be
working. I still have /var/lib/rancher/k3s/agent/containerd, though.
Would that be true even using the Docker engine? The string "docker"
doesn't appear anywhere in journalctl.

[ndarilek]

I learned that kubectl get nodes -o wide outputs the container runtime
being used. But after adding the --docker argument, I get:

NAME      STATUS   ROLES    AGE     VERSION INTERNAL-IP      EXTERNAL-IP
    OS-IMAGE                  KERNEL-VERSION CONTAINER-RUNTIME
k8s-000   Ready    <none>   2d16h   v1.14.1-k3s.4 195.201.35.232   <none>
    Fedora 29 (Twenty Nine)   5.0.17-200.fc29.x86_64 
containerd://1.2.5+unknown

So it still claims containerd. Yet I did submit a pod with:

       imagePullSecrets:
         - name: website-registry

in its specification and it was deployed.

Full disclosure: I didn't even try imagePullSecrets without the
--docker argument since this issue suggested it wouldn't work. So either
it does, or kubectl get nodes -o wide is misrepresenting the container
runtime and I should file a separate issue for that.

[uudens]

@simmessa were you able to get this working?

[abhishek-jain11]

There may be some underlying issue with containerd/cri pull secrets support where it only works with gcr registry containerd/cri/issues/668, it may be possible we need to update our version of cri, or could be just a config issue.

It may help to try creating a file /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl based off the current template at https://github.com/rancher/k3s/blob/master/pkg/agent/templates/templates.go with your added registry information (see https://github.com/containerd/cri/blob/master/docs/registry.md)

Worked for me with private registry in dockerhub. 👍
Thanks a ton!

[andres-blanco]

There may be some underlying issue with containerd/cri pull secrets support where it only works with gcr registry containerd/cri/issues/668, it may be possible we need to update our version of cri, or could be just a config issue.
It may help to try creating a file /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl based off the current template at https://github.com/rancher/k3s/blob/master/pkg/agent/templates/templates.go with your added registry information (see https://github.com/containerd/cri/blob/master/docs/registry.md)

Worked for me with private registry in dockerhub.
Thanks a ton!

I tried this with a private gitlab registry and it didn't work. Here is my config.toml.tmpl

[plugins.opt]
path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.cri]
stream_server_address = "ablanco"
stream_server_port = "10010"
  [plugins.cri.cni]
    bin_dir = "/var/lib/rancher/k3s/data/4e1224c66a9dbb9b03daefff200f4f8eaf45590fb722b6fe2924a201d6de2e8d/bin"
    conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."gitlab.private.com"]
    endpoint = ["http://gitlab.private.com:5005"]
  [plugins.cri.registry.mirrors."gitlab.private.com:5005"]
    endpoint = ["http://gitlab.private.com:5005"]
  [plugins.cri.registry.auths]
    [plugins.cri.registry.auths."gitlab.private.com"]
      username = "myuser"
      password = "mypassword"
      # also tried with the auth token from docker config with no luck
      #auth = "csisddddd5jbzpzcGFuVFIxMg=="

When I try to execute:

k3s crictl --debug pull gitlab.private.com:5005/mygroup/myimage:latest

I get:

DEBU[2019-06-10T16:53:30.803820935-03:00] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gitlab.private.com:5005/mygroup/myimage:latest,},Auth:nil,SandboxConfig:nil,} 
DEBU[2019-06-10T16:53:30.856669975-03:00] PullImageResponse: nil                       
FATA[2019-06-10T16:53:30.856716915-03:00] pulling image failed: rpc error: code = Unknown desc = failed to resolve image "gitlab.private.com:5005/mygroup/myimage:latest": no available registry endpoint: failed to fetch anonymous token: unexpected status: 403 Forbidden

[simmessa]

@simmessa were you able to get this working?

Nope, I made a try with the containerd toml template but I might have missed something and it I wasn't able to make it work. I might make another try though.

[digger18]

Works for me (containerd backend) without any changes using a private gcr.io registry.

[abhishek-jain11]

There may be some underlying issue with containerd/cri pull secrets support where it only works with gcr registry containerd/cri/issues/668, it may be possible we need to update our version of cri, or could be just a config issue.
It may help to try creating a file /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl based off the current template at https://github.com/rancher/k3s/blob/master/pkg/agent/templates/templates.go with your added registry information (see https://github.com/containerd/cri/blob/master/docs/registry.md)

Worked for me with private registry in dockerhub.
Thanks a ton!

I tried this with a private gitlab registry and it didn't work. Here is my config.toml.tmpl

[plugins.opt]
path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.cri]
stream_server_address = "ablanco"
stream_server_port = "10010"
  [plugins.cri.cni]
    bin_dir = "/var/lib/rancher/k3s/data/4e1224c66a9dbb9b03daefff200f4f8eaf45590fb722b6fe2924a201d6de2e8d/bin"
    conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."gitlab.private.com"]
    endpoint = ["http://gitlab.private.com:5005"]
  [plugins.cri.registry.mirrors."gitlab.private.com:5005"]
    endpoint = ["http://gitlab.private.com:5005"]
  [plugins.cri.registry.auths]
    [plugins.cri.registry.auths."gitlab.private.com"]
      username = "myuser"
      password = "mypassword"
      # also tried with the auth token from docker config with no luck
      #auth = "csisddddd5jbzpzcGFuVFIxMg=="

When I try to execute:

k3s crictl --debug pull gitlab.private.com:5005/mygroup/myimage:latest

I get:

DEBU[2019-06-10T16:53:30.803820935-03:00] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gitlab.private.com:5005/mygroup/myimage:latest,},Auth:nil,SandboxConfig:nil,} 
DEBU[2019-06-10T16:53:30.856669975-03:00] PullImageResponse: nil                       
FATA[2019-06-10T16:53:30.856716915-03:00] pulling image failed: rpc error: code = Unknown desc = failed to resolve image "gitlab.private.com:5005/mygroup/myimage:latest": no available registry endpoint: failed to fetch anonymous token: unexpected status: 403 Forbidden

I had added below to config.toml to enable private registery from dockerhub.

[plugins.cri.registry.auths]
  [plugins.cri.registry.auths."https://index.docker.io/"]
    username = ""
    password = ""
    auth = ""
    identitytoken = ""

[ritarya]

There may be some underlying issue with containerd/cri pull secrets support where it only works with gcr registry containerd/cri/issues/668, it may be possible we need to update our version of cri, or could be just a config issue.
It may help to try creating a file /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl based off the current template at https://github.com/rancher/k3s/blob/master/pkg/agent/templates/templates.go with your added registry information (see https://github.com/containerd/cri/blob/master/docs/registry.md)

Worked for me with private registry in dockerhub.
Thanks a ton!

I tried this with a private gitlab registry and it didn't work. Here is my config.toml.tmpl

[plugins.opt]
path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.cri]
stream_server_address = "ablanco"
stream_server_port = "10010"
  [plugins.cri.cni]
    bin_dir = "/var/lib/rancher/k3s/data/4e1224c66a9dbb9b03daefff200f4f8eaf45590fb722b6fe2924a201d6de2e8d/bin"
    conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."gitlab.private.com"]
    endpoint = ["http://gitlab.private.com:5005"]
  [plugins.cri.registry.mirrors."gitlab.private.com:5005"]
    endpoint = ["http://gitlab.private.com:5005"]
  [plugins.cri.registry.auths]
    [plugins.cri.registry.auths."gitlab.private.com"]
      username = "myuser"
      password = "mypassword"
      # also tried with the auth token from docker config with no luck
      #auth = "csisddddd5jbzpzcGFuVFIxMg=="

When I try to execute:

k3s crictl --debug pull gitlab.private.com:5005/mygroup/myimage:latest

I get:

DEBU[2019-06-10T16:53:30.803820935-03:00] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gitlab.private.com:5005/mygroup/myimage:latest,},Auth:nil,SandboxConfig:nil,} 
DEBU[2019-06-10T16:53:30.856669975-03:00] PullImageResponse: nil                       
FATA[2019-06-10T16:53:30.856716915-03:00] pulling image failed: rpc error: code = Unknown desc = failed to resolve image "gitlab.private.com:5005/mygroup/myimage:latest": no available registry endpoint: failed to fetch anonymous token: unexpected status: 403 Forbidden

Did it Worked for you? I am also struggling hard to make it work. Tried with first with tls also. Please let me know if worked for you.

[seerickcode]

For those having an issue with the above snippets :
[plugins.cri.registry.auths]
needs to be at the same indent level as
[plugins.cri.registry.mirrors]
NOT nested inside it.

LIke this:

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."us.gcr.io"]
    endpoint = ["https://us.gcr.io"]
[plugins.cri.registry.auths]
  [plugins.cri.registry.auths."https://us.gcr.io"]
    username = "_dcgcr_1_5_0_token"
    password = "2nald98e93958fnbitlongpassword"

Side note - I have not dug into the proper grc.io auth setup, so just using echo "us.gcr.io" | docker-credential-gcr get for now, but it only lasts a short time.

[upMKuhn]

I am super confused. The documented config in containerd specifies a completly different config layout. I tried out all combinations. The code snippet by @seerickcode seems to work, but only as far to make the creds show up in crictl info.

If I were to execute crictl --debug pull registry.gitlab.com/repo/image:latest. It would request with &PullImageRequest{ ....Auth:nil, SandboxConfig:nil}. The result is the 403 response.

https://github.com/containerd/cri/blob/master/docs/registry.md

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."registry.gitlab.com"]
    endpoint = ["https://registry.gitlab.com"]                            
[plugins.cri.registry.auths."registry.gitlab.com"]
    username = "user"
    password = "PW"

[seerickcode]

@upMKuhn Double check what you have typed. It is not the same as what I have typed :)
Also remember to restart K3S (but really, it's probably that you are missing a line [plugins.cri.registry.auths] all by itself to start the collection)

edit: tried killing the line, restarting k3s, and checking my crictl info. Everything still parsed fine it seems, and I can still pull a fresh image. So it's possible the missing line is not needed by the parser.

I am using k3s version v0.8.1 (d116e74a)

Also the Auth:nil shows up even in a working config.

FYI, I have, since the last posting, switched to a using a service account so I have a persistent token. For those trying that method, this is what it looks like (in /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl)

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."us.gcr.io"]
    endpoint = ["https://us.gcr.io"]
[plugins.cri.registry.auths]
  [plugins.cri.registry.auths."https://us.gcr.io"]
    auth = "X2pzb25fa2V5On---massive-redacted-token---SIKfQ=="

sudo k3s crictl --debug pull us.gcr.io/someproject-1197/somefolderl/jump:28
DEBU[2019-09-21T21:06:47.965621513-04:00] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:us.gcr.io/someproject-1197/somefolder/jump:28,},Auth:nil,SandboxConfig:nil,} 
DEBU[2019-09-21T21:07:02.042535054-04:00] PullImageResponse: &PullImageResponse{ImageRef:sha256:5c909fe7c49c2417e1c59fb8ca6aa7b5d98bf10c28ac1fd21bc71b9ef80ddb8f,} 
Image is up to date for sha256:5c909fe7c49c2417e1c59fb8ca6aa7b5d98bf10c28ac1fd21bc71b9ef80ddb8f

[seerickcode]

Just in case somehow username/password based auth had somehow changed since I last posted (not sure what k3s version I had at the time), I went ahead and just added another repo that was not on my k3s. I used an actual gitlab based repo too, just to make sure we're on the same page.

As a bonus, just to add some more working examples (well working for me and redacted), this configuration is for an url with a non-standard port, so you can see an example of how that is setup.

This is also the FULL /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl as it exists on my instance now

Pulling things down just fine with example like : sudo k3s crictl --debug pull git.redactedurl.ca:4567/redactedname/java-legacyr:1.9.2-fr-201909210921

k3s also pulling fine as the containers are standing up.

[plugins.opt]
path = "{{ .NodeConfig.Containerd.Opt }}"
[plugins.cri]
stream_server_address = "127.0.0.1"
stream_server_port = "10010"
{{- if .IsRunningInUserNS }}
disable_cgroup = true
disable_apparmor = true
restrict_oom_score_adj = true
{{ end -}}
{{- if .NodeConfig.AgentConfig.PauseImage }}
sandbox_image = "{{ .NodeConfig.AgentConfig.PauseImage }}"
{{ end -}}
{{- if not .NodeConfig.NoFlannel }}
  [plugins.cri.cni]
    bin_dir = "{{ .NodeConfig.AgentConfig.CNIBinDir }}"
    conf_dir = "{{ .NodeConfig.AgentConfig.CNIConfDir }}"
{{ end -}}

[plugins.cri.registry.mirrors]
  [plugins.cri.registry.mirrors."us.gcr.io"]
    endpoint = ["https://us.gcr.io"]
  [plugins.cri.registry.mirrors."https://git.redactedurl.ca:4567"]
    endpoint = ["https://git.redactedurl.ca:4567"]
[plugins.cri.registry.auths]
  [plugins.cri.registry.auths."https://us.gcr.io"]
    auth = "X2pzb25faabiiglongredactedgcloudtokenehhh"=="
  [plugins.cri.registry.auths."https://git.redactedurl.ca:4567"]
    username = "maybebabyjenkins"
    password = "trgzXdnrBTRZWzQmtrgzXdnrBTRZWzQm"

[upMKuhn]

Awesome thank you very much @seerickcode .

For anyone facing the same issue:
https:// is for some reason required in [plugins.cri.registry.auths."https://git.redactedurl.ca"]. Even though you declare a mirror as : [plugins.cri.registry.mirrors."git.redactedurl.ca"]

Does anyone know if I will need to specify this config on all my future nodes or is it only required on the master?

[ysolis]

well, in my case i deployed k3s 0.10.1 (latest release), one master and one node, with default options (containerd, not docker)

i deployed without problems private containers from registry.gitlab.com, so works with me in this case, created the token in gitlab.com settings, passing this as a secret to k3s and kubectl apply private_deployment,yaml with imagePullSecrets deploy the pods without problems

[upMKuhn]

Same .... I've just set up an image pull secret on my cluster. (Latest version I think) It pulled the image from my private repo without any problems. So I think this can be closed?

[kable-wilmoth]

Not knowing this was an issue, I was able to deploy a secret for my git lab repository and pull images. Then I deleted my deployment and re-deployed and was getting ErrImagePull errors.
failed to resolve reference "docker.mygitlab.com/emc/mss/mss-zfe:master": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

Images are present using crictl images, so this did work at one point, but is failing now.

Un-installed and now it is failing to pull on the first pass.

k3s version v1.19.2+k3s1 (d38505b1) (on WSL2 - windows server 2004, but doesn't seem to be related)

[brandond]

I'm not aware of any issue with imagePullSecrets. Can you share a sample deployment that's giving you trouble?

From reading the thread at https://github.com/containerd/containerd/issues/3291 it appears that some registries don't implement the registry API properly - they use nonstandard URLs or return unexpected status codes in the authentication workflow. Have you tried this against a different registry?

[CuBiC3D]

Using this with private gitlab.com repository and no docker. I'm experiencing no issues as of today.

[daozhao]

when you Install k3s . Use docker instead of containerd,Don‘t use embedded crictl。

Install command：
curl -sfL https://get.k3s.io | sh -s - server --docker

my k3s version v1.18.9+k3s1 (630bebf9)

[frafra]

It works for me only when adding a docker secret with username/password/email, not with a generic secret containing the Docker config file with the key.

[brandond]

Closing due to age. If anyone is experiencing a problem with this please open a new issue. Most cases it's incorrect configuration or a misunderstanding about how the containerd registry configuration works.