K3s: Use docker mirror with container.d

[simonfr]

Hello,

I'm trying to use docker registry mirror with :

Containerd v1.16.1-k3s.1
K3S v1.16.3-k3s.2

I've modified file /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl on each node :

[plugins.opt]

  path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
[plugins.cri.cni]
  bin_dir = "/var/lib/rancher/k3s/data/2492a28b74abfbdbd3f943c8109a93e979669647bbbe95eadae43b60c667df69/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"
[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"
[plugins.cri.registry]
  [plugins.cri.registry.mirrors]
    [plugins.cri.registry.mirrors."docker.io"]
      endpoint = ["https://mirror.gcr.io"]

After reboot, I check with crictl info all is good :

...
"cni": {
      "binDir": "/var/lib/rancher/k3s/data/2492a28b74abfbdbd3f943c8109a93e979669647bbbe95eadae43b60c667df69/bin",
      "confDir": "/var/lib/rancher/k3s/agent/etc/cni/net.d",
      "maxConfNum": 1,
      "confTemplate": ""
    },
    "registry": {
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "https://mirror.gcr.io",
            "https://registry-1.docker.io"
          ]
        }
      },
      "configs": null,
      "auths": null
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "10010",
...

But when I try to pull image it seems container.d always use registry-1.docker.io because when I shut down network I can see :
pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/rancher/local-path-provisioner:v0.0.11": failed to resolve reference "docker.io/rancher/local-path-provisioner:v0.0.11": failed to do request: Head https://registry-1.docker.io/v2/rancher/local-path-provisioner/manifests/v0.0.11: dial tcp 3.224.11.4:443: i/o timeout

Does anyone know why I'm having this error ?

[onedr0p]

I am too trying to set up a mirror with k3s and containerd but running into similar issues, it doesn't appear to be taking my values.

https://rancher.com/docs/k3s/latest/en/installation/private-registry/#mirrors

My debugging steps:

/etc/rancher/k3s/registries.yaml

mirrors:
  docker.io:
    endpoint:
      - "http://192.168.42.120:5000"

Restarted k3s-agent on the node.

Ran sudo ctr image pull docker.io/library/alpine:latest on that node.

Then I looked in the registries PVC and didn't see the image pulled.

[brandond]

Did you put that file on all of your nodes? What message did you get when pulling the image that indicates that it's not using the mirror?

[onedr0p]

@brandond I did try placing it on all the nodes and it wasn't working, I am trying to debug the issue. I put it on the one node for now and tried pulling on that node specifically, which should yield the results I am looking for... I didn't see any errors in the /var/lib/rancher/k3s/agent/containerd/containerd.log logs, or anything relating.

Edit: Appears to be looking to HTTPS but I have specified http in my endpoint AND cricrl showing is it correctly..

sudo ctr image pull 192.168.42.120:5000/alpine:latest
ctr: failed to resolve reference "192.168.42.120:5000/alpine:latest": failed to do request: Head https://192.168.42.120:5000/v2/alpine/manifests/latest: http: server gave HTTP response to HTTPS client

Relevant part of cricrl info

"registry": {
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "http://192.168.42.120:5000"
          ]
        }
      },
      "configs": null,
      "auths": null
    },

[onedr0p]

For the next person that comes to this issue looking for HTTP image mirroring, looks like it is an issue upstream and will be fixed in containerd 1.4.0

https://github.com/containerd/cri/issues/1433

[brandond]

Common mistake. You have specified an endpoint for docker.io but are trying to explicitly pull from 192.168.42.120:5000. If you want to be able to pull from it explicitly, you need to configure another mirror entry for that registry. This is also necessary if you want to force http for your private registry, as docker and containerd assume https by default.

[onedr0p]

@brandond I am trying to set up a pull-through cache for my cluster using docker registry. k3s should pull images from docker.io and cache them to my registry using that containerd config.

I've spent a couple hours with different configurations, restarting agents, redeploying docker registry and I cannot get k3s to use my cache even though it works perfectly on my macbook using docker and my docker registry hosted in my Cluster.

[brandond]

If you want to pull through for docker.io then don't put your local registry URI in the name. Leave the registry unspecified or specify docker.io. docker.io is all that you've configured at the moment.

[onedr0p]

I've tried that already as I mentioned in my comment https://github.com/rancher/k3s/issues/1210#issuecomment-656347828

If I pull from docker.io using that config, the cache doesn't get populated, no errors, no logs as to why...

This seems to correlate with the OPs issue on how containerd is ignoring mirrors.

[brandond]

The OP was on a much older release of k3s. Since you jumped on an existing issue I don't have any info on which specific version you're using, or how your environment is configured - but this has worked for me and many other people, so if you're on a recent version of k3s I suspect the issue is either your configuration, or how you're testing it.

[brandond]

You could try the following, which should let you both pull directly from your registry, and have that same registry used in place of docker.io:

mirrors:
  docker.io:
    endpoint:
      - "http://192.168.42.120:5000"
  192.168.42.120:5000:
    endpoint:
      - "http://192.168.42.120:5000"

You might also confirm that you don't have a copy of the image you're trying to pull already present on the host. I'm not sure what the behavior is of your registry, but it may not trigger a pull of the layers (so they wouldn't show up in your pvc) if the node already has them cached. Try pulling an image (not just a tag, but something with completely unique layers) that you've never used before.

[onedr0p]

@bradtopol I finally got back to this and tried that, it did not work. I will open a new issue (linked below) instead of hijacking this one anymore. Thanks for your help so far.