K3d: [BUG] k3d local insecure registry example creates a broken cluster

[nicks]

What did you do?
I followed the guide here on how to set up k3d with a local registry:
https://github.com/rancher/k3d/blob/master/docs/examples.md#connect-with-a-local-insecure-registry

When I try to create any container in this cluster, the pod fails with this error:

  Warning  FailedCreatePodSandBox  6s (x8 over 101s)  kubelet, k3d-k3s-default-server  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: runtime "io.containerd.runc.v1" binary not installed "containerd-shim-runc-v1": file does not exist: unknown

• How was the cluster created?

k3d create     --name ${CLUSTER_NAME}     --wait 0     --auto-restart     --volume $HOME/.k3d/config.toml.tmpl:/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl

(this was taken from the guide above)

Note that the cluster works fine if I do a plain k3d create

• What did you do afterwards?

applied a Deployment to the cluster. Here is the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test-registry
  labels:
    app: nginx-test-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-test-registry
  template:
    metadata:
      labels:
        app: nginx-test-registry
    spec:
      containers:
      - name: nginx-test-registry
        image: registry:2
        ports:
        - containerPort: 80

Note that it does not even use the local registry

What did you expect to happen?

The pod to come up

Concise description of what you expected to happen after doing what you described above.

The pod is stuck in ContainerCreating forever

If applicable, add screenshots or terminal output (code block) to help explain your problem.

Which OS & Architecture?

MacOS/amd64

Which version of k3d?

k3d version v1.3.4

Which version of docker?

Client: Docker Engine - Community
Version: 19.03.4
API version: 1.40
Go version: go1.12.10
Git commit: 9013bf5
Built: Thu Oct 17 23:44:48 2019
OS/Arch: darwin/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.4
API version: 1.40 (minimum version 1.12)
Go version: go1.12.10
Git commit: 9013bf5
Built: Thu Oct 17 23:50:38 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683

[fearoffish]

I see exactly the same thing, with docker 19.03.03

Which OS & Architecture?

MacOS/amd64

Which version of k3d?

k3d version v1.3.4

Which version of docker?

Client: Docker Engine - Community
Version: 19.03.3
API version: 1.40
Go version: go1.12.10
Git commit: a872fc2
Built: Tue Oct 8 00:55:12 2019
OS/Arch: darwin/amd64
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 19.03.3
API version: 1.40 (minimum version 1.12)
Go version: go1.12.10
Git commit: a872fc2
Built: Tue Oct 8 01:01:15 2019
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683

[fearoffish]

The documentation is out of date. I used the default template from k3s instead of the one given in the documentation and I no longer get the problem.

I'll get in a pull request to update the docs.

[iwilltry42]

Hi @nicks, thanks for opening this issue and @fearoffish thanks for figuring out the problem :smile:
k3s changed a lot in the containerd configuration since the beginning of this month and we didn't know about this (many people working on k3d, including me, are not part of Rancher, so we also have to check k3s code from time to time to see if things have changed).

I just checked the history of the config.toml file and saw, that you can now also add private registries using a separate YAML file instead of overwriting the config.toml directly: https://github.com/rancher/k3s/commit/5ccc880ddb2e6489d3af310f7035c1b90f607fdf#diff-0ec5efa54f7615eb4ac813e7ce7098aeR114

I guess we should have both options in our docs.

[asksven]

Looks neat and less error prone to other changes to config.toml.tmpl indeed. Tested by changing the k3d create command to:

CLUSTER_NAME=k3s-default
k3d create \
    --name ${CLUSTER_NAME} \
    --wait 0 \
    --auto-restart \
    --volume /home/${USER}/.k3d/registries.yaml:/etc/rancher/k3s/registries.yaml

and adding the following registries.yaml:

mirrors:
  "registry.local:5000":
    endpoint:
    - http://registry.local:5000

Happy to add a PR to the doc if not already on-going

Edit: tested OK with 0.10.2, fails with 0.9.1

[zer0def]

K3s starting with 0.10.0 only packages containerd-shim-v2-runc, so all you really have to do is add a runc v2 shim configuration to the template, like so:

#!/bin/sh -x

# Alter to your liking
K3S_VERSION="0.9.1"

echo "${K3S_VERSION}" | grep -E '^0\.[0-9]\.' && OLD_K3S=1 || OLD_K3S=0

[ ${OLD_K3S} -eq 1 ] && RUNC_SHIM_VERSION=v1 || RUNC_SHIM_VERSION=v2

cat <<EOF >> config.toml.tmpl
[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.${RUNC_SHIM_VERSION}"
EOF

[iwilltry42]

Added new extensive docs on this in commit 3b0c095
Thanks for all the contributions :)

[pojntfx]

First of all: Thanks for all of your work! This enables a offline/mobile development workflow for me, which is an incredible productivity booster.

However, I believe that https://github.com/rancher/k3d/blob/master/docs/examples.md is currently lacking the actual registry.yaml from https://github.com/rancher/k3d/issues/134#issuecomment-549046877; shouldn't this be in line 111?

[iwilltry42]

@pojntfx
I don't know, what you mean :roll_eyes:
Jokes aside, looks like I just forgot to paste it there :sweat_smile: Thank you :+1: