K3d: [BUG] Unable to use custom registries.yaml with 1.5.0

[JohnnyCrazy]

Trying to mount a local registries.yaml leads to the following error. I'm still using a custom registry deployment because I'm also attaching a service and ingress to it. I'm not 100% sure I can migrate to the new registry, because as it seems it's only a docker container, not a full k8 deployment.

INFO[0000] Created cluster network with ID a42198c8431e78135ee0302b877e2b96a7bf640ac1bcbac01ab135300035d5ab
INFO[0000] Created docker volume  k3d-k3s-default-images
INFO[0000] Creating cluster [k3s-default]
INFO[0000] Creating server using docker.io/rancher/k3s:v1.17.0-k3s.1...
INFO[0000] ERROR: Cluster creation failed, rolling back...
INFO[0000] Removing cluster [k3s-default]
INFO[0000] ...Removing server
INFO[0000] ...Removing docker image volume
INFO[0000] Removed cluster [k3s-default]
FATA[0000] failed to copy source code: Error response from daemon: Error processing tar file(exit status 1): unlinkat /etc/rancher/k3s/registries.yaml: device or resource busy

• How was the cluster created?

k3d create\
    -v $HOME:/host\
    -v /dev/mapper:/dev/mapper\
    -v $HOME/projects/my_kubernetes_cluster/devsetup/registries.yaml:/etc/rancher/k3s/registries.yaml\
    --publish 80:80\ #traefik
    --publish 443:443\ #traefik
    --publish 5432:5432\ #postgres
    --publish 32000:32000\ #registry
    --auto-restart\
    --server-arg --no-deploy=traefik
}

registries.yaml

mirrors:
  "registry.by":
    endpoint:
    - http://localhost:32000

• What did you do afterwards?

What did you expect to happen?

It should be able to mount my custom registries.yaml like in 1.4.0.

Which OS & Architecture?

• Linux 4.19.84-microsoft-standard #1 SMP Wed Nov 13 11:44:37 UTC 2019 x86_64 GNU/Linux
• WSL 2

Which version of k3d?

k3d version v1.5.0

Which version of docker?

➜ docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:29:52 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[inercia]

This couild be solved by just putting your registries.yaml file in $HOME/.k3d/registries.yaml. It will be automatically used by k3s, there is no need to mount it anymore with -v.

I think 1) this should be documented and 2) we should make sure users are not trying to mount a file on /etc/rancher/k3s/registries.yaml.

[JohnnyCrazy]

Ah ok, I'll try with copying or a symlink and will report back.

Out of curiosity, does it make sense to have a cluster configuration as a dotfile in the home directory? Since k3d is able to spin up multiple differently configured clusters, I would expect a --registries-file (or similar) flag for the create command.

[iwilltry42]

This was actually a bug @JohnnyCrazy , thanks for reporting this :+1:
I'll release bugfix release v1.5.1 now :+1: