K3d: [BUG] Error response from daemon: invalid reference format

[anuruddhal]

What did you do?
Run k3d create

• How was the cluster created?
  
  
  
  • k3d create
  
  

• What did you do afterwards?
  
  
  
  • k3d commands?
  
  

 anuruddha@Anuruddhas-MacBook-Pro  ~  k3d create
2019/07/24 14:33:41 Created cluster network with ID 2d5b4e7dc27b58c448df1c161d8261af5efb316564a95bd8b659b5760a522e3b
2019/07/24 14:33:41 Created docker volume  k3d-k3s-default-images
2019/07/24 14:33:41 Creating cluster [k3s-default]
2019/07/24 14:33:41 Creating server using docker.io/rancher/k3s:...
2019/07/24 14:33:41 ERROR: couldn't create container k3d-k3s-default-server
ERROR: couldn't create container k3d-k3s-default-server
Error response from daemon: invalid reference format
 ✘ anuruddha@Anuruddhas-MacBook-Pro  ~  k3d create -v /dev/mapper:/dev/mapper
2019/07/24 14:34:31 Created cluster network with ID 2d5b4e7dc27b58c448df1c161d8261af5efb316564a95bd8b659b5760a522e3b
2019/07/24 14:34:31 Created docker volume  k3d-k3s-default-images
2019/07/24 14:34:31 Creating cluster [k3s-default]
2019/07/24 14:34:31 Creating server using docker.io/rancher/k3s:...
2019/07/24 14:34:31 ERROR: couldn't create container k3d-k3s-default-server
ERROR: couldn't create container k3d-k3s-default-server
Error response from daemon: invalid reference format

- docker commands?
- OS operations (e.g. shutdown/reboot)?

What did you expect to happen?
Create the cluster.

Concise description of what you expected to happen after doing what you described above.

Screenshots or terminal output

If applicable, add screenshots or terminal output (code block) to help explain your problem.

Which OS & Architecture?
System Version: macOS 10.14.2 (18C54)
Kernel Version: Darwin 18.2.0

• Linux, Windows, MacOS / amd64, x86, ...?
  System Version: macOS 10.14.2 (18C54)
  Kernel Version: Darwin 18.2.0

Which version of k3d?

 ✘ anuruddha@Anuruddhas-MacBook-Pro  ~  k3d --version
k3d version v1.3.0

Which version of docker?

Client: Docker Engine - Community
 Version:           19.03.0-rc2
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        f97efcc
 Built:             Wed Jun  5 01:37:53 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.0-rc2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       f97efcc
  Built:            Wed Jun  5 01:42:10 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[iwilltry42]

This might actually be caused by a change in the docker API, since you're using a bleeding edge docker release there.
I'll check https://docs.docker.com/engine/api/version-history/ soon to see what's going on.
You also might want to consider updating to the latest stable release instead of the RC version.

[physcat]

I got the same error on Fedora running a slightly older version of docker:

$ k3d c; echo "======"; docker version
2019/07/24 15:41:08 Created cluster network with ID 
d7977801508e3bbbfaa0b8089d3a2eee23ba71dcb2417f2d21f9235e16c5dce1
2019/07/24 15:41:08 Created docker volume  k3d-k3s-default-images
2019/07/24 15:41:08 Creating cluster [k3s-default]
2019/07/24 15:41:08 Creating server using docker.io/rancher/k3s:...
2019/07/24 15:41:08 ERROR: couldn't create container k3d-k3s-default-server
ERROR: couldn't create container k3d-k3s-default-server
Error response from daemon: invalid reference format
======
Client:
 Version:           18.09.5
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        e8ff056
 Built:             Thu Apr 11 04:44:27 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.5
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       e8ff056
  Built:            Thu Apr 11 04:13:40 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[wxdao]

The version of docker sdk used is too old, which is v1.13.1 from 2 years ago. According to the way they publish new versions since then, we should directly get its master branch go get -u github.com/docker/docker/client@master. I've tested locally and it just works. Some minor code changes are needed tho.

[iwilltry42]

Weird enough, it works for me with the following docker version and k3d v1.3.0:

Client: Docker Engine - Community
 Version:           19.03.0
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        aeac9490dc
 Built:             Wed Jul 17 18:14:51 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.0
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       aeac9490dc
  Built:            Wed Jul 17 18:13:27 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

However, @wxdao's proposed changes are good in any case :+1:

[physcat]

@iwilltry42 I'm not sure #94 will close this. I've just tried with @wxdao's branch:

[mitch@hex Dev]$ git clone https://github.com/wxdao/k3d.git
Cloning into 'k3d'...
remote: Enumerating objects: 804, done.
remote: Counting objects: 100% (804/804), done.
remote: Compressing objects: 100% (552/552), done.
remote: Total 1902 (delta 278), reused 638 (delta 229), pack-reused 1098
Receiving objects: 100% (1902/1902), 2.05 MiB | 1.06 MiB/s, done.
Resolving deltas: 100% (777/777), done.
[mitch@hex Dev]$ cd k3d/
[mitch@hex k3d]$ git checkout fix-docker 
Branch 'fix-docker' set up to track remote branch 'fix-docker' from 'origin'.
Switched to a new branch 'fix-docker'
[mitch@hex k3d]$ make build
go build -i  -tags '' -ldflags '-w -s -X github.com/rancher/k3d/version.Version=v1.3.0-2-g058bb1d -X github.com/rancher/k3d/version.K3sVersion=' -o '/home/mitch/Dev/k3d/bin/k3d'

[mitch@hex k3d]$ ./bin/k3d -version
k3d version v1.3.0-2-g058bb1d

[mitch@hex k3d]$ ./bin/k3d c
2019/07/24 23:53:30 Created cluster network with ID d7977801508e3bbbfaa0b8089d3a2eee23ba71dcb2417f2d21f9235e16c5dce1
2019/07/24 23:53:30 Created docker volume  k3d-k3s-default-images
2019/07/24 23:53:30 Creating cluster [k3s-default]
2019/07/24 23:53:30 Creating server using docker.io/rancher/k3s:...
2019/07/24 23:53:30 ERROR: couldn't create container k3d-k3s-default-server
ERROR: couldn't create container k3d-k3s-default-server
Error response from daemon: invalid reference format

[physcat]

It looks like the release image may have been built without internet access and the K3S_TAG was not set, from the Makefile:

 14 K3S_TAG         := $(shell curl --silent "https://api.github.com/repos/rancher/k3s/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

Doing a docker pull with a tag of ... gives this error:

$ docker pull docker.io/rancher/k3s:...
invalid reference format

Rebuilding the master branch with internet access (and working DNS, so not in my hotel) works.

[iwilltry42]

@physcat good catch!
That's really bad :/ Actually, we automatically build the release artifacts with TravisCI, so I wouldn't have guessed, that the network at buildtime was a problem. I will replace the binaries now with locally built ones.
Thank you :+1:

UPDATE: I updated the release with the fixed binaries and pushed a fix to master, which will make make error out if no K3S_TAG was set (manually or automatic)

[physcat]

@iwilltry42 I just downloaded the binary and it works perfectly. Thanks.

[iwilltry42]

@anuruddhal & @physcat & @wxdao , please test it with the new binaries.
I guess @wxdao's upgrade of the docker SDK "fixed" it, because of the new local build which has the tag in it. Anyway, the upgrade might be a good idea :+1:

[wxdao]

Some random thoughts: Is it right to use a dynamic content on building? Besides some awkward problems like this issue, one can't get repeatable builds of a specific version. I think we should hardcode a tag and have it updated along releases, or simply use latest?

[iwilltry42]

We already had some discussions about this and were always unsure, since every approach has its drawbacks.
Hardcoding it means, that you always have to think of updating it, so those awkward problems might then originate from human mistake.
Latest tag unfortunately does not exist and would require changes or manual work in rancher/k3s.
So we thought that getting the latest version automatically upon build time would give the "less bad" result. i.e. quite recent, no human mistakes possible, though it might fail if GitHub or network is down.
Actually, grabbing the latest version at runtime would be cool, but at least just as error prone...

[minhnguyenvan95]

got the same error , any solution for this problem ?

[minhnguyenvan95]

just solve this problem by remove /usr/local/bin/k3d and reinstall the new version of k3d

[iwilltry42]

Just summing this up here for anyone dropping by:
This bug was caused by a bad build, where somehow the Makefile failed to fetch the latest release tag in k3s without a fallback. Thus, k3d is trying to pull an image with an undefined tag and fails.
We uploaded new binaries without this error, so you have to replace your current k3d executable with a newly downloaded one.

Thanks for your help everyone!