K3d: [BUG] k3d create SIGENV fail using `--agents=2` with Calico

[blaisep-vgs]

What did you do

• How was the cluster created?
  Followed the guide for basic Ingress at: https://k3d.io/usage/guides/calico/#2-create-the-cluster-without-flannel-and-with-calico
  Fixed the calico.yaml manifest to include the proper config-map,
  https://github.com/rancher/k3d/blob/eb35667705ab4371b3577aa40b7b822256e5bfd4/docs/usage/guides/calico.yaml
  and then:
  ```
  $ k3d cluster create --api-port 6550 -p "8081:80@loadbalancer" --agents 2 --k3s-server-arg '--flannel-backend=none' --volume "$(pwd):/var/lib/rancher/k3s/server/manifests"
  WARN[0000] No node filter specified
  INFO[0000] Created network 'k3d-k3s-default'
  INFO[0000] Created volume 'k3d-k3s-default-images'
  INFO[0001] Creating node 'k3d-k3s-default-server-0'
  INFO[0012] Creating node 'k3d-k3s-default-agent-0'
  INFO[0012] Creating node 'k3d-k3s-default-agent-1'
  INFO[0013] Creating LoadBalancer 'k3d-k3s-default-serverlb'
  INFO[0014] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
  ERRO[0014] Failed to create exec config for node 'k3d-k3s-default-server-0'
  panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x47d6e79]

goroutine 1 [running]:
github.com/rancher/k3d/v3/pkg/runtimes/docker.Docker.ExecInNodeGetLogs(0x4d155e0, 0xc00003c190, 0xc0003d5e00, 0xc000ea34a0, 0x3, 0x3, 0x4a64240, 0xc000c15c01, 0xc000ea34a0)
github.com/rancher/k3d/v3/pkg/runtimes/docker/node.go:311 +0x79
github.com/rancher/k3d/v3/pkg/cluster.resolveHostnameFromInside(0x4d155e0, 0xc00003c190, 0x4d25380, 0x52ff700, 0xc0003d5e00, 0x4bde2f7, 0x14, 0x41f9305, 0xc0000bc040, 0x4bc1040, ...)
github.com/rancher/k3d/v3/pkg/cluster/host.go:79 +0x17a
github.com/rancher/k3d/v3/pkg/cluster.GetHostIP(0x4d155e0, 0xc00003c190, 0x4d25380, 0x52ff700, 0xc000428900, 0x0, 0x0, 0x0, 0x0, 0x0)
github.com/rancher/k3d/v3/pkg/cluster/host.go:60 +0x17e
github.com/rancher/k3d/v3/pkg/cluster.ClusterCreate(0x4d155e0, 0xc00003c190, 0x4d25380, 0x52ff700, 0xc000428900, 0x0, 0x0)
github.com/rancher/k3d/v3/pkg/cluster/cluster.go:360 +0x1935
github.com/rancher/k3d/v3/cmd/cluster.NewCmdClusterCreate.func1(0xc000414b00, 0xc0003d8b00, 0x0, 0x8)
github.com/rancher/k3d/v3/cmd/cluster/clusterCreate.go:83 +0x14c
github.com/spf13/cobra.(Command).execute(0xc000414b00, 0xc0003d8a80, 0x8, 0x8, 0xc000414b00, 0xc0003d8a80)
github.com/spf13/[email protected]/command.go:846 +0x2c2
github.com/spf13/cobra.(Command).ExecuteC(0x52bfd40, 0xc0000320c0, 0xa, 0xa)
github.com/spf13/[email protected]/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/[email protected]/command.go:887
github.com/rancher/k3d/v3/cmd.Execute()
github.com/rancher/k3d/v3/cmd/root.go:91 +0x5a
main.main()
github.com/rancher/k3d/v3/main.go:27 +0x25

- What did you do afterwards?

I confirmed that it would work with no agents created and then I discussed this with @iwilltry42 and he suggested that I open this issue.

## What did you expect to happen

I expected:

This works (builds server-0 and serverlb containers, with Calico and Traefik 1.8)

$ k3d cluster create --api-port 6550 -p "8081:80@loadbalancer" --k3s-server-arg '--flannel-backend=none' --volume "$(pwd):/var/lib/rancher/k3s/server/manifests"
WARN[0000] No node filter specified
INFO[0000] Created network 'k3d-k3s-default'
INFO[0000] Created volume 'k3d-k3s-default-images'
INFO[0001] Creating node 'k3d-k3s-default-server-0'
INFO[0012] Creating LoadBalancer 'k3d-k3s-default-serverlb'
INFO[0013] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
INFO[0017] Successfully added host record to /etc/hosts in 2/2 nodes and to the CoreDNS ConfigMap
INFO[0017] Cluster 'k3s-default' created successfully!
```

Screenshots or terminal output

see above...

Which OS & Architecture

Darwin bpabon 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64

Which version of k3d

k3d version v3.1.3
k3s version latest (default)

Which version of docker

Client:
Debug Mode: false

Server:
Containers: 7
Running: 2
Paused: 0
Stopped: 5
Images: 384
Server Version: 19.03.13
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.76-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 5.811GiB
Name: docker-desktop
ID: P6EF:ED7L:DZQH:2D33:KVU7:VV5B:ZZI2:HEAV:FES3:YLKQ:7H3A:CZJJ
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 56
Goroutines: 59
System Time: 2020-10-24T18:25:24.675013Z
EventsListeners: 3
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

[iwilltry42]

Hi @blaisep-vgs , thanks for opening this issue! :)
Follow-Up question: what do you have at $(pwd) that you mount to the manifests folder?
Since you're mounting a whole folder, you're overwriting everything that k3s usually deploys (like CoreDNS, etc.).
You may want to mount to a subfolder instead, e.g. --volume "$(pwd):/var/lib/rancher/k3s/server/manifests/mystuff".

[iwilltry42]

Also, for debugging purposes, can you please do a docker ps -a and a docker logs k3d-k3s-default-server-0 and post the output here?

[iwilltry42]

The nil pointer exception will probably already be fixed by b5eeda7 .. can you build from main branch and try again?
If you want, I can also build it for you and link you to the executable for download.

[blaisep-vgs]

Hi @iwilltry42 , thanks for the suggestions, I will build from main and follow the new instructions re logs and volume mounts.

[blaisep-vgs]

@iwilltry42 I think you can close this issue now.
I built from main (b5eeda74d61788328e7ac513078d2ada4644f6d0) and fully qualified the volume mount.
It works now, using the instructions in the docs. :

 $ k3d cluster create --agents 2 --api-port 6550 -p "8081:80@loadbalancer" --k3s-server-arg '--flannel-backend=none' --volume "$(pwd)/docs/usage/guides/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml"

WARN[0000] No node filter specified
INFO[0000] Created network 'k3d-k3s-default'
INFO[0000] Created volume 'k3d-k3s-default-images'
INFO[0001] Creating node 'k3d-k3s-default-server-0'
INFO[0002] Pulling image 'docker.io/rancher/k3s:v1.18.9-k3s1'
INFO[0022] Creating node 'k3d-k3s-default-agent-0'
INFO[0022] Creating node 'k3d-k3s-default-agent-1'
INFO[0022] Creating LoadBalancer 'k3d-k3s-default-serverlb'
INFO[0024] Pulling image 'docker.io/rancher/k3d-proxy:v3.1.5'
INFO[0027] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
INFO[0034] Successfully added host record to /etc/hosts in 4/4 nodes and to the CoreDNS ConfigMap
INFO[0034] Cluster 'k3s-default' created successfully!

[iwilltry42]

That's great 🙂
I guess it also works without the custom build?
Still nice that you found that edge case we could fix now 👍

[blaisep-vgs]

@iwilltry42 , yes, it also works with the standard build using install.sh
I tried both methods.