K3d: [BUG] : Longhorn install on K3D

[fabricev]

What did you do

kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml

• How was the cluster created?

k3d cluster create tools-cluster --agents=3 --api-port=10448 -v /data/longhorn:/var/lib/longhorn:shared

• What did you do afterwards?

N/A

What did you expect to happen

Longhorn managers pods from the daemonset created and running

Screenshots or terminal output

kubectl get pods -n longhorn-system 

NAME                                        READY   STATUS                 RESTARTS   AGE
longhorn-ui-65d76ddf9b-ns8g2                1/1     Running                0          6m1s
longhorn-driver-deployer-8558cf45fb-7pgkt   0/1     Init:0/1               0          6m1s
longhorn-manager-xqwhb                      0/1     CreateContainerError   0          6m1s
longhorn-manager-cmgzh                      0/1     CreateContainerError   0          6m1s
longhorn-manager-ddxgk                      0/1     CreateContainerError   0          6m1s
longhorn-manager-w7ldg                      0/1     CreateContainerError   0          6m1s

Which OS & Architecture

Ubuntu 20.04.1 on VMWare Workstation Player 15 / Windows 10

Which version of k3d

k3d version v3.0.2-5-g60069f6
k3s version v1.18.9-k3s1 (default)

Which version of docker

docker version

Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:02:52 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:01:20 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

docker info

```
Client:
Debug Mode: false

Server:
Containers: 68
Running: 5
Paused: 0
Stopped: 63
Images: 247
Server Version: 19.03.13
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-48-generic
Operating System: Ubuntu 20.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.61GiB
Name: ubuntu
ID: 5IB7:5YQH:X4OA:6MN6:X3EX:SA7E:UQJ7:UHYQ:YWGC:W6VR:JHNG:SGZL
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: fabricevergnenegre
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
````

[fabricev]

Followed the recommendation (in #206) and installed k3d beta, but facing more or the same issue on /var/run

` kubectl describe pods -n longhorn-system longhorn-manager-cmgzh ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned longhorn-system/longhorn-manager-cmgzh to k3d-tools-cluster-agent-2 Normal Pulling 27m kubelet, k3d-tools-cluster-agent-2 Pulling image "longhornio/longhorn-manager:v1.0.2" Normal Pulled 26m kubelet, k3d-tools-cluster-agent-2 Successfully pulled image "longhornio/longhorn-manager:v1.0.2" Warning Failed 26m kubelet, k3d-tools-cluster-agent-2 Error: failed to generate container "4b43b0690f952c10de6085b9787032a17cf1be2b7fb5b7422fe73f40ed0916cb" spec: path "/var/run/" is mounted on "/var/run" but it is not a shared mount Warning Failed 26m kubelet, k3d-tools-cluster-agent-2 Error: failed to generate container "b474522974b5634fafd18a72bfd3865b367e05551cc76e08faa7ac6ea74c7c7f" spec: path "/var/run/" is mounted on "/var/run" but it is not a shared mount Warning Failed 26m kubelet, k3d-tools-cluster-agent-2 Error: failed to generate container "cc1650a3d32c155661f639af1b66bb3cb2c4cf973df422e88f103fd6a0574d2a" spec: path "/var/run/" is mounted on "/var/run" but it is not a shared mount Warning Failed 26m kubelet, k3d-tools-cluster-agent-2 Error: failed to generate container "97f0faef27a374f4427103c6f808cdda6520652a85c4923cdd6af9be5c32a3fd" spec: path "/var/run/" is mounted on "/var/run" but it is not a shared mount Warning Failed 26m kubelet, k3d-tools-cluster-agent-2 Error: failed to generate container "b8a7fcb5a46aad1f2a24bf9b059409e3cbcc49617bfc7d40002fcf342a8356a8" spec: path "/var/run/" is mounted on "/var/run" but it is not a shared mount Warning Failed 26m kubelet, k3d-tools-cluster-agent-2 Error: failed to generate container "50398857c5c6e7e1a7643dd9ed14d83b815d0dfd6a4e164adb177807c05ed44f" spec: path "/var/run/" is mounted on "/var/run" but it is not a shared mount Warning Failed 25m kubelet, k3d-tools-cluster-agent-2 Error: failed to generate container "07935e3033ac1520f3f89b149b5076ad740da14009d3b9293d1f22cd4a14a0a3" spec: path "/var/run/" is mounted on "/var/run" but it is not a shared mount Warning Failed 25m kubelet, k3d-tools-cluster-agent-2 Error: failed to generate container "b07eca6b096510b251dfc1b37375a1efb41d99005279af53c029e151343099e9" spec: path "/var/run/" is mounted on "/var/run" but it is not a shared mount Warning Failed 25m kubelet, k3d-tools-cluster-agent-2 Error: failed to generate container "17d74cb9f895c2accaa1291767db84d43c90917754d63c0eb16df150b832ba05" spec: path "/var/run/" is mounted on "/var/run" but it is not a shared mount

[mamiu]

@fabricev Your issue is the longhorn.yaml, because it tries to mount on /var/run as you can see here.

But interesting that your cluster create command works, because it's already failing for me if I run:

k3d cluster create demo --volume /private/tmp/longhorn:/var/lib/longhorn:shared

That's the error message I'm getting:

WARN[0000] No node filter specified
INFO[0000] Created network 'k3d-demo15'
INFO[0000] Created volume 'k3d-demo15-images'
INFO[0001] Creating node 'k3d-demo15-server-0'
ERRO[0001] Failed to start container
ERRO[0001] Failed to create node 'k3d-demo15-server-0'
ERRO[0001] Failed to create node
ERRO[0001] Error response from daemon: path /host_mnt/private/tmp/longhorn is mounted on /host_mnt but it is not a shared mount
ERRO[0001] Failed to create cluster >>> Rolling Back
INFO[0001] Deleting cluster 'demo15'
INFO[0001] Deleted k3d-demo15-server-0
INFO[0001] Deleting cluster network '39b9e2331ab514036390a780cda81f365f36cbade7b6972524d5092e46cc96f1'
FATA[0001] Cluster creation FAILED, all changes have been rolled back!

I'm using macOS Catalina and have no clue where the /host_mnt comes from.

Edit:

It seems to be related with that bug but I have the latest docker version where it should be fixed (according tho these release notes).

[rdxmb]

Same problem here on ubuntu 20.04

The following workarounds did not help:

MountFlags=shared in /etc/default/docker (with docker restart of course)

$ k3d cluster create test -v /home/rdxmb/k3d/run:/var/run:shared
WARN[0000] No node filter specified                     
INFO[0000] Created network 'k3d-test'                   
INFO[0000] Created volume 'k3d-test-images'             
INFO[0001] Creating node 'k3d-test-server-0'            
ERRO[0001] Failed to create container 'k3d-test-server-0' 
ERRO[0001] Failed to create node 'k3d-test-server-0'    
ERRO[0001] Failed to create node                        
ERRO[0001] Error response from daemon: Duplicate mount point: /var/run 
ERRO[0001] Failed to create cluster >>> Rolling Back    
INFO[0001] Deleting cluster 'test'                      
ERRO[0001] Failed to delete container 'k3d-test-server-0' 
WARN[0001] Failed to delete node 'k3d-test-server-0': Try to delete it manually 
INFO[0001] Deleting cluster network '8eaf162f5a0f1829bfd0953abfe160848c97ea98da5b64e20c72b000fa2007dc' 
ERRO[0001] Failed to delete 1 nodes: Try to delete them manually 
FATA[0001] Cluster creation FAILED, also FAILED to rollback changes! 

# kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
# k3d version
k3d version v3.1.3
k3s version v1.18.9-k3s1 (default)

[iwilltry42]

Hi @rdxmb , actually your workaround doesn't work because of a different issue.
See this line in your logs: ERRO[0001] Error response from daemon: Duplicate mount point: /var/run.
k3d by default has tmpfs mounts for /run and /var/run (check the code).
You can however have additional volume mounts for subpaths of /var/run.

[rdxmb]

@iwilltry42 Thanks. So it would be interesting, if mounting subpaths is enough for longhorn...

[balibebas]

@mamiu I believe in your case the issue—which I'm also bumping up against right now and it's a real head-scratcher—is rooted in https://github.com/docker/for-mac/issues/3431 (still open and not receiving enough attention IMO). A simple way to reproduce w/o k3d:

docker run --rm -it -v /tmp:/mnt/tmp:shared alpine sh

You should see output like:

docker: Error response from daemon: path /host_mnt/private/tmp is mounted on /host_mnt but it is not a shar
ed mount.

EDIT: Per Docker Docs "Bind propagation defaults to rprivate for both bind mounts and volumes. It is only configurable for bind mounts, and only on Linux host machines." The way I read that shared is an illegal value unless os is *nix—which helps explain why what the OP is trying failed/fails on macOS (and should also fail on WSL). Guilty of RTFM on my part. 😄

[mamiu]

@balibebas You're right, that seems to be the issue for me! Thanks for investigating it and letting me know.

@fabricev Did my previous comment help you to solve your error? If so you can close this issue.

[doug62]

I'm having a similar problem on K3s trying to get Longhorn going

[balibebas]

@doug62 if by K3s you mean the docker wrapper k3d let me save you some time and suggest to observe the iSCSI requirements of the device used and make sure you have dynamic provisioning enabled. My issue with Longhorn was because I was trying to run on macOS and didn't have iSCSI enabled (doable but a rather large hack). If you're doing stuff with K3s outside of docker your issue is not this issue.