K3d: [BUG] Failed waiting for log message 'start worker processes' from node 'k3d-cluster3Bplus-serverlb'
What did you do
How was the cluster created?
k3d cluster create cluster3Bplus
What did you do afterwards?
- Nothing
What did you expect to happen
Create without any errors !
Screenshots or terminal output
~ % k3d cluster create cluster3Bplus --trace
DEBU[0000] Selected runtime is 'docker.Docker'
DEBU[0000] API-Port Mapping didn't specify hostPort, choosing one randomly...
DEBU[0000] Got free port for API: '35307'
TRAC[0000] PortFilterMap: map[]
TRAC[0000] LabelFilterMap: map[]
DEBU[0000] '--update-default-kubeconfig set: enabling wait-for-server
INFO[0000] Created network 'k3d-cluster3Bplus'
INFO[0000] Created volume 'k3d-cluster3Bplus-images'
INFO[0001] Creating node 'k3d-cluster3Bplus-server-0'
TRAC[0001] Creating node from spec
&{Name:k3d-cluster3Bplus-server-0 Role:server Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:[k3d-cluster3Bplus-images:/k3d/images] Env:[K3S_TOKEN=kMdwgZZPGeacyOqeuBmc] Cmd:[] Args:[] Ports:[] Restart:false Labels:map[k3d.cluster:cluster3Bplus k3d.cluster.imageVolume:k3d-cluster3Bplus-images k3d.cluster.network:cb83c98f6b5d394d49c2ce90f107a24ff93e8ba64ec1a339f952d0c824373fc8 k3d.cluster.network.external:false k3d.cluster.token:kMdwgZZPGeacyOqeuBmc k3d.cluster.url:https://k3d-cluster3Bplus-server-0:6443] Network:cb83c98f6b5d394d49c2ce90f107a24ff93e8ba64ec1a339f952d0c824373fc8 ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host:0.0.0.0 HostIP:0.0.0.0 Port:35307}} AgentOpts:{} State:{Running:false Status:}}
TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-cluster3Bplus-server-0 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=kMdwgZZPGeacyOqeuBmc K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san 0.0.0.0] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:cluster3Bplus k3d.cluster.imageVolume:k3d-cluster3Bplus-images k3d.cluster.network:cb83c98f6b5d394d49c2ce90f107a24ff93e8ba64ec1a339f952d0c824373fc8 k3d.cluster.network.external:false k3d.cluster.token:kMdwgZZPGeacyOqeuBmc k3d.cluster.url:https://k3d-cluster3Bplus-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:35307] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-cluster3Bplus-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0x2406624} NetworkingConfig:{EndpointsConfig:map[cb83c98f6b5d394d49c2ce90f107a24ff93e8ba64ec1a339f952d0c824373fc8:0x242e060]}}
DEBU[0001] Created container k3d-cluster3Bplus-server-0 (ID: e7f0984f1cc3c5297be07a9225683417ab8cd8da35150e01d633397731b53853)
DEBU[0003] Created node 'k3d-cluster3Bplus-server-0'
DEBU[0003] Waiting for server node 'k3d-cluster3Bplus-server-0' to get ready
DEBU[0049] Finished waiting for log message 'k3s is up and running' from node 'k3d-cluster3Bplus-server-0'
INFO[0049] Creating LoadBalancer 'k3d-cluster3Bplus-serverlb'
TRAC[0049] Creating node from spec
&{Name:k3d-cluster3Bplus-serverlb Role:loadbalancer Image:docker.io/rancher/k3d-proxy:v3.1.5 Volumes:[] Env:[SERVERS=k3d-cluster3Bplus-server-0 PORTS=6443 WORKER_PROCESSES=1] Cmd:[] Args:[] Ports:[0.0.0.0:35307:6443/tcp] Restart:false Labels:map[app:k3d k3d.cluster:cluster3Bplus] Network:cb83c98f6b5d394d49c2ce90f107a24ff93e8ba64ec1a339f952d0c824373fc8 ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host: HostIP: Port:}} AgentOpts:{} State:{Running:false Status:}}
TRAC[0049] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-cluster3Bplus-serverlb Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[6443/tcp:{}] Tty:false OpenStdin:false StdinOnce:false Env:[SERVERS=k3d-cluster3Bplus-server-0 PORTS=6443 WORKER_PROCESSES=1 K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3d-proxy:v3.1.5 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:cluster3Bplus k3d.role:loadbalancer] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[6443/tcp:[{HostIP:0.0.0.0 HostPort:35307}]] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0x56ba804} NetworkingConfig:{EndpointsConfig:map[cb83c98f6b5d394d49c2ce90f107a24ff93e8ba64ec1a339f952d0c824373fc8:0x56b87e0]}}
DEBU[0050] Created container k3d-cluster3Bplus-serverlb (ID: 0cd82d0dbf4f2ffd1bffe18690af6ce533488dfe73ba0e86ebac7e5596aeb54c)
DEBU[0051] Starting to wait for loadbalancer node 'k3d-cluster3Bplus-serverlb'
ERRO[0052] Failed waiting for log message 'start worker processes' from node 'k3d-cluster3Bplus-serverlb'
ERRO[0052] Failed to bring up all server nodes (and loadbalancer) in time. Check the logs:
ERRO[0052] >>> Node 'k3d-cluster3Bplus-serverlb' (container '0cd82d0dbf4f2ffd1bffe18690af6ce533488dfe73ba0e86ebac7e5596aeb54c') not running
ERRO[0052] Failed to bring up cluster
ERRO[0052] Failed to create cluster >>> Rolling Back
INFO[0052] Deleting cluster 'cluster3Bplus'
DEBU[0052] Cluster Details: &{Name:cluster3Bplus Network:{Name:cb83c98f6b5d394d49c2ce90f107a24ff93e8ba64ec1a339f952d0c824373fc8 External:false} Token:kMdwgZZPGeacyOqeuBmc Nodes:[0x21246e0 0x4e3c5a0] InitNode:<nil> ExternalDatastore:{Endpoint: CAFile: CertFile: KeyFile: Network:} CreateClusterOpts:0x2380330 ExposeAPI:{Host:0.0.0.0 HostIP:0.0.0.0 Port:35307} ServerLoadBalancer:0x2124640 ImageVolume:}
INFO[0052] Deleted k3d-cluster3Bplus-server-0
INFO[0052] Deleted k3d-cluster3Bplus-serverlb
INFO[0052] Deleting cluster network 'cb83c98f6b5d394d49c2ce90f107a24ff93e8ba64ec1a339f952d0c824373fc8'
FATA[0052] Cluster creation FAILED, all changes have been rolled back!
Which OS & Architecture
- Raspbian (10 / Buster) on raspberrypi 3B+ -> arm
Which version of k3d
k3d version v3.1.5
k3s version v1.18.9-k3s1 (default)
Which version of docker
- output of
docker versionanddocker info
~ % docker version
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46
Built: Wed Sep 16 17:07:02 2020
OS/Arch: linux/arm
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.13
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4484c46
Built: Wed Sep 16 17:00:52 2020
OS/Arch: linux/arm
Experimental: false
containerd:
Version: 1.3.7
GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
~ % docker info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 4
Server Version: 19.03.13
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 5.4.72-v7+
Operating System: Raspbian GNU/Linux 10 (buster)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 924.8MiB
Name: raspberrypi3Bplus
ID: I6NQ:UBMA:BYKN:IGA7:BME5:U3M3:77QV:XXIR:6FLS:FWAC:M2UY:CFT6
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support
bug
GannonTdW
All 15 comments
Hum, I didn’t see that.
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No kernel memory TCP limit support
WARNING: No oom kill disable support
Could that be the problem?
GannonTdW
on 26 Oct 2020
Hi @GannonTdW thanks for opening this issue.
First off: I'm on my phone, so I may have missed some details and I didn't do any tests.
Then, it could be, that k3d (especially with multiple nodes) is not that performant on the RPi (as already docker uses quite some resources there and then you have 4 k3s containers).
But anyway.. for some container features, you'll need to append the following to /boot/cmdline.txt on your RPi (and then reboot):
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
Let me know, if that already did the trick 👍
iwilltry42
on 26 Oct 2020
Hello @iwilltry42,
Thank you for the answer.
Nop, always the same issue.
I thought the problem might be from k3S, so I installed k3s ... but k3s works fine !
GannonTdW
on 26 Oct 2020
@GannonTdW , then let's start debugging.
Can you please try to create the cluster again, but this time add the --no-rollback flag to leave the containers there in failed mode instead of cleaning up?
Then please paste the output of docker ps -a and docker logs k3d-cluster3Bplus-server-0 and k3d-cluster3Bplus-serverlb here.
iwilltry42
on 27 Oct 2020
Hello @iwilltry42,
first I have unstall K3S.
k3d cluster create cluster3Bplus --trace --no-rollback
DEBU[0000] Selected runtime is 'docker.Docker'
DEBU[0000] API-Port Mapping didn't specify hostPort, choosing one randomly...
DEBU[0000] Got free port for API: '45435'
TRAC[0000] PortFilterMap: map[]
TRAC[0000] LabelFilterMap: map[]
DEBU[0000] '--update-default-kubeconfig set: enabling wait-for-server
INFO[0001] Created network 'k3d-cluster3Bplus'
INFO[0001] Created volume 'k3d-cluster3Bplus-images'
INFO[0002] Creating node 'k3d-cluster3Bplus-server-0'
TRAC[0002] Creating node from spec
&{Name:k3d-cluster3Bplus-server-0 Role:server Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:[k3d-cluster3Bplus-images:/k3d/images] Env:[K3S_TOKEN=WPokfHtFnYeoiSzecUdh] Cmd:[] Args:[] Ports:[] Restart:false Labels:map[k3d.cluster:cluster3Bplus k3d.cluster.imageVolume:k3d-cluster3Bplus-images k3d.cluster.network:cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66 k3d.cluster.network.external:false k3d.cluster.token:WPokfHtFnYeoiSzecUdh k3d.cluster.url:https://k3d-cluster3Bplus-server-0:6443] Network:cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66 ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host:0.0.0.0 HostIP:0.0.0.0 Port:45435}} AgentOpts:{} State:{Running:false Status:}}
TRAC[0002] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-cluster3Bplus-server-0 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=WPokfHtFnYeoiSzecUdh K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san 0.0.0.0] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:cluster3Bplus k3d.cluster.imageVolume:k3d-cluster3Bplus-images k3d.cluster.network:cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66 k3d.cluster.network.external:false k3d.cluster.token:WPokfHtFnYeoiSzecUdh k3d.cluster.url:https://k3d-cluster3Bplus-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:45435] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-cluster3Bplus-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0x32f36b8} NetworkingConfig:{EndpointsConfig:map[cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66:0x32cff20]}}
DEBU[0002] Created container k3d-cluster3Bplus-server-0 (ID: e483e034df9c5d9e1769fc2372c94db702340b4b5312f27f25606941bca16e4d)
DEBU[0004] Created node 'k3d-cluster3Bplus-server-0'
DEBU[0004] Waiting for server node 'k3d-cluster3Bplus-server-0' to get ready
DEBU[0064] Finished waiting for log message 'k3s is up and running' from node 'k3d-cluster3Bplus-server-0'
INFO[0064] Creating LoadBalancer 'k3d-cluster3Bplus-serverlb'
TRAC[0064] Creating node from spec
&{Name:k3d-cluster3Bplus-serverlb Role:loadbalancer Image:docker.io/rancher/k3d-proxy:v3.1.5 Volumes:[] Env:[SERVERS=k3d-cluster3Bplus-server-0 PORTS=6443 WORKER_PROCESSES=1] Cmd:[] Args:[] Ports:[0.0.0.0:45435:6443/tcp] Restart:false Labels:map[app:k3d k3d.cluster:cluster3Bplus] Network:cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66 ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host: HostIP: Port:}} AgentOpts:{} State:{Running:false Status:}}
TRAC[0064] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-cluster3Bplus-serverlb Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[6443/tcp:{}] Tty:false OpenStdin:false StdinOnce:false Env:[SERVERS=k3d-cluster3Bplus-server-0 PORTS=6443 WORKER_PROCESSES=1 K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3d-proxy:v3.1.5 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:cluster3Bplus k3d.role:loadbalancer] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[6443/tcp:[{HostIP:0.0.0.0 HostPort:45435}]] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0x61f8954} NetworkingConfig:{EndpointsConfig:map[cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66:0x618b6e0]}}
DEBU[0065] Created container k3d-cluster3Bplus-serverlb (ID: 4807ecf4397a427d82550c7fa1bcce886cbd47b49648b700ba22549fe71d2031)
DEBU[0067] Starting to wait for loadbalancer node 'k3d-cluster3Bplus-serverlb'
ERRO[0069] Failed waiting for log message 'start worker processes' from node 'k3d-cluster3Bplus-serverlb'
ERRO[0069] Failed to bring up all server nodes (and loadbalancer) in time. Check the logs:
ERRO[0069] >>> Node 'k3d-cluster3Bplus-serverlb' (container '4807ecf4397a427d82550c7fa1bcce886cbd47b49648b700ba22549fe71d2031') not running
ERRO[0069] Failed to bring up cluster
FATA[0069] Cluster creation FAILED, rollback deactivated.
Docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4807ecf4397a rancher/k3d-proxy:v3.1.5 "/bin/sh -c nginx-pr…" 54 seconds ago Exited (1) 49 seconds ago k3d-cluster3Bplus-serverlb
e483e034df9c rancher/k3s:v1.18.9-k3s1 "/bin/k3s server --t…" About a minute ago Exited (1) 53 seconds ago k3d-cluster3Bplus-server-0
docker logs k3d-cluster3Bplus-server-0
time="2020-10-27T17:06:57.123420606Z" level=info msg="Starting k3s v1.18.9+k3s1 (630bebf9)"
time="2020-10-27T17:06:57.249624135Z" level=info msg="Kine listening on unix://kine.sock"
time="2020-10-27T17:07:01.024560028Z" level=info msg="Active TLS secret (ver=) (count 8): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.18.0.2:172.18.0.2 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/hash:487d94c3733c645ce6dc9677524068e59fa12dab6573a055718cc933c7fefd97]"
time="2020-10-27T17:07:01.076203672Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --basic-auth-file=/var/lib/rancher/k3s/server/cred/passwd --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Flag --basic-auth-file has been deprecated, Basic authentication mode is deprecated and will be removed in a future release. It is not recommended for production environments.
I1027 17:07:01.103183 6 server.go:645] external host was not specified, using 172.18.0.2
I1027 17:07:01.105053 6 server.go:162] Version: v1.18.9+k3s1
I1027 17:07:24.089610 6 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I1027 17:07:24.089748 6 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I1027 17:07:24.104560 6 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I1027 17:07:24.104701 6 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I1027 17:07:24.327833 6 master.go:270] Using reconciler: lease
I1027 17:07:24.713532 6 rest.go:113] the default service ipfamily for this cluster is: IPv4
W1027 17:07:27.170471 6 genericapiserver.go:409] Skipping API batch/v2alpha1 because it has no resources.
W1027 17:07:27.340434 6 genericapiserver.go:409] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W1027 17:07:27.491529 6 genericapiserver.go:409] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W1027 17:07:27.632060 6 genericapiserver.go:409] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W1027 17:07:27.658137 6 genericapiserver.go:409] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W1027 17:07:27.768546 6 genericapiserver.go:409] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W1027 17:07:27.912889 6 genericapiserver.go:409] Skipping API apps/v1beta2 because it has no resources.
W1027 17:07:27.913188 6 genericapiserver.go:409] Skipping API apps/v1beta1 because it has no resources.
I1027 17:07:27.986906 6 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I1027 17:07:27.987180 6 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I1027 17:07:43.923113 6 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
I1027 17:07:43.923315 6 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
I1027 17:07:43.927027 6 dynamic_serving_content.go:130] Starting serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key
I1027 17:07:43.933859 6 secure_serving.go:178] Serving securely on 127.0.0.1:6444
I1027 17:07:43.948548 6 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I1027 17:07:43.948825 6 shared_informer.go:223] Waiting for caches to sync for cluster_authentication_trust_controller
I1027 17:07:43.949069 6 tlsconfig.go:240] Starting DynamicServingCertificateController
I1027 17:07:43.956666 6 autoregister_controller.go:141] Starting autoregister controller
I1027 17:07:43.956750 6 cache.go:32] Waiting for caches to sync for autoregister controller
I1027 17:07:43.956880 6 controller.go:81] Starting OpenAPI AggregationController
I1027 17:07:43.958958 6 apiservice_controller.go:94] Starting APIServiceRegistrationController
I1027 17:07:43.959027 6 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I1027 17:07:43.959195 6 crd_finalizer.go:266] Starting CRDFinalizer
I1027 17:07:43.964807 6 available_controller.go:387] Starting AvailableConditionController
I1027 17:07:43.965153 6 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I1027 17:07:43.966140 6 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
I1027 17:07:43.966316 6 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
2020-10-27 17:07:43.982863 I | http: TLS handshake error from 127.0.0.1:46804: EOF
I1027 17:07:43.987236 6 crdregistration_controller.go:111] Starting crd-autoregister controller
I1027 17:07:43.987303 6 shared_informer.go:223] Waiting for caches to sync for crd-autoregister
I1027 17:07:43.987493 6 controller.go:86] Starting OpenAPI controller
I1027 17:07:43.987552 6 customresource_discovery_controller.go:209] Starting DiscoveryController
I1027 17:07:43.987612 6 naming_controller.go:291] Starting NamingConditionController
I1027 17:07:43.987666 6 establishing_controller.go:76] Starting EstablishingController
I1027 17:07:44.001790 6 nonstructuralschema_controller.go:186] Starting NonStructuralSchemaConditionController
I1027 17:07:44.001958 6 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I1027 17:07:44.792833 6 shared_informer.go:230] Caches are synced for crd-autoregister
I1027 17:07:44.819673 6 cache.go:39] Caches are synced for autoregister controller
I1027 17:07:44.823436 6 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I1027 17:07:44.863823 6 shared_informer.go:230] Caches are synced for cluster_authentication_trust_controller
I1027 17:07:44.864087 6 cache.go:39] Caches are synced for AvailableConditionController controller
I1027 17:07:44.951671 6 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I1027 17:07:44.951772 6 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I1027 17:07:45.177222 6 storage_scheduling.go:134] created PriorityClass system-node-critical with value 2000001000
E1027 17:07:45.237481 6 controller.go:151] Unable to perform initial Kubernetes service initialization: Service "kubernetes" is invalid: spec.clusterIP: Invalid value: "10.43.0.1": cannot allocate resources of type serviceipallocations at this time
E1027 17:07:45.258876 6 controller.go:156] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/172.18.0.2, ResourceVersion: 0, AdditionalErrorMsg:
I1027 17:07:45.295365 6 storage_scheduling.go:134] created PriorityClass system-cluster-critical with value 2000000000
I1027 17:07:45.295483 6 storage_scheduling.go:143] all system priority classes are created successfully or already exist.
I1027 17:07:48.274080 6 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I1027 17:07:48.553607 6 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
W1027 17:07:49.005856 6 lease.go:224] Resetting endpoints for master service "kubernetes" to [172.18.0.2]
I1027 17:07:49.013240 6 controller.go:606] quota admission added evaluator for: endpoints
I1027 17:07:49.038185 6 controller.go:606] quota admission added evaluator for: endpointslices.discovery.k8s.io
I1027 17:07:49.154546 6 registry.go:150] Registering EvenPodsSpread predicate and priority function
I1027 17:07:49.154921 6 registry.go:150] Registering EvenPodsSpread predicate and priority function
time="2020-10-27T17:07:49.156698762Z" level=info msg="Running kube-scheduler --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --secure-port=0"
time="2020-10-27T17:07:49.159986896Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
I1027 17:07:49.211980 6 controllermanager.go:161] Version: v1.18.9+k3s1
I1027 17:07:49.223157 6 deprecated_insecure_serving.go:53] Serving insecurely on [::]:10252
time="2020-10-27T17:07:49.251702393Z" level=info msg="Waiting for cloudcontroller rbac role to be created"
I1027 17:07:49.334001 6 registry.go:150] Registering EvenPodsSpread predicate and priority function
I1027 17:07:49.334112 6 registry.go:150] Registering EvenPodsSpread predicate and priority function
time="2020-10-27T17:07:49.347644255Z" level=info msg="Creating CRD addons.k3s.cattle.io"
W1027 17:07:49.351397 6 authorization.go:47] Authorization is disabled
W1027 17:07:49.351465 6 authentication.go:40] Authentication is disabled
I1027 17:07:49.351516 6 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
time="2020-10-27T17:07:49.550750822Z" level=info msg="Creating CRD helmcharts.helm.cattle.io"
time="2020-10-27T17:07:49.823167358Z" level=info msg="Waiting for CRD addons.k3s.cattle.io to become available"
I1027 17:07:50.413286 6 plugins.go:100] No cloud provider specified.
time="2020-10-27T17:07:50.433687189Z" level=info msg="Waiting for cloudcontroller rbac role to be created"
I1027 17:07:50.436114 6 shared_informer.go:223] Waiting for caches to sync for tokens
time="2020-10-27T17:07:50.457185998Z" level=info msg="Done waiting for CRD addons.k3s.cattle.io to become available"
time="2020-10-27T17:07:50.457321519Z" level=info msg="Waiting for CRD helmcharts.helm.cattle.io to become available"
I1027 17:07:50.539633 6 shared_informer.go:230] Caches are synced for tokens
I1027 17:07:50.612544 6 controller.go:606] quota admission added evaluator for: serviceaccounts
I1027 17:07:50.697677 6 node_lifecycle_controller.go:78] Sending events to api server
E1027 17:07:50.710451 6 core.go:229] failed to start cloud node lifecycle controller: no cloud provider provided
W1027 17:07:50.710550 6 controllermanager.go:525] Skipping "cloud-node-lifecycle"
I1027 17:07:50.962522 6 controllermanager.go:533] Started "persistentvolume-expander"
I1027 17:07:50.966575 6 expand_controller.go:319] Starting expand controller
I1027 17:07:50.966654 6 shared_informer.go:223] Waiting for caches to sync for expand
time="2020-10-27T17:07:51.108356954Z" level=info msg="Done waiting for CRD helmcharts.helm.cattle.io to become available"
time="2020-10-27T17:07:51.290019431Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-1.81.0.tgz"
time="2020-10-27T17:07:51.291522299Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml"
time="2020-10-27T17:07:51.292414333Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
time="2020-10-27T17:07:51.293031990Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
time="2020-10-27T17:07:51.293618711Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
time="2020-10-27T17:07:51.294222775Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
time="2020-10-27T17:07:51.294814964Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
time="2020-10-27T17:07:51.295376059Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml"
time="2020-10-27T17:07:51.296178561Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
time="2020-10-27T17:07:51.297225699Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/local-storage.yaml"
time="2020-10-27T17:07:51.298011482Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
time="2020-10-27T17:07:51.298819296Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
time="2020-10-27T17:07:51.299668621Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
I1027 17:07:51.513324 6 controllermanager.go:533] Started "job"
I1027 17:07:51.514931 6 job_controller.go:145] Starting job controller
I1027 17:07:51.515395 6 shared_informer.go:223] Waiting for caches to sync for job
time="2020-10-27T17:07:51.560471814Z" level=info msg="Waiting for cloudcontroller rbac role to be created"
W1027 17:07:51.773324 6 probe.go:268] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
I1027 17:07:51.777570 6 controllermanager.go:533] Started "attachdetach"
I1027 17:07:51.779720 6 attach_detach_controller.go:348] Starting attach detach controller
I1027 17:07:51.779909 6 shared_informer.go:223] Waiting for caches to sync for attach detach
time="2020-10-27T17:07:51.826270070Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
time="2020-10-27T17:07:51.826750957Z" level=info msg="To join node to cluster: k3s agent -s https://172.18.0.2:6443 -t ${NODE_TOKEN}"
time="2020-10-27T17:07:51.866131314Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
time="2020-10-27T17:07:51.882557604Z" level=info msg="Waiting for master node startup: resource name may not be empty"
2020-10-27 17:07:52.649754 I | http: TLS handshake error from 127.0.0.1:55128: remote error: tls: bad certificate
I1027 17:07:52.684115 6 controllermanager.go:533] Started "ttl"
W1027 17:07:52.684226 6 controllermanager.go:525] Skipping "ttl-after-finished"
I1027 17:07:52.711378 6 ttl_controller.go:118] Starting TTL controller
I1027 17:07:52.711467 6 shared_informer.go:223] Waiting for caches to sync for TTL
time="2020-10-27T17:07:52.735779237Z" level=info msg="Waiting for cloudcontroller rbac role to be created"
time="2020-10-27T17:07:52.900733070Z" level=info msg="Waiting for master node startup: resource name may not be empty"
time="2020-10-27T17:07:52.901463644Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
time="2020-10-27T17:07:52.901668020Z" level=info msg="Starting batch/v1, Kind=Job controller"
time="2020-10-27T17:07:52.997341114Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2020-10-27T17:07:52.998616794Z" level=info msg="Starting /v1, Kind=Node controller"
time="2020-10-27T17:07:52.998880076Z" level=info msg="Starting /v1, Kind=Service controller"
time="2020-10-27T17:07:52.999204295Z" level=info msg="Starting /v1, Kind=Pod controller"
time="2020-10-27T17:07:52.999432733Z" level=info msg="Starting /v1, Kind=Endpoints controller"
I1027 17:07:53.035744 6 controllermanager.go:533] Started "endpoint"
I1027 17:07:53.036688 6 endpoints_controller.go:181] Starting endpoint controller
I1027 17:07:53.036917 6 shared_informer.go:223] Waiting for caches to sync for endpoint
time="2020-10-27T17:07:53.113271958Z" level=info msg="Active TLS secret k3s-serving (ver=178) (count 8): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.18.0.2:172.18.0.2 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/hash:487d94c3733c645ce6dc9677524068e59fa12dab6573a055718cc933c7fefd97]"
I1027 17:07:53.140429 6 controller.go:606] quota admission added evaluator for: addons.k3s.cattle.io
time="2020-10-27T17:07:53.211941928Z" level=info msg="Wrote kubeconfig /output/kubeconfig.yaml"
time="2020-10-27T17:07:53.212082032Z" level=info msg="Run: k3s kubectl"
time="2020-10-27T17:07:53.212127918Z" level=info msg="k3s is up and running"
time="2020-10-27T17:07:53.212953961Z" level=error msg="Failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
time="2020-10-27T17:07:53.213216306Z" level=fatal msg="failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
docker logs k3d-cluster3Bplus-serverlb
2020-10-27T17:07:56Z k3d-cluster3Bplus-serverlb confd[7]: INFO Backend set to env
2020-10-27T17:07:56Z k3d-cluster3Bplus-serverlb confd[7]: INFO Starting confd
2020-10-27T17:07:56Z k3d-cluster3Bplus-serverlb confd[7]: INFO Backend source(s) set to
2020-10-27T17:07:56Z k3d-cluster3Bplus-serverlb confd[7]: INFO /etc/nginx/nginx.conf has md5sum 9c9065b8e74f4b01f6eed5a7af1141b6 should be 165cc6471e516ee57a8612156e73f070
2020-10-27T17:07:56Z k3d-cluster3Bplus-serverlb confd[7]: INFO Target config /etc/nginx/nginx.conf out of sync
2020-10-27T17:07:56Z k3d-cluster3Bplus-serverlb confd[7]: INFO Target config /etc/nginx/nginx.conf has been updated
===== Initial nginx configuration =====
error_log stderr notice;
worker_processes auto;
events {
multi_accept on;
use epoll;
worker_connections 1025;
}
stream {
upstream server_nodes_6443 {
server k3d-cluster3Bplus-server-0:6443 max_fails=1 fail_timeout=10s;
}
server {
listen 6443;
proxy_pass server_nodes_6443;
proxy_timeout 600;
proxy_connect_timeout 2s;
}
}=======================================
2020/10/27 17:07:57 [emerg] 13#13: host not found in upstream "k3d-cluster3Bplus-server-0:6443" in /etc/nginx/nginx.conf:12
nginx: [emerg] host not found in upstream "k3d-cluster3Bplus-server-0:6443" in /etc/nginx/nginx.conf:12
time="2020-10-27T17:07:53.212953961Z" level=error msg="Failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
time="2020-10-27T17:07:53.213216306Z" level=fatal msg="failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
??? wtf ?
cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=82663860-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
Only, on one line ?
ls -la /sys/fs/cgroup/
total 0
drwxr-xr-x 12 root root 320 Feb 14 2019 .
drwxr-xr-x 7 root root 0 Jan 1 1970 ..
dr-xr-xr-x 5 root root 0 Feb 14 2019 blkio
lrwxrwxrwx 1 root root 11 Feb 14 2019 cpu -> cpu,cpuacct
lrwxrwxrwx 1 root root 11 Feb 14 2019 cpuacct -> cpu,cpuacct
dr-xr-xr-x 5 root root 0 Feb 14 2019 cpu,cpuacct
dr-xr-xr-x 3 root root 0 Feb 14 2019 cpuset
dr-xr-xr-x 5 root root 0 Feb 14 2019 devices
dr-xr-xr-x 3 root root 0 Feb 14 2019 freezer
lrwxrwxrwx 1 root root 16 Feb 14 2019 net_cls -> net_cls,net_prio
dr-xr-xr-x 3 root root 0 Feb 14 2019 net_cls,net_prio
lrwxrwxrwx 1 root root 16 Feb 14 2019 net_prio -> net_cls,net_prio
dr-xr-xr-x 3 root root 0 Feb 14 2019 perf_event
dr-xr-xr-x 5 root root 0 Feb 14 2019 pids
dr-xr-xr-x 6 root root 0 Feb 14 2019 systemd
dr-xr-xr-x 5 root root 0 Feb 14 2019 unified
lol ok my bad.
INFO[0061] Cluster 'cluster3Bplus' created successfully!
With cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=82663860-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
On a single line !!!
k3d cluster create cluster3Bplus --trace --no-rollback
DEBU[0000] Selected runtime is 'docker.Docker'
DEBU[0000] API-Port Mapping didn't specify hostPort, choosing one randomly...
DEBU[0000] Got free port for API: '42803'
TRAC[0000] PortFilterMap: map[]
TRAC[0000] LabelFilterMap: map[]
DEBU[0000] '--update-default-kubeconfig set: enabling wait-for-server
INFO[0000] Network with name 'k3d-cluster3Bplus' already exists with ID 'cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66'
INFO[0000] Created volume 'k3d-cluster3Bplus-images'
INFO[0001] Creating node 'k3d-cluster3Bplus-server-0'
TRAC[0001] Creating node from spec
&{Name:k3d-cluster3Bplus-server-0 Role:server Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:[k3d-cluster3Bplus-images:/k3d/images] Env:[K3S_TOKEN=IwSYBJYMffuULdOpDOOm] Cmd:[] Args:[] Ports:[] Restart:false Labels:map[k3d.cluster:cluster3Bplus k3d.cluster.imageVolume:k3d-cluster3Bplus-images k3d.cluster.network:cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66 k3d.cluster.network.external:true k3d.cluster.token:IwSYBJYMffuULdOpDOOm k3d.cluster.url:https://k3d-cluster3Bplus-server-0:6443] Network:cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66 ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host:0.0.0.0 HostIP:0.0.0.0 Port:42803}} AgentOpts:{} State:{Running:false Status:}}
TRAC[0001] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-cluster3Bplus-server-0 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=IwSYBJYMffuULdOpDOOm K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san 0.0.0.0] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:cluster3Bplus k3d.cluster.imageVolume:k3d-cluster3Bplus-images k3d.cluster.network:cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66 k3d.cluster.network.external:true k3d.cluster.token:IwSYBJYMffuULdOpDOOm k3d.cluster.url:https://k3d-cluster3Bplus-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:42803] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-cluster3Bplus-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0x20021d8} NetworkingConfig:{EndpointsConfig:map[cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66:0x20240c0]}}
DEBU[0001] Created container k3d-cluster3Bplus-server-0 (ID: 74b434073f2853da6ccd16e1e1cd45c14ce4c6d97ac3028c444048f5622c8bf6)
DEBU[0003] Created node 'k3d-cluster3Bplus-server-0'
DEBU[0003] Waiting for server node 'k3d-cluster3Bplus-server-0' to get ready
DEBU[0049] Finished waiting for log message 'k3s is up and running' from node 'k3d-cluster3Bplus-server-0'
INFO[0049] Creating LoadBalancer 'k3d-cluster3Bplus-serverlb'
TRAC[0049] Creating node from spec
&{Name:k3d-cluster3Bplus-serverlb Role:loadbalancer Image:docker.io/rancher/k3d-proxy:v3.1.5 Volumes:[] Env:[SERVERS=k3d-cluster3Bplus-server-0 PORTS=6443 WORKER_PROCESSES=1] Cmd:[] Args:[] Ports:[0.0.0.0:42803:6443/tcp] Restart:false Labels:map[app:k3d k3d.cluster:cluster3Bplus] Network:cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66 ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host: HostIP: Port:}} AgentOpts:{} State:{Running:false Status:}}
TRAC[0049] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-cluster3Bplus-serverlb Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[6443/tcp:{}] Tty:false OpenStdin:false StdinOnce:false Env:[SERVERS=k3d-cluster3Bplus-server-0 PORTS=6443 WORKER_PROCESSES=1 K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3d-proxy:v3.1.5 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:cluster3Bplus k3d.role:loadbalancer] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[6443/tcp:[{HostIP:0.0.0.0 HostPort:42803}]] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0x3af4c84} NetworkingConfig:{EndpointsConfig:map[cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66:0x3ac2fc0]}}
DEBU[0049] Created container k3d-cluster3Bplus-serverlb (ID: d9dd13eb27e4287bd8e328f132813bcd084c21a0c7b8785808b3057522830159)
DEBU[0051] Starting to wait for loadbalancer node 'k3d-cluster3Bplus-serverlb'
DEBU[0053] Finished waiting for log message 'start worker processes' from node 'k3d-cluster3Bplus-serverlb'
INFO[0053] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
TRAC[0053] Runtime GOOS: linux
DEBU[0053] Adding extra host entry '172.18.0.1 host.k3d.internal'...
DEBU[0053] Executing command '[sh -c echo '172.18.0.1 host.k3d.internal' >> /etc/hosts]' in node 'k3d-cluster3Bplus-server-0'
TRAC[0053] Exec process '[sh -c echo '172.18.0.1 host.k3d.internal' >> /etc/hosts]' still running in node 'k3d-cluster3Bplus-server-0'.. sleeping for 1 second...
DEBU[0054] Exec process in node 'k3d-cluster3Bplus-server-0' exited with '0'
DEBU[0054] Executing command '[sh -c echo '172.18.0.1 host.k3d.internal' >> /etc/hosts]' in node 'k3d-cluster3Bplus-serverlb'
TRAC[0054] Exec process '[sh -c echo '172.18.0.1 host.k3d.internal' >> /etc/hosts]' still running in node 'k3d-cluster3Bplus-serverlb'.. sleeping for 1 second...
DEBU[0055] Exec process in node 'k3d-cluster3Bplus-serverlb' exited with '0'
DEBU[0055] Executing command '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a172.18.0.1 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' in node 'k3d-cluster3Bplus-server-0'
TRAC[0055] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a172.18.0.1 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-cluster3Bplus-server-0'.. sleeping for 1 second...
TRAC[0056] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a172.18.0.1 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-cluster3Bplus-server-0'.. sleeping for 1 second...
TRAC[0057] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a172.18.0.1 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-cluster3Bplus-server-0'.. sleeping for 1 second...
TRAC[0058] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a172.18.0.1 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-cluster3Bplus-server-0'.. sleeping for 1 second...
TRAC[0059] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a172.18.0.1 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-cluster3Bplus-server-0'.. sleeping for 1 second...
TRAC[0060] Exec process '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a172.18.0.1 host.k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' still running in node 'k3d-cluster3Bplus-server-0'.. sleeping for 1 second...
DEBU[0061] Exec process in node 'k3d-cluster3Bplus-server-0' exited with '0'
INFO[0061] Successfully added host record to /etc/hosts in 2/2 nodes and to the CoreDNS ConfigMap
INFO[0061] Cluster 'cluster3Bplus' created successfully!
DEBU[0061] Updating default kubeconfig with a new context for cluster cluster3Bplus
TRAC[0061] Container Details: &{ID:74b434073f2853da6ccd16e1e1cd45c14ce4c6d97ac3028c444048f5622c8bf6 Names:[/k3d-cluster3Bplus-server-0] Image:docker.io/rancher/k3s:v1.18.9-k3s1 ImageID:sha256:3687540964ce031fd5828bed671984cf9d29c8137dd3f5cfdb23b173a198460b Command:/bin/k3s server --tls-san 0.0.0.0 Created:1603819499 Ports:[] SizeRw:0 SizeRootFs:0 Labels:map[app:k3d k3d.cluster:cluster3Bplus k3d.cluster.imageVolume:k3d-cluster3Bplus-images k3d.cluster.network:cf48960b8b83fe125e29c0a61e8ac4e5fcc8e232acbe4dffc621198474ed1a66 k3d.cluster.network.external:true k3d.cluster.token:IwSYBJYMffuULdOpDOOm k3d.cluster.url:https://k3d-cluster3Bplus-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:42803 org.label-schema.build-date:2020-09-17T19:07:38Z org.label-schema.schema-version:1.0 org.label-schema.vcs-ref:630bebf94b9dce6b8cd3d402644ed023b3af8f90 org.label-schema.vcs-url:https://github.com/rancher/k3s.git] State:running Status:Up 58 seconds HostConfig:{NetworkMode:default} NetworkSettings:0x3e985f0 Mounts:[{Type:volume Name:k3d-cluster3Bplus-images Source:/var/lib/docker/volumes/k3d-cluster3Bplus-images/_data Destination:/k3d/images Driver:local Mode:z RW:true Propagation:} {Type:volume Name:71237544523fdd711be92c5b07128b8ee586911fef58045a2f2d108c55495a11 Source: Destination:/var/lib/cni Driver:local Mode: RW:true Propagation:} {Type:volume Name:5c2b9d8365fcaba35154c912127c5698307e8f3be7a5d74308924567fd8d9d51 Source: Destination:/var/lib/kubelet Driver:local Mode: RW:true Propagation:} {Type:volume Name:23cfc87288d6a996d75fa3b962f0e3ced86a8478403d8bd8784a63a5325fae74 Source: Destination:/var/lib/rancher/k3s Driver:local Mode: RW:true Propagation:} {Type:volume Name:1d7022769cfbb3bc0afcc8c0e11dcbbbd4e256e2040e021d18ed9929f8548586 Source: Destination:/var/log Driver:local Mode: RW:true Propagation:}]}
TRAC[0062] Modified Kubeconfig: &{Kind: APIVersion: Preferences:{Colors:false Extensions:map[]} Clusters:map[k3d-cluster3Bplus:0x4aa6330] AuthInfos:map[admin@k3d-cluster3Bplus:0x4a92d80] Contexts:map[k3d-cluster3Bplus:0x4aa6360] CurrentContext:k3d-cluster3Bplus Extensions:map[]}
DEBU[0062] Output path '/home/myuser/.kube/config' doesn't exist, trying to create it...
TRAC[0062] Merging new Kubeconfig:
&{Kind: APIVersion: Preferences:{Colors:false Extensions:map[]} Clusters:map[k3d-cluster3Bplus:0x4aa6330] AuthInfos:map[admin@k3d-cluster3Bplus:0x4a92d80] Contexts:map[k3d-cluster3Bplus:0x4aa6360] CurrentContext:k3d-cluster3Bplus Extensions:map[]}
>>> into existing Kubeconfig:
&{Kind: APIVersion: Preferences:{Colors:false Extensions:map[]} Clusters:map[] AuthInfos:map[] Contexts:map[] CurrentContext: Extensions:map[]}
DEBU[0062] Setting new current-context 'k3d-cluster3Bplus'
TRAC[0062] Merged Kubeconfig:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJWakNCL3FBREFnRUNBZ0VBTUFvR0NDcUdTTTQ5QkFNQ01DTXhJVEFmQmdOVkJBTU1HR3N6Y3kxelpYSjIKWlhJdFkyRkFNVFl3TXpneE9UVXdOVEFlRncweU1ERXdNamN4TnpJMU1EVmFGdzB6TURFd01qVXhOekkxTURWYQpNQ014SVRBZkJnTlZCQU1NR0dzemN5MXpaWEoyWlhJdFkyRkFNVFl3TXpneE9UVXdOVEJaTUJNR0J5cUdTTTQ5CkFnRUdDQ3FHU000OUF3RUhBMElBQkIyOWVndFlHVklWZkVZM1BkVVBRSkhyOWdCZ25vUmx3TE80UWRNbElzT0wKaVpPOXh4b2ROeGF1SVBkRnI0TDVFYmdJby95WkRYSCtocWUzMjZta0o0T2pJekFoTUE0R0ExVWREd0VCL3dRRQpBd0lDcERBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUFvR0NDcUdTTTQ5QkFNQ0EwY0FNRVFDSUFjK0RFZ2tmeXBUCnVab0VlUU9aeUsvWndWUm11c2EzT2hoT2tSVVBRNVFiQWlBZVBpWlhISnptZ2JDY3RRRmZCZ1ZQZG5nbTNjOVQKREI1UVZPWG9Ba1E2Z1E9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
server: https://0.0.0.0:42803
name: k3d-cluster3Bplus
contexts:
- context:
cluster: k3d-cluster3Bplus
user: admin@k3d-cluster3Bplus
name: k3d-cluster3Bplus
current-context: k3d-cluster3Bplus
kind: Config
preferences: {}
users:
- name: admin@k3d-cluster3Bplus
user:
password: 13adf1e21e4c92a7e4eb90f5ea18fa1b
username: admin
DEBU[0062] Wrote kubeconfig to '/home/myuser/.kube/config'
INFO[0062] You can now use it like this:
kubectl cluster-info
I deleted my cluster, re-created a new one and it works.
Conclusion:
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory was missing in the /boot/cmdline.txt file.
And the file must have only one line
@iwilltry42 Thx for your help.
GannonTdW
on 27 Oct 2020
👍1
I'm still seeing a similar issue on macOS 10.15.7 when running the following command:
Input:
k3d cluster create testcluster --agents 3 -p "8082:30080@agent[0]" --servers 3 --wait
Output:
➜ k3d cluster create testcluster --agents 3 -p "8082:30080@agent[0]" --servers 3 --wait
INFO[0000] Created network 'k3d-testcluster'
INFO[0000] Created volume 'k3d-testcluster-images'
INFO[0000] Creating initializing server node
INFO[0000] Creating node 'k3d-testcluster-server-0'
INFO[0008] Creating node 'k3d-testcluster-server-1'
ERRO[0323] Failed waiting for log message 'k3s is up and running' from node 'k3d-testcluster-server-1'
ERRO[0323] Server node 'k3d-testcluster-server-1' failed to get ready: Node 'k3d-testcluster-server-1' (container '662786e30be5c843a795c1c91e4dc58d75fba292af7a12b0c91b1cbd2d5eae5d') not running
ERRO[0323] Failed to create cluster >>> Rolling Back
INFO[0323] Deleting cluster 'testcluster'
INFO[0323] Deleted k3d-testcluster-server-0
INFO[0323] Deleted k3d-testcluster-server-1
ERRO[0323] Failed to delete container ''
WARN[0323] Failed to delete node '': Try to delete it manually
ERRO[0323] Failed to delete container ''
WARN[0323] Failed to delete node '': Try to delete it manually
ERRO[0323] Failed to delete container ''
WARN[0323] Failed to delete node '': Try to delete it manually
ERRO[0323] Failed to delete container ''
WARN[0323] Failed to delete node '': Try to delete it manually
INFO[0323] Deleting cluster network '061f71b6c1e3d8b9af2555dce85e0ad7fa0ad4975f40c701a2c53d28949149bc'
ERRO[0323] Failed to delete 4 nodes: Try to delete them manually
FATA[0323] Cluster creation FAILED, also FAILED to rollback changes!
k3d version:
➜ k3d version
k3d version v3.1.5
k3s version v1.18.9-k3s1 (default)
conradwt
on 28 Oct 2020
Hi @conradwt :wave:
Your case looks a little different though. Can you please re-execute that command and add the --trace --no-rollback flags to it?
Afterwards, please paste the output of that command and the output of docker logs k3d-testcluster-server-0 and of docker logs k3d-testcluster-server-1 here, so we can have a look at what's going on there :+1:
iwilltry42
on 28 Oct 2020
@iwilltry42 Here's the requested output:
Cluster Creation
~ at ☸️ docker-desktop
➜ k3d cluster create testcluster --agents 3 -p "8082:30080@agent[0]" --servers 3 --wait --trace --no-rollback
DEBU[0000] Selected runtime is 'docker.Docker'
DEBU[0000] API-Port Mapping didn't specify hostPort, choosing one randomly...
DEBU[0000] Got free port for API: '56646'
TRAC[0000] PortFilterMap: map[8082:30080:[agent[0]]]
TRAC[0000] LabelFilterMap: map[]
DEBU[0000] '--update-default-kubeconfig set: enabling wait-for-server
INFO[0000] Created network 'k3d-testcluster'
INFO[0000] Created volume 'k3d-testcluster-images'
INFO[0000] Creating initializing server node
INFO[0000] Creating node 'k3d-testcluster-server-0'
TRAC[0000] Creating node from spec
&{Name:k3d-testcluster-server-0 Role:server Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:[k3d-testcluster-images:/k3d/images] Env:[K3S_TOKEN=HarDTsHgVommVgBITdXT] Cmd:[] Args:[--cluster-init] Ports:[] Restart:false Labels:map[k3d.cluster:testcluster k3d.cluster.imageVolume:k3d-testcluster-images k3d.cluster.network:b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0 k3d.cluster.network.external:false k3d.cluster.token:HarDTsHgVommVgBITdXT k3d.cluster.url:https://k3d-testcluster-server-0:6443] Network:b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0 ExtraHosts:[] ServerOpts:{IsInit:true ExposeAPI:{Host:0.0.0.0 HostIP:0.0.0.0 Port:56646}} AgentOpts:{} State:{Running:false Status:}}
TRAC[0000] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-testcluster-server-0 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=HarDTsHgVommVgBITdXT K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --cluster-init --tls-san 0.0.0.0] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:testcluster k3d.cluster.imageVolume:k3d-testcluster-images k3d.cluster.network:b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0 k3d.cluster.network.external:false k3d.cluster.token:HarDTsHgVommVgBITdXT k3d.cluster.url:https://k3d-testcluster-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:56646] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-testcluster-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc000036dc8} NetworkingConfig:{EndpointsConfig:map[b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0:0xc0003b8000]}}
INFO[0001] Pulling image 'docker.io/rancher/k3s:v1.18.9-k3s1'
DEBU[0005] Created container k3d-testcluster-server-0 (ID: 949dd6ba931a9ae30c62147459b0b2add1106ccd8900343e70e4f9738f4a9655)
DEBU[0005] Created node 'k3d-testcluster-server-0'
DEBU[0005] Waiting for initializing server node...
DEBU[0006] Waiting for initializing server node...
DEBU[0008] Waiting for initializing server node...
DEBU[0009] Waiting for initializing server node...
DEBU[0010] Waiting for initializing server node...
DEBU[0011] Waiting for initializing server node...
DEBU[0012] Waiting for initializing server node...
DEBU[0013] Waiting for initializing server node...
DEBU[0013] Initializing server node is up... continuing
INFO[0014] Creating node 'k3d-testcluster-server-1'
TRAC[0014] Creating node from spec
&{Name:k3d-testcluster-server-1 Role:server Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:[k3d-testcluster-images:/k3d/images] Env:[K3S_TOKEN=HarDTsHgVommVgBITdXT K3S_URL=https://k3d-testcluster-server-0:6443] Cmd:[] Args:[] Ports:[] Restart:false Labels:map[k3d.cluster:testcluster k3d.cluster.imageVolume:k3d-testcluster-images k3d.cluster.network:b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0 k3d.cluster.network.external:false k3d.cluster.token:HarDTsHgVommVgBITdXT k3d.cluster.url:https://k3d-testcluster-server-0:6443] Network:b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0 ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host:0.0.0.0 HostIP:0.0.0.0 Port:56646}} AgentOpts:{} State:{Running:false Status:}}
TRAC[0014] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-testcluster-server-1 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=HarDTsHgVommVgBITdXT K3S_URL=https://k3d-testcluster-server-0:6443 K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san 0.0.0.0] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:testcluster k3d.cluster.imageVolume:k3d-testcluster-images k3d.cluster.network:b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0 k3d.cluster.network.external:false k3d.cluster.token:HarDTsHgVommVgBITdXT k3d.cluster.url:https://k3d-testcluster-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:56646] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-testcluster-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc000407290} NetworkingConfig:{EndpointsConfig:map[b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0:0xc00014ca80]}}
DEBU[0014] Created container k3d-testcluster-server-1 (ID: 5f3e30c5983a267f02338ec09b676f4e2f140d7c98915fa590d68419fd3941d0)
DEBU[0014] Created node 'k3d-testcluster-server-1'
DEBU[0014] Waiting for server node 'k3d-testcluster-server-1' to get ready
DEBU[0036] Finished waiting for log message 'k3s is up and running' from node 'k3d-testcluster-server-1'
INFO[0037] Creating node 'k3d-testcluster-server-2'
TRAC[0037] Creating node from spec
&{Name:k3d-testcluster-server-2 Role:server Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:[k3d-testcluster-images:/k3d/images] Env:[K3S_TOKEN=HarDTsHgVommVgBITdXT K3S_URL=https://k3d-testcluster-server-0:6443] Cmd:[] Args:[] Ports:[] Restart:false Labels:map[k3d.cluster:testcluster k3d.cluster.imageVolume:k3d-testcluster-images k3d.cluster.network:b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0 k3d.cluster.network.external:false k3d.cluster.token:HarDTsHgVommVgBITdXT k3d.cluster.url:https://k3d-testcluster-server-0:6443] Network:b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0 ExtraHosts:[] ServerOpts:{IsInit:false ExposeAPI:{Host:0.0.0.0 HostIP:0.0.0.0 Port:56646}} AgentOpts:{} State:{Running:false Status:}}
TRAC[0037] Creating docker container with translated config
&{ContainerConfig:{Hostname:k3d-testcluster-server-2 Domainname: User: AttachStdin:false AttachStdout:false AttachStderr:false ExposedPorts:map[] Tty:false OpenStdin:false StdinOnce:false Env:[K3S_TOKEN=HarDTsHgVommVgBITdXT K3S_URL=https://k3d-testcluster-server-0:6443 K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml] Cmd:[server --tls-san 0.0.0.0] Healthcheck:<nil> ArgsEscaped:false Image:docker.io/rancher/k3s:v1.18.9-k3s1 Volumes:map[] WorkingDir: Entrypoint:[] NetworkDisabled:false MacAddress: OnBuild:[] Labels:map[app:k3d k3d.cluster:testcluster k3d.cluster.imageVolume:k3d-testcluster-images k3d.cluster.network:b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0 k3d.cluster.network.external:false k3d.cluster.token:HarDTsHgVommVgBITdXT k3d.cluster.url:https://k3d-testcluster-server-0:6443 k3d.role:server k3d.server.api.host:0.0.0.0 k3d.server.api.hostIP:0.0.0.0 k3d.server.api.port:56646] StopSignal: StopTimeout:<nil> Shell:[]} HostConfig:{Binds:[k3d-testcluster-images:/k3d/images] ContainerIDFile: LogConfig:{Type: Config:map[]} NetworkMode: PortBindings:map[] RestartPolicy:{Name: MaximumRetryCount:0} AutoRemove:false VolumeDriver: VolumesFrom:[] CapAdd:[] CapDrop:[] Capabilities:[] CgroupnsMode: DNS:[] DNSOptions:[] DNSSearch:[] ExtraHosts:[] GroupAdd:[] IpcMode: Cgroup: Links:[] OomScoreAdj:0 PidMode: Privileged:true PublishAllPorts:false ReadonlyRootfs:false SecurityOpt:[] StorageOpt:map[] Tmpfs:map[/run: /var/run:] UTSMode: UsernsMode: ShmSize:0 Sysctls:map[] Runtime: ConsoleSize:[0 0] Isolation: Resources:{CPUShares:0 Memory:0 NanoCPUs:0 CgroupParent: BlkioWeight:0 BlkioWeightDevice:[] BlkioDeviceReadBps:[] BlkioDeviceWriteBps:[] BlkioDeviceReadIOps:[] BlkioDeviceWriteIOps:[] CPUPeriod:0 CPUQuota:0 CPURealtimePeriod:0 CPURealtimeRuntime:0 CpusetCpus: CpusetMems: Devices:[] DeviceCgroupRules:[] DeviceRequests:[] KernelMemory:0 KernelMemoryTCP:0 MemoryReservation:0 MemorySwap:0 MemorySwappiness:<nil> OomKillDisable:<nil> PidsLimit:<nil> Ulimits:[] CPUCount:0 CPUPercent:0 IOMaximumIOps:0 IOMaximumBandwidth:0} Mounts:[] MaskedPaths:[] ReadonlyPaths:[] Init:0xc0005819c0} NetworkingConfig:{EndpointsConfig:map[b21e81b1fa86583f248b71444543bff9d26f7445e4c976162cda319712501ca0:0xc0018ef140]}}
DEBU[0037] Created container k3d-testcluster-server-2 (ID: 91ffa4ba15415ce4ed646da2438a8337a3550b08de6a483660017405a9709d98)
DEBU[0037] Created node 'k3d-testcluster-server-2'
DEBU[0037] Waiting for server node 'k3d-testcluster-server-2' to get ready
ERRO[0049] Failed waiting for log message 'k3s is up and running' from node 'k3d-testcluster-server-2'
ERRO[0049] Server node 'k3d-testcluster-server-2' failed to get ready: Node 'k3d-testcluster-server-2' (container '91ffa4ba15415ce4ed646da2438a8337a3550b08de6a483660017405a9709d98') not running
FATA[0049] Cluster creation FAILED, rollback deactivated.
The logs for k3d-testcluster-server-0 and k3d-testcluster-server-1 can be found here.
conradwt
on 28 Oct 2020
Thanks @conradwt , unfortunately, this time it's k3d-testcluster-server-2 that is not running :D
So I'll need the logs from that container :grimacing:
iwilltry42
on 28 Oct 2020
@iwilltry42 Here's the logs for the k3d-testcluster-server-2:
➜ docker logs k3d-testcluster-server-2
time="2020-10-28T10:08:13.336455556Z" level=info msg="Starting k3s v1.18.9+k3s1 (630bebf9)"
time="2020-10-28T10:08:13.621816230Z" level=info msg="Active TLS secret (ver=) (count 8): map[listener.cattle.io/cn-0.0.0.0:0.0.0.0 listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-172.19.0.4:172.19.0.4 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/hash:c634e195192395409b2c79807475512c7b9be747b0f77eba4205888cd00f9640]"
time="2020-10-28T10:08:13.698000905Z" level=info msg="Joining dqlite cluster as address=172.19.0.4:6443, id=998596"
time="2020-10-28T10:08:20.624904963Z" level=fatal msg="starting kubernetes: preparing server: post join: no connection to remote server available (1)"
@conradwt now this is a log line I haven't seen so far from k3s :thinking:
You can increase the log level by passing --k3s-server-arg "-v=3" to the cluster create command to get some more information.
I actually have no clue right now, what could be causing that error you see there, as it seems to be network related :thinking:
I also cannot reproduce it locally, so I assume it's some runtime environment issue..
iwilltry42
on 28 Oct 2020
@iwilltry42 After resetting to default settings and restarting Docker For Mac, things appear to be working as expected with k3d. Thanks again for all your assistance with this issue.
conradwt
on 29 Oct 2020
❤1
Was this page helpful?
0 / 5 - 0 ratings
Related issues
rjshrjndrn
·
5Comments
kafai97
·
3Comments
stephenc
·
4Comments
stephenpope
·
4Comments
rpocase
·
3Comments