K3d: [BUG]

Created on 5 Jul 2020 · 9Comments · Source: rancher/k3d

What did you do?

How was the cluster created?
- k3d create cluster --k3s-server-arg "k3s server --tls-san \"192.168.1.28\"" topgunCluster --masters 5 --workers 9
What did you do before that command?

I created and deleted many clusters to test command line options

What did you do afterwards?
- k3d commands?

#  clean everything up between two clusters (when you tear down one, to create another) :
k3d delete cluster topgunCluster
sudo rm /etc/docker/*.json
docker system prune -f --all && docker system prune -f --volumes
sudo systemctl daemon-reload
sudo systemctl restart docker
# and then the exact same command successfuly gave me a cluster
k3d create cluster --k3s-server-arg "k3s server --tls-san \"192.168.1.28\"" topgunCluster --masters 5 --workers 9

- docker commands? none
- OS operations (e.g. shutdown/reboot)? none

What did you expect to happen?

That the cluster creation would always succeed would always succeed, regardless of how many times I run them :

k3d delete cluster topgunCluster 
k3d create cluster --k3s-server-arg "k3s server --tls-san \"192.168.1.28\"" topgunCluster --masters 5 --workers 9

Screenshots or terminal output

Indeed, you can see below an example of a cluster creation that fails without cleaning, and succeeds after cleaning up :

jbl@pc-alienware-jbl:~$ k3d create cluster --k3s-server-arg "k3s server --tls-san \"192.168.1.28\"" topgunCluster --masters 5 --workers 9
INFO[0000] Created network 'k3d-topgunCluster'
INFO[0000] Created volume 'k3d-topgunCluster-images'
INFO[0000] Creating initializing master node
INFO[0000] Creating node 'k3d-topgunCluster-master-0'
INFO[0013] Creating node 'k3d-topgunCluster-master-1'
INFO[0014] Creating node 'k3d-topgunCluster-master-2'
INFO[0016] Creating node 'k3d-topgunCluster-master-3'
INFO[0018] Creating node 'k3d-topgunCluster-master-4'
INFO[0019] Creating node 'k3d-topgunCluster-worker-0'
INFO[0021] Creating node 'k3d-topgunCluster-worker-1'
INFO[0022] Creating node 'k3d-topgunCluster-worker-2'
INFO[0023] Creating node 'k3d-topgunCluster-worker-3'
INFO[0025] Creating node 'k3d-topgunCluster-worker-4'
INFO[0026] Creating node 'k3d-topgunCluster-worker-5'
INFO[0027] Creating node 'k3d-topgunCluster-worker-6'
INFO[0028] Creating node 'k3d-topgunCluster-worker-7'
INFO[0029] Creating node 'k3d-topgunCluster-worker-8'
INFO[0030] Creating LoadBalancer 'k3d-topgunCluster-masterlb'
ERRO[0031] Failed waiting for log message 'start worker processes' from node 'k3d-topgunCluster-masterlb'
ERRO[0031] Failed to bring up all master nodes (and loadbalancer) in time. Check the logs:
ERRO[0031] >>> Node 'k3d-topgunCluster-masterlb' (container '3465df51d5ec289044eb794f76d605d1e7fdd2d51e9925b0fdb5ab66d6069166') not running
ERRO[0031] Failed to bring up cluster
ERRO[0031] Failed to create cluster >>> Rolling Back
INFO[0031] Deleting cluster 'topgunCluster'
INFO[0032] Deleted k3d-topgunCluster-master-0
INFO[0032] Deleted k3d-topgunCluster-master-1
INFO[0033] Deleted k3d-topgunCluster-master-2
INFO[0034] Deleted k3d-topgunCluster-master-3
INFO[0034] Deleted k3d-topgunCluster-master-4
INFO[0035] Deleted k3d-topgunCluster-worker-0
INFO[0036] Deleted k3d-topgunCluster-worker-1
INFO[0036] Deleted k3d-topgunCluster-worker-2
INFO[0037] Deleted k3d-topgunCluster-worker-3
INFO[0038] Deleted k3d-topgunCluster-worker-4
INFO[0038] Deleted k3d-topgunCluster-worker-5
INFO[0039] Deleted k3d-topgunCluster-worker-6
INFO[0040] Deleted k3d-topgunCluster-worker-7
INFO[0040] Deleted k3d-topgunCluster-worker-8
INFO[0040] Deleted k3d-topgunCluster-masterlb
INFO[0040] Deleting cluster network '7935e282ead88eaa3857b79ec4cc22cc2303a7066deac4d7c6dfca7dd0d15ec8'
FATA[0040] Cluster creation FAILED, all changes have been rolled back!
jbl@pc-alienware-jbl:~$ k3d delete cluster topgunCluster
FATA[0000] No nodes found for cluster 'topgunCluster'
jbl@pc-alienware-jbl:~$ sudo rm /etc/docker/*.json
jbl@pc-alienware-jbl:~$ docker system prune -f --all && docker system prune -f --volumes
Deleted Images:
untagged: rancher/k3s:v1.18.4-k3s1
untagged: rancher/k3s@sha256:2555ed1896512b320c175859a95a5f1d6a2c64b501a84bc45d853a9bd0be0dfd
deleted: sha256:8600923bc2c7dda1d67732d4c9498f79204889d4a4619212e5301e8ebd88683c
deleted: sha256:9112bc341c35bc180f6526457537650bc9c281a60b4960d515422642e137cd6e
deleted: sha256:4a580f02a4fae85d3c01c88ac827b935db862e2842a4722a0073962f38b7704e
deleted: sha256:ec1433ebbebd8aefcdc2d9f8bec9668defb5d1e0984a4035c510f99fe4ba0562
untagged: rancher/k3d-proxy:v3.0.0-rc.6
untagged: rancher/k3d-proxy@sha256:a9d150f33da45cfab590d8094b034d30510f142332fbb007c1e5030aaddf304f
deleted: sha256:4b44f08c525f5c026222c8d78635672e3a6150179562f9f9b572a66d9a104600
deleted: sha256:818da9004fb9448cb438215e04e9c9c5853561c54d17cdc187b0d493c369c314
deleted: sha256:335fc598937327294d4748e2bc3eec1d6589f7c485a0690d54e0397df64800eb
deleted: sha256:ec88223b6e690f3a2285f3c13dcff4e18ab75411fc2b63f363d3ec676371487a
deleted: sha256:7c5c8fccda028cec476ee644a77046ef66b7d9eaa3c57ddc1ac8f64a9aafe07c
deleted: sha256:c08fbfc0bf8619da11d55fcaad512b2824bd3add3078d3b8ca0e68e9c5e41ab5
deleted: sha256:f1b5933fe4b5f49bbe8258745cf396afe07e625bdab3168e364daf7c956b6b81

Total reclaimed space: 194MB
Deleted Volumes:
k3d-topgunCluster-images

Total reclaimed space: 0B
jbl@pc-alienware-jbl:~$ sudo systemctl daemon-reload
jbl@pc-alienware-jbl:~$ sudo systemctl restart docker
jbl@pc-alienware-jbl:~$ k3d delete cluster topgunCluster
FATA[0000] No nodes found for cluster 'topgunCluster'
jbl@pc-alienware-jbl:~$ k3d create cluster --k3s-server-arg "k3s server --tls-san \"192.168.1.28\"" topgunCluster --masters 5 --workers 9
INFO[0000] Created network 'k3d-topgunCluster'
INFO[0000] Created volume 'k3d-topgunCluster-images'
INFO[0000] Creating initializing master node
INFO[0000] Creating node 'k3d-topgunCluster-master-0'
INFO[0001] Pulling image 'docker.io/rancher/k3s:v1.18.4-k3s1'
INFO[0018] Creating node 'k3d-topgunCluster-master-1'
INFO[0019] Creating node 'k3d-topgunCluster-master-2'
INFO[0021] Creating node 'k3d-topgunCluster-master-3'
INFO[0023] Creating node 'k3d-topgunCluster-master-4'
INFO[0025] Creating node 'k3d-topgunCluster-worker-0'
INFO[0026] Creating node 'k3d-topgunCluster-worker-1'
INFO[0027] Creating node 'k3d-topgunCluster-worker-2'
INFO[0029] Creating node 'k3d-topgunCluster-worker-3'
INFO[0031] Creating node 'k3d-topgunCluster-worker-4'
INFO[0032] Creating node 'k3d-topgunCluster-worker-5'
INFO[0034] Creating node 'k3d-topgunCluster-worker-6'
INFO[0036] Creating node 'k3d-topgunCluster-worker-7'
INFO[0038] Creating node 'k3d-topgunCluster-worker-8'
INFO[0039] Creating LoadBalancer 'k3d-topgunCluster-masterlb'
INFO[0040] Pulling image 'docker.io/rancher/k3d-proxy:v3.0.0-rc.6'
INFO[0046] Cluster 'topgunCluster' created successfully!
INFO[0046] You can now use it like this:
export KUBECONFIG=$(k3d get kubeconfig topgunCluster)
kubectl cluster-info
jbl@pc-alienware-jbl:~$ export KUBECONFIG=$(k3d get kubeconfig topgunCluster)
jbl@pc-alienware-jbl:~$ kubectl get all,nodes
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   52s

NAME                              STATUS   ROLES    AGE   VERSION
node/k3d-topguncluster-master-0   Ready    master   49s   v1.18.4+k3s1
node/k3d-topguncluster-master-2   Ready    master   35s   v1.18.4+k3s1
node/k3d-topguncluster-master-1   Ready    master   32s   v1.18.4+k3s1
node/k3d-topguncluster-worker-2   Ready    <none>   34s   v1.18.4+k3s1
node/k3d-topguncluster-worker-1   Ready    <none>   34s   v1.18.4+k3s1
node/k3d-topguncluster-master-4   Ready    master   31s   v1.18.4+k3s1
node/k3d-topguncluster-worker-3   Ready    <none>   32s   v1.18.4+k3s1
node/k3d-topguncluster-worker-4   Ready    <none>   31s   v1.18.4+k3s1
node/k3d-topguncluster-worker-5   Ready    <none>   28s   v1.18.4+k3s1
node/k3d-topguncluster-worker-6   Ready    <none>   27s   v1.18.4+k3s1
node/k3d-topguncluster-master-3   Ready    master   26s   v1.18.4+k3s1
node/k3d-topguncluster-worker-8   Ready    <none>   24s   v1.18.4+k3s1
node/k3d-topguncluster-worker-0   Ready    <none>   37s   v1.18.4+k3s1
node/k3d-topguncluster-worker-7   Ready    <none>   23s   v1.18.4+k3s1
jbl@pc-alienware-jbl:~$

Which OS & Architecture?

Linux, Windows, MacOS / amd64, x86, ...? see uname -a above :+1: A Debian Stretch
amd64 cpu arch

jbl@pc-alienware-jbl:~$ uname -a
Linux pc-alienware-jbl 4.9.0-7-amd64 #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x86_64 GNU/Linux

Which version of k3d?

output of k3d --version

Which version of docker?

output of docker version

jbl@pc-alienware-jbl:~$ docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:26:02 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:24:36 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
jbl@pc-alienware-jbl:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4+k3s1", GitCommit:"97b7a0e9df2883f08028fb7171c1e62fc1899a0c", GitTreeState:"clean", BuildDate:"2020-06-18T01:30:45Z", GoVersion:"go1.13.11", Compiler:"gc", Platform:"linux/amd64"}
jbl@pc-alienware-jbl:~$ uname -a
Linux pc-alienware-jbl 4.9.0-7-amd64 #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x86_64 GNU/Linux
jbl@pc-alienware-jbl:~$ k3d version
k3d version v3.0.0-rc.6
k3s version v1.18.4-k3s1 (default)
jbl@pc-alienware-jbl:~$ k3d --version
k3d version v3.0.0-rc.6
k3s version v1.18.4-k3s1 (default)

bug

Source

Jean-Baptiste-Lasselle

All 9 comments

ps: I like the new rollback feature.

Jean-Baptiste-Lasselle on 5 Jul 2020

Hi @Jean-Baptiste-Lasselle , thanks for opening this issue and sharing all the details.
I'll read through it soon and will try to reproduce it.
One thing up front:

--k3s-server-arg "k3s server --tls-san \"192.168.1.28\""

should/could just be
--k3s-server-arg '--tls-san="192.168.1.28"' (without the additional k3s server part) :+1:

iwilltry42 on 8 Jul 2020

❤1

While creating the cluster, can you please try to get the logs of k3d-topgunCluster-masterlb via docker logs -f k3d-topgunCluster-masterlb? Also, please check docker ps -a to see if all node containers are up and running (especially all master nodes).

iwilltry42 on 8 Jul 2020

👍1

should/could just be
--k3s-server-arg '--tls-san="192.168.1.28"' (without the additional k3s server part) +1

yeah, totally, n what I experienced is that the "should" does not work, while I found the hack with --k3s-server-arg "k3s server --tls-san \"192.168.1.28\""

Jean-Baptiste-Lasselle on 8 Jul 2020

While creating the cluster, can you please try to get the logs of k3d-topgunCluster-masterlb via docker logs -f k3d-topgunCluster-masterlb? Also, please check docker ps -a to see if all node containers are up and running (especially all master nodes).

will do that asap, im on the rin at work, but asap and will bring back feedback

Jean-Baptiste-Lasselle on 8 Jul 2020

HI @iwilltry42 , HAve today a bit of time to go further :

I tried and reproduce the error I describred first : running the same k3d create cluster command, I still had an error, but not the same : see below stdout.
we hd something else, about the --tls-san : there I found something new. Here is what is new :

I tried and apply what you gave me, that is running k3d create cluster --k3s-server-arg '--tls-san 192.168.1.35' topgunCluster --masters 3 --workers 9. But there I had an error, which is tricky to diagnose, because the rollout very quickly wipes out the container which has a problem (a master node, the first created). Well I managed to still get the logs, and when I use blabla "--tsl-san '192.168.1.35'" , the k3s server does not receive --tls-san, but it receives -tls-san instead (note the missing first hyphen) :
the logs from the failing initial master node zero :

jbl@pegasusio:~$ docker logs -f k3d-topgunCluster-master-0
Incorrect Usage: flag provided but not defined: -tls-san 192.168.1.35

NAME:
   k3s server - Run management server

USAGE:
   k3s server [OPTIONS]

OPTIONS:
   -v value                                   (logging) Number for the log level verbosity (default: 0)
   --vmodule value                            (logging) Comma-separated list of pattern=N settings for file-filtered logging
   --log value, -l value                      (logging) Log to file
   --alsologtostderr                          (logging) Log to standard error as well as file (if set)
   --bind-address value                       (listener) k3s bind address (default: 0.0.0.0)
   --https-listen-port value                  (listener) HTTPS listen port (default: 6443)
   --advertise-address value                  (listener) IP address that apiserver uses to advertise to members of the cluster (default: node-external-ip/node-ip)
   --advertise-port value                     (listener) Port that apiserver uses to advertise to members of the cluster (default: listen-port) (default: 0)
   --tls-san value                            (listener) Add additional hostname or IP as a Subject Alternative Name in the TLS cert (default: "0.0.0.0")
   --data-dir value, -d value                 (data) Folder to hold state default /var/lib/rancher/k3s or ${HOME}/.rancher/k3s if not root
   --cluster-cidr value                       (networking) Network CIDR to use for pod IPs (default: "10.42.0.0/16")
   --service-cidr value                       (networking) Network CIDR to use for services IPs (default: "10.43.0.0/16")
   --cluster-dns value                        (networking) Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10)
   --cluster-domain value                     (networking) Cluster Domain (default: "cluster.local")
   --flannel-backend value                    (networking) One of 'none', 'vxlan', 'ipsec', 'host-gw', or 'wireguard' (default: "vxlan")
   --token value, -t value                    (cluster) Shared secret used to join a server or agent to a cluster [$K3S_TOKEN]
   --token-file value                         (cluster) File containing the cluster-secret/token [$K3S_TOKEN_FILE]
   --write-kubeconfig value, -o value         (client) Write kubeconfig for admin client to this file [$K3S_KUBECONFIG_OUTPUT]
   --write-kubeconfig-mode value              (client) Write kubeconfig with this mode [$K3S_KUBECONFIG_MODE]
   --kube-apiserver-arg value                 (flags) Customized flag for kube-apiserver process
   --kube-scheduler-arg value                 (flags) Customized flag for kube-scheduler process
   --kube-controller-manager-arg value        (flags) Customized flag for kube-controller-manager process
   --kube-cloud-controller-manager-arg value  (flags) Customized flag for kube-cloud-controller-manager process
   --datastore-endpoint value                 (db) Specify etcd, Mysql, Postgres, or Sqlite (default) data source name [$K3S_DATASTORE_ENDPOINT]
   --datastore-cafile value                   (db) TLS Certificate Authority file used to secure datastore backend communication [$K3S_DATASTORE_CAFILE]
   --datastore-certfile value                 (db) TLS certification file used to secure datastore backend communication [$K3S_DATASTORE_CERTFILE]
   --datastore-keyfile value                  (db) TLS key file used to secure datastore backend communication [$K3S_DATASTORE_KEYFILE]
   --default-local-storage-path value         (storage) Default local storage path for local provisioner storage class
   --disable value                            (components) Do not deploy packaged components and delete any deployed components (valid items: coredns, servicelb, traefik, local-storage, metrics-server)
   --disable-scheduler                        (components) Disable Kubernetes default scheduler
   --disable-cloud-controller                 (components) Disable k3s default cloud controller manager
   --disable-network-policy                   (components) Disable k3s default network policy controller
   --node-name value                          (agent/node) Node name [$K3S_NODE_NAME]
   --with-node-id                             (agent/node) Append id to node name
   --node-label value                         (agent/node) Registering and starting kubelet with set of labels
   --node-taint value                         (agent/node) Registering kubelet with set of taints
   --docker                                   (agent/runtime) Use docker instead of containerd
   --container-runtime-endpoint value         (agent/runtime) Disable embedded containerd and use alternative CRI implementation
   --pause-image value                        (agent/runtime) Customized pause image for containerd or docker sandbox (default: "docker.io/rancher/pause:3.1")
   --private-registry value                   (agent/runtime) Private registry configuration file (default: "/etc/rancher/k3s/registries.yaml")
   --node-ip value, -i value                  (agent/networking) IP address to advertise for node
   --node-external-ip value                   (agent/networking) External IP address to advertise for node
   --resolv-conf value                        (agent/networking) Kubelet resolv.conf file [$K3S_RESOLV_CONF]
   --flannel-iface value                      (agent/networking) Override default flannel interface
   --flannel-conf value                       (agent/networking) Override default flannel config file
   --kubelet-arg value                        (agent/flags) Customized flag for kubelet process
   --kube-proxy-arg value                     (agent/flags) Customized flag for kube-proxy process
   --rootless                                 (experimental) Run rootless
   --agent-token value                        (experimental/cluster) Shared secret used to join agents to the cluster, but not servers [$K3S_AGENT_TOKEN]
   --agent-token-file value                   (experimental/cluster) File containing the agent secret [$K3S_AGENT_TOKEN_FILE]
   --server value, -s value                   (experimental/cluster) Server to connect to, used to join a cluster [$K3S_URL]
   --cluster-init                             (experimental/cluster) Initialize new cluster master [$K3S_CLUSTER_INIT]
   --cluster-reset                            (experimental/cluster) Forget all peers and become a single cluster new cluster master [$K3S_CLUSTER_RESET]
   --secrets-encryption                       (experimental) Enable Secret encryption at rest
   --no-flannel                               (deprecated) use --flannel-backend=none
   --no-deploy value                          (deprecated) Do not deploy packaged components (valid items: coredns, servicelb, traefik, local-storage, metrics-server)
   --cluster-secret value                     (deprecated) use --token [$K3S_CLUSTER_SECRET]

time="2020-07-16T15:51:49.535957091Z" level=fatal msg="flag provided but not defined: -tls-san 192.168.1.35"

And so I thought of "protecting" the first hyphen character, witha backslash, ... and it worked, look :) :

```bash
jbl@pegasusio:~$ k3d create cluster --k3s-server-arg '--tls-san 192.168.1.35' topgunCluster --masters 3 --workers 9
INFO[0000] Created network 'k3d-topgunCluster'
INFO[0000] Created volume 'k3d-topgunCluster-images'
INFO[0000] Creating initializing master node
INFO[0000] Creating node 'k3d-topgunCluster-master-0'
ERRO[0001] Node 'k3d-topgunCluster-master-0' (container '9e4bd7cce69751388c364340c581284c1b3f45691059d05ceebf526d9e13ce09') not running
ERRO[0001] Failed to get logs from the initializig master node.. waiting for 3 seconds instead
INFO[0005] Creating node 'k3d-topgunCluster-master-1'
ERRO[0006] Failed waiting for log message 'Wrote kubeconfig' from node 'k3d-topgunCluster-master-1'
INFO[0007] Creating node 'k3d-topgunCluster-master-2'
ERRO[0007] Failed to create container 'k3d-topgunCluster-master-2'
ERRO[0007] Failed to create node 'k3d-topgunCluster-master-2'
ERRO[0007] Failed to create node
ERRO[0007] error during connect: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create?name=k3d-topgunCluster-master-2": context canceled
ERRO[0007] Failed to create cluster >>> Rolling Back
INFO[0007] Deleting cluster 'topgunCluster'
INFO[0007] Deleted k3d-topgunCluster-master-0
INFO[0007] Deleted k3d-topgunCluster-master-1
ERRO[0007] Failed to delete container 'k3d-topgunCluster-master-2'
WARN[0007] Failed to delete node 'k3d-topgunCluster-master-2': Try to delete it manually
ERRO[0007] Failed to delete container ''
WARN[0007] Failed to delete node '': Try to delete it manually
ERRO[0007] Failed to delete container ''
WARN[0007] Failed to delete node '': Try to delete it manually
ERRO[0007] Failed to delete container ''
WARN[0007] Failed to delete node '': Try to delete it manually
ERRO[0007] Failed to delete container ''
WARN[0007] Failed to delete node '': Try to delete it manually
ERRO[0007] Failed to delete container ''
WARN[0007] Failed to delete node '': Try to delete it manually
ERRO[0007] Failed to delete container ''
WARN[0007] Failed to delete node '': Try to delete it manually
ERRO[0007] Failed to delete container ''
WARN[0007] Failed to delete node '': Try to delete it manually
ERRO[0007] Failed to delete container ''
WARN[0007] Failed to delete node '': Try to delete it manually
ERRO[0007] Failed to delete container ''
WARN[0007] Failed to delete node '': Try to delete it manually
INFO[0007] Deleting cluster network 'cef77d0962c75fb342e130003cf6283b68ef22ad1090e74b15385e835d6b58a4'
ERRO[0007] Failed to delete 10 nodes: Try to delete them manually
FATA[0007] Cluster creation FAILED, also FAILED to rollback changes!
jbl@pegasusio:~$ k3d create cluster --k3s-server-arg "--tls-san 192.168.1.35" topgunCluster --masters 3 --workers 9
INFO[0000] Created network 'k3d-topgunCluster'
INFO[0000] Created volume 'k3d-topgunCluster-images'
INFO[0000] Creating initializing master node
INFO[0000] Creating node 'k3d-topgunCluster-master-0'
INFO[0012] Creating node 'k3d-topgunCluster-master-1'
INFO[0014] Creating node 'k3d-topgunCluster-master-2'
INFO[0015] Creating node 'k3d-topgunCluster-worker-0'
INFO[0016] Creating node 'k3d-topgunCluster-worker-1'
INFO[0017] Creating node 'k3d-topgunCluster-worker-2'
INFO[0018] Creating node 'k3d-topgunCluster-worker-3'
INFO[0019] Creating node 'k3d-topgunCluster-worker-4'
INFO[0020] Creating node 'k3d-topgunCluster-worker-5'
INFO[0021] Creating node 'k3d-topgunCluster-worker-6'
INFO[0023] Creating node 'k3d-topgunCluster-worker-7'
INFO[0024] Creating node 'k3d-topgunCluster-worker-8'
INFO[0026] Creating LoadBalancer 'k3d-topgunCluster-masterlb'
INFO[0027] Pulling image 'docker.io/rancher/k3d-proxy:v3.0.0-rc.6'
INFO[0033] Cluster 'topgunCluster' created successfully!
INFO[0033] You can now use it like this:
export KUBECONFIG=$(k3d get kubeconfig topgunCluster)
kubectl cluster-info
jbl@pegasusio:~$ export KUBECONFIG=$(k3d get kubeconfig topgunCluster)
jbl@pegasusio:~$
jbl@pegasusio:~$ kubectl get all,nodes
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.43.0.1 443/TCP 43s

NAME STATUS ROLES AGE VERSION
node/k3d-topguncluster-worker-4 Ready 27s v1.18.4+k3s1
node/k3d-topguncluster-worker-1 Ready 30s v1.18.4+k3s1
node/k3d-topguncluster-worker-3 Ready 27s v1.18.4+k3s1
node/k3d-topguncluster-worker-0 Ready 32s v1.18.4+k3s1
node/k3d-topguncluster-worker-6 Ready 25s v1.18.4+k3s1
node/k3d-topguncluster-worker-8 Ready 22s v1.18.4+k3s1
node/k3d-topguncluster-worker-7 Ready 22s v1.18.4+k3s1
node/k3d-topguncluster-worker-5 Ready 25s v1.18.4+k3s1
node/k3d-topguncluster-master-1 Ready master 21s v1.18.4+k3s1
node/k3d-topguncluster-master-2 Ready master 17s v1.18.4+k3s1
node/k3d-topguncluster-worker-2 Ready 23s v1.18.4+k3s1
node/k3d-topguncluster-master-0 Ready master 39s v1.18.4+k3s1
jbl@pegasusio:~$

Jean-Baptiste-Lasselle on 16 Jul 2020

also wanted to note : I am constantly trying multi master with latest k3d : until today, I have clusters that have very high instability, at least master zero fails every time very soon, and eventually all masters, especially if you restart them. I am currently working on making available a solid external datastore etcd cluster (a simple docker compose in the same docker network that the k3s, or why not another, but single master k3d cluster...)

Jean-Baptiste-Lasselle on 16 Jul 2020

Thanks for the feedback @Jean-Baptiste-Lasselle :)
The struggle with passing flags on to k3s is real :grimacing:
Also, there are open issues on the k3s repo already regarding the stability of multi-server clusters, which are mostly introduced by the instable embedded dqlite.
As far as I know, it is soon to be replaced by embedded etcd, which will hopefully be more stable.
The most prominent issue is that you cannot stop/restart/remove master-0/server-0 due to dqlite problems.
I'll try to bring external-datastore support into the next release (3.1.0) :+1:

Anything else I can help you with on this issue?

iwilltry42 on 17 Jul 2020

👍1

Hi @iwilltry42

Thanks for the feedback @Jean-Baptiste-Lasselle :)

My pleasure :) : I really get to do awesome things thanks to k3d/k3s, and the team members like you are there to support and give exaplanations / perspective, so it's just great

The struggle with passing flags on to k3s is real :grimacing:

Yeah I saw that, but what like, is that the errors that I find, make sense : there is no dirty tricks to hide awful things into development, and that is why I think the dev team is particularly rigourous in its work and method : If can read their minds even in their errors, well that is really a good sign. Actually, that I can read in a dev teams ' mind even in errors, and guess a workaround, is probably one of the most powerful sign of a rigourous team work .

Also, there are open issues on the k3s repo already regarding the stability of multi-server clusters, which are mostly introduced by the instable embedded dqlite.

Like thank you so much or confirming me that. I analyzed logs of masters that were going down, and down again, til I finally picked that up : hey it is when the master tries to connect to datastore that it fails. Then I found out about the k3s embedded datastore by default. And I immediately thought of bringing up etcd clustered, the classic kubernetes high availability recommandation : dont do just 3 masters, also do a dedicated 3 etcd nodes .

As far as I know, it is soon to be replaced by embedded etcd, which will hopefully be more stable.

Oh really ? interesting I will keep an eye on that. I will be interesting to find out the limits of each possible set up

The most prominent issue is that you cannot stop/restart/remove master-0/server-0 due to dqlite problems.

Very interesting to know too, thank you so much for the info there.

I'll try to bring external-datastore support into the next release (3.1.0) +1

Gosh I so support the feature : yes, It would bring a killer feature to the platform, cause of course then we can operate multi master with high availabilty

My last idea about any feature would be this (if it does not already exist) : How can run a k3s cluster, across physical machines, or maybe, inside kubernetes ? I don't think hyperkube is k3s based (not sure), but I think that between a kubernetes in kubernetes, and a k3s on one physical machine, well there is room for something else.