K3os: Unable to build k3s-Cluster by using config.yaml

Created on 18 Feb 2020 · 5Comments · Source: rancher/k3os

Version (k3OS / kernel)

$ k3os --version
k3os version v0.9.0

$ uname --kernel-release --kernel-version
5.0.0-37-generic #40~18.04.1 SMP Wed Jan 15 04:09:29 UTC 2020

Architecture

$ uname --machine
x86_64

Describe the bug
When using the config.yaml as described in the documentation, I can't get up and running a cluster with 3 master nodes.

First-Server

The first cluster-server bootstraps itself just fine (I assume).

$ hostname
kcluster-master-01

$ cat /var/lib/rancher/k3os/config.d/config.yaml

ssh_authorized_keys:
  - ssh-rsa AAAAB3NzaC1yc...
#write_files:
# files can be added here
# After creating two masters setting the hostname was useless
hostname: "kcluster-master-01.cluster.k8s"
k3os:
  data_sources:
  modules:
    - kvm
    - nvme
  sysctl:
    kernel.printk: 4 4 1 7
    kernel.kptr_restrict: 1
  ntp_servers:
    - 1.de.pool.ntp.org
    - 2.de.pool.ntp.org
    - 3.de.pool.ntp.org
  dns_nameservers:
    - 192.168.0.10
    - 192.168.0.11
  # Rancher User PW
  password: "rancher"
  token: "ue3ooch4aep7Yubae3lahliejaequal6"
  labels:
    region: eu-de-01
    prupose: testing
  k3s_args:
    - server
  environment:
  taints:
    - k3s-controlplane=true:NoExecute
    - k3s-controlplane=true:NoSchedule

# manual clean up old installations or stuff from previous bootstraps
$ rm -rf /var/lib/rancher/k3s/*
$ reboot

After the restart:
The one node is showing up:

$ kubectl get nodes
NAME                          STATUS   ROLES    AGE     VERSION
kcluster-master-01   Ready    master   8m54s   v1.17.2+k3s1

Cluster-info looks fine, too:

$ kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

$ kubectl cluster-info dump is available if required

$ tail -n 50 /var/log/k3s-service.log
I0218 16:30:43.663042    2484 range_allocator.go:82] Sending events to api server.
I0218 16:30:43.663162    2484 range_allocator.go:110] No Service CIDR provided. Skipping filtering out service addresses.
I0218 16:30:43.663173    2484 range_allocator.go:116] No Secondary Service CIDR provided. Skipping filtering out secondary service addresses.
I0218 16:30:43.663219    2484 controllermanager.go:533] Started "nodeipam"
I0218 16:30:43.663778    2484 shared_informer.go:197] Waiting for caches to sync for garbage collector
I0218 16:30:43.663806    2484 node_ipam_controller.go:162] Starting ipam controller
I0218 16:30:43.663812    2484 shared_informer.go:197] Waiting for caches to sync for node
I0218 16:30:43.672769    2484 shared_informer.go:197] Waiting for caches to sync for resource quota
I0218 16:30:43.738496    2484 shared_informer.go:204] Caches are synced for PV protection
I0218 16:30:43.738572    2484 shared_informer.go:204] Caches are synced for endpoint
I0218 16:30:43.738760    2484 shared_informer.go:204] Caches are synced for service account
I0218 16:30:43.738818    2484 shared_informer.go:204] Caches are synced for certificate-csrapproving
I0218 16:30:43.747118    2484 shared_informer.go:204] Caches are synced for deployment
I0218 16:30:43.749836    2484 shared_informer.go:204] Caches are synced for certificate-csrsigning
I0218 16:30:43.755273    2484 shared_informer.go:204] Caches are synced for namespace
I0218 16:30:43.788765    2484 shared_informer.go:204] Caches are synced for HPA
I0218 16:30:43.794228    2484 shared_informer.go:204] Caches are synced for ReplicaSet
I0218 16:30:43.820018    2484 shared_informer.go:204] Caches are synced for ClusterRoleAggregator
I0218 16:30:43.910684    2484 shared_informer.go:204] Caches are synced for PVC protection
I0218 16:30:43.911129    2484 shared_informer.go:204] Caches are synced for expand
I0218 16:30:44.078735    2484 shared_informer.go:204] Caches are synced for job
I0218 16:30:44.138432    2484 shared_informer.go:204] Caches are synced for ReplicationController
W0218 16:30:44.167735    2484 actual_state_of_world.go:506] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="kcluster-master-01" does not exist
I0218 16:30:44.197157    2484 shared_informer.go:204] Caches are synced for taint
I0218 16:30:44.197296    2484 node_lifecycle_controller.go:1443] Initializing eviction metric for zone:
W0218 16:30:44.197419    2484 node_lifecycle_controller.go:1058] Missing timestamp for Node kcluster-master-01. Assuming now as a timestamp.
I0218 16:30:44.197473    2484 node_lifecycle_controller.go:1259] Controller detected that zone  is now in state Normal.
I0218 16:30:44.197907    2484 taint_manager.go:186] Starting NoExecuteTaintManager
I0218 16:30:44.198115    2484 event.go:281] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"kcluster-master-01", UID:"f675d809-b2de-4052-af5e-deda09b12016", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node kcluster-master-01 event: Registered Node kcluster-master-01 in Controller
I0218 16:30:44.201723    2484 shared_informer.go:204] Caches are synced for disruption
I0218 16:30:44.201759    2484 disruption.go:338] Sending events to api server.
I0218 16:30:44.201846    2484 shared_informer.go:204] Caches are synced for stateful set
I0218 16:30:44.203625    2484 shared_informer.go:204] Caches are synced for daemon sets
I0218 16:30:44.206260    2484 shared_informer.go:204] Caches are synced for TTL
I0218 16:30:44.208324    2484 shared_informer.go:204] Caches are synced for attach detach
I0218 16:30:44.217677    2484 shared_informer.go:204] Caches are synced for persistent volume
I0218 16:30:44.220930    2484 shared_informer.go:204] Caches are synced for resource quota
I0218 16:30:44.243274    2484 shared_informer.go:204] Caches are synced for GC
I0218 16:30:44.264099    2484 shared_informer.go:204] Caches are synced for node
I0218 16:30:44.264162    2484 range_allocator.go:172] Starting range CIDR allocator
I0218 16:30:44.264174    2484 shared_informer.go:197] Waiting for caches to sync for cidrallocator
I0218 16:30:44.264185    2484 shared_informer.go:204] Caches are synced for cidrallocator
I0218 16:30:44.264291    2484 shared_informer.go:204] Caches are synced for garbage collector
I0218 16:30:44.273017    2484 shared_informer.go:204] Caches are synced for resource quota
I0218 16:30:44.273410    2484 shared_informer.go:204] Caches are synced for garbage collector
I0218 16:30:44.273448    2484 garbagecollector.go:138] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
E0218 16:30:44.675588    2484 memcache.go:199] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0218 16:30:44.726152    2484 memcache.go:111] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
time="2020-02-18T16:30:46.288832963Z" level=error msg="failed to process config: failed to process /var/lib/rancher/k3s/server/manifests/system-upgrade-plans/k3os-latest.yaml: the server could not find the requested resource"
time="2020-02-18T16:31:01.907056384Z" level=error msg="failed to process config: failed to process /var/lib/rancher/k3s/server/manifests/system-upgrade-plans/k3os-latest.yaml: the server could not find the requested resource"

This is the generated node-token:

$ cat /var/lib/rancher/k3s/server/node-token
K10017c8c06255b55d07b38127d3877608cac6d71050ff08ddec394bd688c174825::server:ue3ooch4aep7Yubae3lahliejaequal6

2nd Cluster-Server

$ cat /var/lib/rancher/k3os/config.d/config.yaml
ssh_authorized_keys:
  - ssh-rsa AAAAB3NzaC1yc...
#write_files:
# files can be added here
# After creating two masters setting the hostname was useless
hostname: "kcluster-master-02.cluster.k8s"
k3os:
  data_sources:
  modules:
    - kvm
    - nvme
  sysctl:
    kernel.printk: 4 4 1 7
    kernel.kptr_restrict: 1
  ntp_servers:
    - 1.de.pool.ntp.org
    - 2.de.pool.ntp.org
    - 3.de.pool.ntp.org
  dns_nameservers:
    - 192.168.0.10
    - 192.168.0.11
  # Rancher User PW
  password: "rancher"
  token: "ue3ooch4aep7Yubae3lahliejaequal6"
  labels:
    region: eu-de-01
    prupose: testing
  k3s_args:
    - server
    - "--server=https://kcluster-master-01.cluster.k8s:6443"
  environment:
  taints:
    - k3s-controlplane=true:NoExecute
    - k3s-controlplane=true:NoSchedule

# manual clean up old installations or stuff from previous bootstraps
$ rm -rf /var/lib/rancher/k3s/*
$ reboot

$ tail -n 23 /var/log/k3s-service.log
time="2020-02-18T15:30:41Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/7c4aaa633ac3ff4849166ba2759da158a70beb5734940e84b6e67011a35f4c59"
time="2020-02-18T15:30:44.015492952Z" level=info msg="Starting k3s v1.17.2+k3s1 (cdab19b0)"
time="2020-02-18T15:30:44.079256291Z" level=fatal msg="starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden"
time="2020-02-18T15:30:49.657380836Z" level=info msg="Starting k3s v1.17.2+k3s1 (cdab19b0)"
time="2020-02-18T15:30:49.712356802Z" level=fatal msg="starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden"
time="2020-02-18T15:30:55.360412137Z" level=info msg="Starting k3s v1.17.2+k3s1 (cdab19b0)"
time="2020-02-18T15:30:55.421552281Z" level=fatal msg="starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden"
time="2020-02-18T15:31:01.002112591Z" level=info msg="Starting k3s v1.17.2+k3s1 (cdab19b0)"
time="2020-02-18T15:31:01.067896429Z" level=fatal msg="starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden"
time="2020-02-18T15:31:06.662785328Z" level=info msg="Starting k3s v1.17.2+k3s1 (cdab19b0)"
time="2020-02-18T15:31:06.728213395Z" level=fatal msg="starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden"
time="2020-02-18T15:31:12.352637364Z" level=info msg="Starting k3s v1.17.2+k3s1 (cdab19b0)"
time="2020-02-18T15:31:12.426491291Z" level=fatal msg="starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden"
time="2020-02-18T15:31:17.998174906Z" level=info msg="Starting k3s v1.17.2+k3s1 (cdab19b0)"
time="2020-02-18T15:31:18.067189425Z" level=fatal msg="starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden"
time="2020-02-18T15:31:23.632295883Z" level=info msg="Starting k3s v1.17.2+k3s1 (cdab19b0)"
time="2020-02-18T15:31:23.700657530Z" level=fatal msg="starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden"
time="2020-02-18T15:31:29.306664232Z" level=info msg="Starting k3s v1.17.2+k3s1 (cdab19b0)"
time="2020-02-18T15:31:29.371734934Z" level=fatal msg="starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden"
time="2020-02-18T15:31:34.952518969Z" level=info msg="Starting k3s v1.17.2+k3s1 (cdab19b0)"
time="2020-02-18T15:31:35.014908720Z" level=fatal msg="starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden"
time="2020-02-18T15:31:40.585244942Z" level=info msg="Starting k3s v1.17.2+k3s1 (cdab19b0)"
time="2020-02-18T15:31:40.637836937Z" level=fatal msg="starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden"

I manual clean up old installations or stuff from previous bootstraps again.

$ rm -rf /var/lib/rancher/k3s/*

I manual try to join the cluster

$ K3S_TOKEN=ue3ooch4aep7Yubae3lahliejaequal6 k3s server --server https://kcluster-master-01.cluster.k8s:6443
INFO[2020-02-18T16:17:32.187619686Z] Starting k3s v1.17.2+k3s1 (cdab19b0)         
FATA[2020-02-18T16:17:32.253155062Z] starting kubernetes: preparing server: https://kcluster-master-01.cluster.k8s:6443/v1-k3s/server-bootstrap: 403 Forbidden

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

To Reproduce

Use the provided configuration files on two to three servers

Expected behavior

I expected the first server to bootstrap using the provided token and then join the second server using the same token.
I orintend myself here:

k3os.token
The cluster secret or node token. If the value matches the format of a node token it will automatically be assume to be a node token. Otherwise it is treated as a cluster secret.
Example
k3os:  
 token: myclustersecret

Actual behavior
The token or cluster-secret seamingle to be ignored or wrongly specified.

kinbug

Source

Moep90

👍1

Most helpful comment

@dweomer Yes thats what I‘m mostly working with.
Its not required since the first node will initiate itself if there is no configuration. It also feels a little dirty to always start a server with --cluster-init

Also whats confusing is that the documentation tells about the token/clustersecret.
When trying to match a node-token format you‘ve to know (from the code) to prefix it wit K10 and if its fully set, the first joining server complains about a missmatch between token and server.

Update: 2020-02-19:

The flag seams to do it:

  k3s_args:
    - server
    - "-v=3"
    - "--cluster-init"

But it seams to work but the logs are indicating a --cluster-init behavior on every reboot.

Moep90 on 19 Feb 2020

🎉2 👍1

All 5 comments

@Moep90 have you reviewed https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/ ? I believe that the first node needs to be run with the --cluster-init argument for multi-master.

dweomer on 18 Feb 2020

Update: 2020-02-19:

The flag seams to do it:

  k3s_args:
    - server
    - "-v=3"
    - "--cluster-init"

But it seams to work but the logs are indicating a --cluster-init behavior on every reboot.

Moep90 on 19 Feb 2020

🎉2 👍1

I have the exact same issue. Thanks, that option solved my problem. However, don't you think it would be helpful to put a URL to this in the Readme for k3os?