Rke: bug: rke up on rke 0.2.4 will not remove a node properly
RKE version:
0.2.4
Docker version: (docker version,docker info preferred)
[ec2-user@ip-172-31-1-194 ~]$ docker info
Containers: 33
Running: 20
Paused: 0
Stopped: 13
Images: 17
Server Version: 18.09.6
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-862.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.637GiB
Name: ip-172-31-1-194.us-east-2.compute.internal
ID: 76KV:NCYG:KW36:AXCQ:PJSL:B3GH:TXFH:RYMV:AW2M:2GQK:FIBQ:ZLQK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
[ec2-user@ip-172-31-1-194 ~]$ uname -r
3.10.0-862.el7.x86_64
[ec2-user@ip-172-31-1-194 ~]$ cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.5 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.5"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.5 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.5:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.5"
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
AWS
cluster.yml file:
nodes:
- address: 52.14.223.228
internal_address: 172.31.13.57
port: "22"
role:
- controlplane
- worker
- etcd
hostname_override: ""
user: ec2-user
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: ~/.ssh/id_rsa
labels: {}
- address: 3.16.139.11
internal_address: 172.31.0.77
port: "22"
role:
- controlplane
- worker
- etcd
hostname_override: ""
user: ec2-user
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: ~/.ssh/id_rsa
labels: {}
Steps to Reproduce:
Create a three node rke cluster using 0.2.4, remove one node from the cluster.yml then run rke up again.
Results:
The node is not actually removed and all containers are still running on the node.
$ kubectl get node
NAME STATUS ROLES AGE VERSION
18.219.94.39 Ready controlplane,etcd,worker 2d v1.13.5
3.16.139.11 Ready controlplane,etcd,worker 2d v1.13.5
52.14.223.228 Ready controlplane,etcd,worker 2d v1.13.5
[ec2-user@ip-172-31-1-194 ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
34245a7d1928 2e4d862afebb "/usr/bin/cert-manag…" 12 minutes ago Up 12 minutes k8s_cert-manager_cert-manager-6464494858-4pcqv_kube-system_84e19d97-915a-11e9-999f-02abedb050dc_1
06a91f550340 rancher/rke-tools:v0.1.28 "/opt/rke-tools/rke-…" 31 hours ago Up 31 hours etcd-rolling-snapshots
c565265f3719 rancher/rancher-agent "run.sh" 31 hours ago Up 31 hours k8s_cluster-register_cattle-cluster-agent-6598d655c6-6rch8_cattle-system_0760f252-91d0-11e9-999f-02abedb050dc_0
d6dc36330017 rancher/rancher-agent "run.sh" 31 hours ago Up 31 hours k8s_agent_cattle-node-agent-47swm_cattle-system_0770aebc-91d0-11e9-999f-02abedb050dc_0
c16ccd0e93ce rancher/pause:3.1 "/pause" 31 hours ago Up 31 hours k8s_POD_cattle-node-agent-47swm_cattle-system_0770aebc-91d0-11e9-999f-02abedb050dc_0
e76db9026e89 rancher/pause:3.1 "/pause" 31 hours ago Up 31 hours k8s_POD_cattle-cluster-agent-6598d655c6-6rch8_cattle-system_0760f252-91d0-11e9-999f-02abedb050dc_0
9330d5c6af6b 2c8edab4e2d9 "entrypoint.sh --htt…" 45 hours ago Up 45 hours k8s_rancher_rancher-6fd48d6f59-g8jff_cattle-system_84e20eff-915a-11e9-999f-02abedb050dc_0
36ae2a6844c4 rancher/pause:3.1 "/pause" 45 hours ago Up 45 hours k8s_POD_rancher-6fd48d6f59-g8jff_cattle-system_84e20eff-915a-11e9-999f-02abedb050dc_0
c10e9fe009e4 rancher/pause:3.1 "/pause" 45 hours ago Up 45 hours k8s_POD_cert-manager-6464494858-4pcqv_kube-system_84e19d97-915a-11e9-999f-02abedb050dc_0
e46e4d2491a6 2c8edab4e2d9 "entrypoint.sh --htt…" 2 days ago Up 2 days k8s_rancher_rancher-6fd48d6f59-6c9cc_cattle-system_55db7e74-9139-11e9-aaa2-02cab5d5952a_1
23f4694d8fde rancher/pause:3.1 "/pause" 2 days ago Up 2 days k8s_POD_rancher-6fd48d6f59-6c9cc_cattle-system_55db7e74-9139-11e9-aaa2-02cab5d5952a_0
51c19fbc0b2e gcr.io/kubernetes-helm/tiller "/tiller" 2 days ago Up 2 days k8s_tiller_tiller-deploy-7b489d95c4-jh6q6_kube-system_44b35f89-9139-11e9-aaa2-02cab5d5952a_0
76cc6a3ba25d rancher/nginx-ingress-controller "/entrypoint.sh /ngi…" 2 days ago Up 2 days k8s_nginx-ingress-controller_nginx-ingress-controller-222sf_ingress-nginx_41f50144-9139-11e9-aaa2-02cab5d5952a_0
c49e0e90ec2e rancher/pause:3.1 "/pause" 2 days ago Up 2 days k8s_POD_tiller-deploy-7b489d95c4-jh6q6_kube-system_44b35f89-9139-11e9-aaa2-02cab5d5952a_0
e9ae0a5ed726 rancher/pause:3.1 "/pause" 2 days ago Up 2 days k8s_POD_nginx-ingress-controller-222sf_ingress-nginx_41f50144-9139-11e9-aaa2-02cab5d5952a_0
def71e1cf56e rancher/coreos-flannel "/opt/bin/flanneld -…" 2 days ago Up 2 days k8s_kube-flannel_canal-kdzr2_kube-system_376f1a34-9139-11e9-aaa2-02cab5d5952a_0
75a5a7a92be5 rancher/calico-node "start_runit" 2 days ago Up 2 days k8s_calico-node_canal-kdzr2_kube-system_376f1a34-9139-11e9-aaa2-02cab5d5952a_0
5695bc2f3648 rancher/pause:3.1 "/pause" 2 days ago Up 2 days k8s_POD_canal-kdzr2_kube-system_376f1a34-9139-11e9-aaa2-02cab5d5952a_0
1b2092a5b533 rancher/hyperkube:v1.13.5-rancher1 "/opt/rke-tools/entr…" 2 days ago Up 2 days kube-proxy
2e20ea7e9cca rancher/hyperkube:v1.13.5-rancher1 "/opt/rke-tools/entr…" 2 days ago Up 2 days kubelet
patrick0057
All 3 comments
Ran in the same problem today. Happens with 0.2.3 and 0.2.4, 0.2.2 works.
Here is the log after removing node 10.22.141.38 in cluster.yml:
rke up --disable-port-check --ssh-agent-auth --config cluster.yml
time="2019-07-02T08:36:46+02:00" level=info msg="Initiating Kubernetes cluster"
time="2019-07-02T08:36:46+02:00" level=info msg="[certificates] Generating Kubernetes API server certificates"
time="2019-07-02T08:36:46+02:00" level=info msg="[certificates] Generating admin certificates and kubeconfig"
time="2019-07-02T08:36:46+02:00" level=info msg="[certificates] Generating etcd-10.22.141.36 certificate and key"
time="2019-07-02T08:36:46+02:00" level=info msg="[certificates] Generating etcd-10.22.141.37 certificate and key"
time="2019-07-02T08:36:46+02:00" level=info msg="Successfully Deployed state file at [./cluster.rkestate]"
time="2019-07-02T08:36:46+02:00" level=info msg="Building Kubernetes cluster"
time="2019-07-02T08:36:46+02:00" level=info msg="[dialer] Setup tunnel for host [10.22.141.36]"
time="2019-07-02T08:36:46+02:00" level=info msg="[dialer] Setup tunnel for host [10.22.141.37]"
time="2019-07-02T08:36:46+02:00" level=info msg="[certificates] kube-apiserver certificate changed, force deploying certs"
time="2019-07-02T08:36:46+02:00" level=info msg="[certificates] Deploying kubernetes certificates to Cluster nodes"
time="2019-07-02T08:36:52+02:00" level=info msg="[reconcile] Rebuilding and updating local kube config"
time="2019-07-02T08:36:52+02:00" level=info msg="Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]"
time="2019-07-02T08:36:52+02:00" level=info msg="[reconcile] host [10.22.141.36] is active master on the cluster"
time="2019-07-02T08:36:52+02:00" level=info msg="[certificates] Successfully deployed kubernetes certificates to Cluster nodes"
time="2019-07-02T08:36:52+02:00" level=info msg="[reconcile] Reconciling cluster state"
time="2019-07-02T08:36:52+02:00" level=info msg="[reconcile] Check etcd hosts to be deleted"
time="2019-07-02T08:36:52+02:00" level=info msg="[remove/etcd] Removing member [etcd-10.22.141.38] from etcd cluster"
time="2019-07-02T08:36:53+02:00" level=info msg="[remove/etcd] Successfully removed member [etcd-10.22.141.38] from etcd cluster"
time="2019-07-02T08:36:53+02:00" level=info msg="[hosts] host [10.22.141.38] has another role, skipping delete from kubernetes cluster"
time="2019-07-02T08:36:53+02:00" level=info msg="[dialer] Setup tunnel for host [10.22.141.38]"
time="2019-07-02T08:36:53+02:00" level=info msg="[etcd] Tearing down etcd plane.."
time="2019-07-02T08:36:54+02:00" level=info msg="[remove/etcd] Successfully removed container on host [10.22.141.38]"
time="2019-07-02T08:36:54+02:00" level=info msg="[etcd] Successfully tore down etcd plane.."
time="2019-07-02T08:36:54+02:00" level=info msg="[hosts] Host [10.22.141.38] is already a worker or control host, skipping cleanup certs."
time="2019-07-02T08:36:54+02:00" level=info msg="[hosts] Cleaning up host [10.22.141.38]"
time="2019-07-02T08:36:54+02:00" level=info msg="[hosts] Running cleaner container on host [10.22.141.38]"
time="2019-07-02T08:36:54+02:00" level=info msg="[kube-cleaner] Successfully started [kube-cleaner] container on host [10.22.141.38]"
time="2019-07-02T08:36:54+02:00" level=info msg="Waiting for [kube-cleaner] container to exit on host [10.22.141.38]"
time="2019-07-02T08:36:54+02:00" level=info msg="Container [kube-cleaner] is still running on host [10.22.141.38]"
time="2019-07-02T08:36:55+02:00" level=info msg="Waiting for [kube-cleaner] container to exit on host [10.22.141.38]"
time="2019-07-02T08:36:55+02:00" level=info msg="[hosts] Removing cleaner container on host [10.22.141.38]"
time="2019-07-02T08:36:55+02:00" level=info msg="[hosts] Removing dead container logs on host [10.22.141.38]"
time="2019-07-02T08:36:56+02:00" level=info msg="[cleanup] Successfully started [rke-log-cleaner] container on host [10.22.141.38]"
time="2019-07-02T08:36:56+02:00" level=info msg="[remove/rke-log-cleaner] Successfully removed container on host [10.22.141.38]"
time="2019-07-02T08:36:56+02:00" level=info msg="[hosts] Successfully cleaned up host [10.22.141.38]"
time="2019-07-02T08:36:56+02:00" level=info msg="[reconcile] Check etcd hosts to be added"
time="2019-07-02T08:36:56+02:00" level=info msg="[hosts] host [10.22.141.38] has another role, skipping delete from kubernetes cluster"
time="2019-07-02T08:36:56+02:00" level=info msg="[worker] Tearing down Worker Plane.."
time="2019-07-02T08:36:56+02:00" level=info msg="[worker] Host [10.22.141.38] is already a controlplane host, nothing to do."
time="2019-07-02T08:36:56+02:00" level=info msg="[worker] Successfully tore down Worker Plane.."
time="2019-07-02T08:36:56+02:00" level=info msg="[hosts] Host [10.22.141.38] is already a controlplane or etcd host, skipping cleanup."
time="2019-07-02T08:36:56+02:00" level=info msg="[hosts] host [10.22.141.38] has another role, skipping delete from kubernetes cluster"
time="2019-07-02T08:36:56+02:00" level=info msg="[controlplane] Tearing down the Controller Plane.."
time="2019-07-02T08:36:57+02:00" level=info msg="[remove/kube-apiserver] Successfully removed container on host [10.22.141.38]"
time="2019-07-02T08:36:57+02:00" level=info msg="[remove/kube-controller-manager] Successfully removed container on host [10.22.141.38]"
time="2019-07-02T08:36:57+02:00" level=info msg="[remove/kube-scheduler] Successfully removed container on host [10.22.141.38]"
time="2019-07-02T08:36:57+02:00" level=info msg="[controlplane] Successfully tore down Controller Plane.."
time="2019-07-02T08:36:57+02:00" level=info msg="[hosts] Host [10.22.141.38] is already a worker or etcd host, skipping cleanup."
time="2019-07-02T08:36:57+02:00" level=info msg="[reconcile] Rebuilding and updating local kube config"
time="2019-07-02T08:36:57+02:00" level=info msg="Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml]"
time="2019-07-02T08:36:57+02:00" level=info msg="[reconcile] host [10.22.141.36] is active master on the cluster"
time="2019-07-02T08:37:03+02:00" level=info msg="[restart/kube-apiserver] Successfully restarted container on host [10.22.141.36]"
time="2019-07-02T08:37:05+02:00" level=info msg="[restart/kube-controller-manager] Successfully restarted container on host [10.22.141.36]"
time="2019-07-02T08:37:06+02:00" level=info msg="[restart/kube-apiserver] Successfully restarted container on host [10.22.141.37]"
time="2019-07-02T08:37:12+02:00" level=info msg="[restart/kube-controller-manager] Successfully restarted container on host [10.22.141.37]"
time="2019-07-02T08:37:17+02:00" level=info msg="[restart/etcd] Successfully restarted container on host [10.22.141.36]"
time="2019-07-02T08:37:23+02:00" level=info msg="[restart/etcd] Successfully restarted container on host [10.22.141.37]"
time="2019-07-02T08:37:23+02:00" level=info msg="[reconcile] Reconciled cluster state successfully"
time="2019-07-02T08:37:23+02:00" level=info msg="Pre-pulling kubernetes images"
time="2019-07-02T08:37:23+02:00" level=info msg="Kubernetes images pulled successfully"
time="2019-07-02T08:37:23+02:00" level=info msg="[etcd] Building up etcd plane.."
time="2019-07-02T08:37:24+02:00" level=info msg="Waiting for [etcd] container to exit on host [10.22.141.36]"
time="2019-07-02T08:37:24+02:00" level=info msg="[etcd] Successfully updated [etcd] container on host [10.22.141.36]"
time="2019-07-02T08:37:24+02:00" level=info msg="[etcd] Saving snapshot [etcd-rolling-snapshots] on host [10.22.141.36]"
time="2019-07-02T08:37:25+02:00" level=info msg="[remove/etcd-rolling-snapshots] Successfully removed container on host [10.22.141.36]"
time="2019-07-02T08:37:25+02:00" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.22.141.36]"
time="2019-07-02T08:37:31+02:00" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [10.22.141.36]"
time="2019-07-02T08:37:31+02:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.22.141.36]"
time="2019-07-02T08:37:31+02:00" level=info msg="Container [rke-bundle-cert] is still running on host [10.22.141.36]"
time="2019-07-02T08:37:32+02:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.22.141.36]"
time="2019-07-02T08:37:32+02:00" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.22.141.36]"
time="2019-07-02T08:37:33+02:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.22.141.36]"
time="2019-07-02T08:37:33+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.36]"
time="2019-07-02T08:37:35+02:00" level=info msg="Waiting for [etcd] container to exit on host [10.22.141.37]"
time="2019-07-02T08:37:36+02:00" level=info msg="[etcd] Successfully updated [etcd] container on host [10.22.141.37]"
time="2019-07-02T08:37:36+02:00" level=info msg="[etcd] Saving snapshot [etcd-rolling-snapshots] on host [10.22.141.37]"
time="2019-07-02T08:37:36+02:00" level=info msg="[remove/etcd-rolling-snapshots] Successfully removed container on host [10.22.141.37]"
time="2019-07-02T08:37:37+02:00" level=info msg="[etcd] Successfully started [etcd-rolling-snapshots] container on host [10.22.141.37]"
time="2019-07-02T08:37:42+02:00" level=info msg="[certificates] Successfully started [rke-bundle-cert] container on host [10.22.141.37]"
time="2019-07-02T08:37:42+02:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.22.141.37]"
time="2019-07-02T08:37:42+02:00" level=info msg="Container [rke-bundle-cert] is still running on host [10.22.141.37]"
time="2019-07-02T08:37:43+02:00" level=info msg="Waiting for [rke-bundle-cert] container to exit on host [10.22.141.37]"
time="2019-07-02T08:37:43+02:00" level=info msg="[certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on host [10.22.141.37]"
time="2019-07-02T08:37:44+02:00" level=info msg="[etcd] Successfully started [rke-log-linker] container on host [10.22.141.37]"
time="2019-07-02T08:37:44+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.37]"
time="2019-07-02T08:37:44+02:00" level=info msg="[etcd] Successfully started etcd plane.. Checking etcd cluster health"
time="2019-07-02T08:37:45+02:00" level=info msg="[controlplane] Building up Controller Plane.."
time="2019-07-02T08:37:45+02:00" level=info msg="[sidekick] Sidekick container already created on host [10.22.141.37]"
time="2019-07-02T08:37:45+02:00" level=info msg="[sidekick] Sidekick container already created on host [10.22.141.36]"
time="2019-07-02T08:37:45+02:00" level=info msg="Waiting for [kube-apiserver] container to exit on host [10.22.141.37]"
time="2019-07-02T08:37:45+02:00" level=info msg="Waiting for [kube-apiserver] container to exit on host [10.22.141.36]"
time="2019-07-02T08:37:46+02:00" level=info msg="[controlplane] Successfully updated [kube-apiserver] container on host [10.22.141.37]"
time="2019-07-02T08:37:46+02:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.22.141.37]"
time="2019-07-02T08:37:46+02:00" level=info msg="[controlplane] Successfully updated [kube-apiserver] container on host [10.22.141.36]"
time="2019-07-02T08:37:46+02:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-apiserver] on host [10.22.141.36]"
time="2019-07-02T08:37:58+02:00" level=info msg="[healthcheck] service [kube-apiserver] on host [10.22.141.37] is healthy"
time="2019-07-02T08:37:58+02:00" level=info msg="[healthcheck] service [kube-apiserver] on host [10.22.141.36] is healthy"
time="2019-07-02T08:37:59+02:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.22.141.37]"
time="2019-07-02T08:37:59+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.37]"
time="2019-07-02T08:37:59+02:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.22.141.37]"
time="2019-07-02T08:37:59+02:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.22.141.36]"
time="2019-07-02T08:37:59+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.36]"
time="2019-07-02T08:37:59+02:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-controller-manager] on host [10.22.141.36]"
time="2019-07-02T08:37:59+02:00" level=info msg="[healthcheck] service [kube-controller-manager] on host [10.22.141.37] is healthy"
time="2019-07-02T08:38:00+02:00" level=info msg="[healthcheck] service [kube-controller-manager] on host [10.22.141.36] is healthy"
time="2019-07-02T08:38:00+02:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.22.141.37]"
time="2019-07-02T08:38:00+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.37]"
time="2019-07-02T08:38:00+02:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.22.141.37]"
time="2019-07-02T08:38:01+02:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.22.141.36]"
time="2019-07-02T08:38:01+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.36]"
time="2019-07-02T08:38:01+02:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-scheduler] on host [10.22.141.36]"
time="2019-07-02T08:38:01+02:00" level=info msg="[healthcheck] service [kube-scheduler] on host [10.22.141.37] is healthy"
time="2019-07-02T08:38:01+02:00" level=info msg="[healthcheck] service [kube-scheduler] on host [10.22.141.36] is healthy"
time="2019-07-02T08:38:01+02:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.22.141.37]"
time="2019-07-02T08:38:02+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.37]"
time="2019-07-02T08:38:02+02:00" level=info msg="[controlplane] Successfully started [rke-log-linker] container on host [10.22.141.36]"
time="2019-07-02T08:38:02+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.36]"
time="2019-07-02T08:38:02+02:00" level=info msg="[controlplane] Successfully started Controller Plane.."
time="2019-07-02T08:38:02+02:00" level=info msg="[authz] Creating rke-job-deployer ServiceAccount"
time="2019-07-02T08:38:02+02:00" level=info msg="[authz] rke-job-deployer ServiceAccount created successfully"
time="2019-07-02T08:38:02+02:00" level=info msg="[authz] Creating system:node ClusterRoleBinding"
time="2019-07-02T08:38:02+02:00" level=info msg="[authz] system:node ClusterRoleBinding created successfully"
time="2019-07-02T08:38:02+02:00" level=info msg="Successfully Deployed state file at [./cluster.rkestate]"
time="2019-07-02T08:38:02+02:00" level=info msg="[state] Saving full cluster state to Kubernetes"
time="2019-07-02T08:38:02+02:00" level=info msg="[state] Successfully Saved full cluster state to Kubernetes ConfigMap: cluster-state"
time="2019-07-02T08:38:02+02:00" level=info msg="[worker] Building up Worker Plane.."
time="2019-07-02T08:38:02+02:00" level=info msg="[sidekick] Sidekick container already created on host [10.22.141.36]"
time="2019-07-02T08:38:02+02:00" level=info msg="[sidekick] Sidekick container already created on host [10.22.141.37]"
time="2019-07-02T08:38:02+02:00" level=info msg="[healthcheck] Start Healthcheck on service [kubelet] on host [10.22.141.36]"
time="2019-07-02T08:38:02+02:00" level=info msg="[healthcheck] Start Healthcheck on service [kubelet] on host [10.22.141.37]"
time="2019-07-02T08:38:03+02:00" level=info msg="[healthcheck] service [kubelet] on host [10.22.141.36] is healthy"
time="2019-07-02T08:38:03+02:00" level=info msg="[healthcheck] service [kubelet] on host [10.22.141.37] is healthy"
time="2019-07-02T08:38:03+02:00" level=info msg="[worker] Successfully started [rke-log-linker] container on host [10.22.141.37]"
time="2019-07-02T08:38:04+02:00" level=info msg="[worker] Successfully started [rke-log-linker] container on host [10.22.141.36]"
time="2019-07-02T08:38:04+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.37]"
time="2019-07-02T08:38:04+02:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.22.141.37]"
time="2019-07-02T08:38:04+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.36]"
time="2019-07-02T08:38:04+02:00" level=info msg="[healthcheck] Start Healthcheck on service [kube-proxy] on host [10.22.141.36]"
time="2019-07-02T08:38:04+02:00" level=info msg="[healthcheck] service [kube-proxy] on host [10.22.141.37] is healthy"
time="2019-07-02T08:38:04+02:00" level=info msg="[healthcheck] service [kube-proxy] on host [10.22.141.36] is healthy"
time="2019-07-02T08:38:05+02:00" level=info msg="[worker] Successfully started [rke-log-linker] container on host [10.22.141.37]"
time="2019-07-02T08:38:05+02:00" level=info msg="[worker] Successfully started [rke-log-linker] container on host [10.22.141.36]"
time="2019-07-02T08:38:05+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.37]"
time="2019-07-02T08:38:05+02:00" level=info msg="[remove/rke-log-linker] Successfully removed container on host [10.22.141.36]"
time="2019-07-02T08:38:05+02:00" level=info msg="[worker] Successfully started Worker Plane.."
time="2019-07-02T08:38:06+02:00" level=info msg="[cleanup] Successfully started [rke-log-cleaner] container on host [10.22.141.37]"
time="2019-07-02T08:38:06+02:00" level=info msg="[cleanup] Successfully started [rke-log-cleaner] container on host [10.22.141.36]"
time="2019-07-02T08:38:06+02:00" level=info msg="[remove/rke-log-cleaner] Successfully removed container on host [10.22.141.37]"
time="2019-07-02T08:38:06+02:00" level=info msg="[remove/rke-log-cleaner] Successfully removed container on host [10.22.141.36]"
time="2019-07-02T08:38:06+02:00" level=info msg="[sync] Syncing nodes Labels and Taints"
time="2019-07-02T08:38:06+02:00" level=info msg="[sync] Successfully synced nodes Labels and Taints"
time="2019-07-02T08:38:06+02:00" level=info msg="[network] Setting up network plugin: canal"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Executing deploy job rke-network-plugin"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Setting up kube-dns"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Saving ConfigMap for addon rke-kube-dns-addon to Kubernetes"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Successfully saved ConfigMap for addon rke-kube-dns-addon to Kubernetes"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Executing deploy job rke-kube-dns-addon"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] kube-dns deployed successfully"
time="2019-07-02T08:38:06+02:00" level=info msg="[dns] DNS provider kube-dns deployed successfully"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Setting up Metrics Server"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Executing deploy job rke-metrics-addon"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Metrics Server deployed successfully"
time="2019-07-02T08:38:06+02:00" level=info msg="[ingress] Setting up nginx ingress controller"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Executing deploy job rke-ingress-controller"
time="2019-07-02T08:38:06+02:00" level=info msg="[ingress] ingress controller nginx deployed successfully"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] Setting up user addons"
time="2019-07-02T08:38:06+02:00" level=info msg="[addons] no user addons defined"
time="2019-07-02T08:38:06+02:00" level=info msg="Finished building Kubernetes cluster successfully"
yvespp
on 2 Jul 2019
reproduced the issue using rke v0.2.4
Verified with rke: v0.3.0-rc5
Steps taken:
Create a three node rke cluster
Remove one node from the cluster.yml then run rke up again.
The node is removed
Sowmyas-MBP:rke-import-cluster$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
x.x.x.x Ready controlplane,etcd,worker 45m v1.15.0
y.y.y.y Ready controlplane,etcd,worker 45m v1.15.0
But these containers are still running. Logging this in a separate issue - https://github.com/rancher/rke/issues/1510
sowmyav27
on 25 Jul 2019
To workaround this issue on RKE v0.2.x -
- Remove node(s) from cluster.yml
- Run
rke up - Run
kubectl delete node <node name>
dnoland1
on 20 Aug 2019
👍3
Was this page helpful?
0 / 5 - 0 ratings
Related issues
HighwayofLife
·
4Comments
galal-hussein
·
4Comments
dnoland1
·
3Comments
yasker
·
4Comments
armanriazi
·
3Comments
Most helpful comment
To workaround this issue on RKE v0.2.x -
rke upkubectl delete node <node name>