Rke: 'rke up' fails to get current full cluster state

[garyhuang123]

RKE version:
v0.2.2

Docker version: (docker version,docker info preferred)
Client:
Version: 17.12.1-ce
API version: 1.35
Go version: go1.9.4
Git commit: 7390fc6
Built: Tue Feb 27 22:15:20 2018
OS/Arch: linux/amd64

Server:
Engine:
Version: 17.12.1-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.9.4
Git commit: 7390fc6
Built: Tue Feb 27 22:17:54 2018
OS/Arch: linux/amd64
Experimental: false

Operating system and kernel: (cat /etc/os-release, uname -r preferred)
CentOS Linux release 7.5.1804 (Core)
3.10.0-957.10.1.el7.x86_64

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
VMWare

cluster.yml file:

nodes:
- address: xxxx01           # air gap node external IP
  user: rancher
  role: [ "controlplane", "etcd", "worker" ]
ignore_docker_version: true
private_registries:
- url: docker.xxx.com/infra
  user: xxx
  password: xxx
  is_default: true
services:
  etcd:
    snapshot: true
    retention: "24h"
    creation: "6h0s"

Steps to Reproduce:

1. rke(0.2.2) up to strike up a new cluster
2. remove rkestate file
3. rke up to retrieve the rkestate
4. get rke error,
   FATA[0202] [etcd] Failed to bring up Etcd Plane: [etcd] Etcd Cluster is not healthy
5. reuse the previous rkestate file, it works again.

Results:
Fail to get current rkestate file with error,
FATA[0202] [etcd] Failed to bring up Etcd Plane: [etcd] Etcd Cluster is not healthy
Seems the certs are regenerated.

Expected Result:
'rke up' using the same kubeconfig should be able to get the previous rkestate.

[guidoilbaldo]

Hi @garyhuang123 ,

as you can see from my issue here -> https://github.com/rancher/rke/issues/1343
I'm getting the same error as you but I can't seem to be able to upgrade the cluster using the rkestate file created when I first run rke up.

Would you mind suggesting me how to use old rkestate to try and successfully upgrade the cluster?
Thank you

[garyhuang123]

Hi @guidoilbaldo,

Did you rke up with the right rkestate file? Make sure you have the right .rkestate file originally used to set up your cluster.

If you are missing with the .rkestate file. Below script can retrieve it back,
kubectl --kubeconfig kube_config_cluster.yml get configmap -n kube-system full-cluster-state -o json | jq -r .data.\"full-cluster-state\" | jq -r . > rancher_cluster.rkestate

[guidoilbaldo]

Ok but how can I pass it to rke?

[garyhuang123]

@guidoilbaldo, place the rancher_cluster.rkestate file under the work directory where you run rke up. So does the kube_config_cluster.yml BTW.

[mung720]

I have experenced this bug too on v0.2.4 at this point it seems necessary to backup your state file.

[stale[bot]]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.