RKE version:
rke version v1.1.6
Docker version: (docker version,docker info preferred)
Client:
Version: 19.03.12
API version: 1.40
Go version: go1.13.14
Git commit: 48a66213fe1747e8873f849862ff3fb981899fc6
Built: Fri Jul 24 11:43:16 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.12
API version: 1.40 (minimum version 1.12)
Go version: go1.13.14
Git commit: 48a66213fe1747e8873f849862ff3fb981899fc6
Built: Fri Jul 24 11:38:39 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.3.7
GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit:
Operating system and kernel: (cat /etc/os-release, uname -r preferred)
alpine linux 5.4.43-1-virt
Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
cluster.yml file:
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: 80.0.0.55
port: "22"
internal_address: ""
role:
- controlplane
- etcd
hostname_override: ""
user: root
docker_socket: /var/run/docker.sock
ssh_key: ""
ssh_key_path: ~/.ssh/id_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
taints: []
- address: 80.0.0.54
port: "22"
internal_address: ""
role:
- worker
- etcd
hostname_override: ""
user: root
docker_socket: /var/run/docker.sock
ssh_key: "admin"
ssh_key_path: ~/.ssh/id_rsa
ssh_cert: ""
ssh_cert_path: ""
labels: {}
taints: []
services:
etcd:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
external_urls: []
ca_cert: ""
cert: ""
key: ""
path: ""
uid: 0
gid: 0
snapshot: null
retention: ""
creation: ""
backup_config: null
kube-api:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
service_cluster_ip_range: 10.43.0.0/16
service_node_port_range: ""
pod_security_policy: false
always_pull_images: false
secrets_encryption_config: null
audit_log: null
admission_configuration: null
event_rate_limit: null
kube-controller:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
cluster_cidr: 10.42.0.0/16
service_cluster_ip_range: 10.43.0.0/16
scheduler:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
kubelet:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
cluster_domain: cluster.local
infra_container_image: ""
cluster_dns_server: 10.43.0.10
fail_swap_on: false
generate_serving_certificate: false
kubeproxy:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
win_extra_args: {}
win_extra_binds: []
win_extra_env: []
network:
plugin: canal
options: {}
mtu: 0
node_selector: {}
update_strategy: null
authentication:
strategy: x509
sans: []
webhook: null
addons: ""
addons_include: []
system_images:
etcd: rancher/coreos-etcd:v3.4.3-rancher1
alpine: rancher/rke-tools:v0.1.64
nginx_proxy: rancher/rke-tools:v0.1.64
cert_downloader: rancher/rke-tools:v0.1.64
kubernetes_services_sidecar: rancher/rke-tools:v0.1.64
kubedns: rancher/k8s-dns-kube-dns:1.15.2
dnsmasq: rancher/k8s-dns-dnsmasq-nanny:1.15.2
kubedns_sidecar: rancher/k8s-dns-sidecar:1.15.2
kubedns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
coredns: rancher/coredns-coredns:1.6.9
coredns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
nodelocal: rancher/k8s-dns-node-cache:1.15.7
kubernetes: rancher/hyperkube:v1.18.6-rancher1
flannel: rancher/coreos-flannel:v0.12.0
flannel_cni: rancher/flannel-cni:v0.3.0-rancher6
calico_node: rancher/calico-node:v3.13.4
calico_cni: rancher/calico-cni:v3.13.4
calico_controllers: rancher/calico-kube-controllers:v3.13.4
calico_ctl: rancher/calico-ctl:v3.13.4
calico_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.4
canal_node: rancher/calico-node:v3.13.4
canal_cni: rancher/calico-cni:v3.13.4
canal_flannel: rancher/coreos-flannel:v0.12.0
canal_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.4
weave_node: weaveworks/weave-kube:2.6.4
weave_cni: weaveworks/weave-npc:2.6.4
pod_infra_container: rancher/pause:3.1
ingress: rancher/nginx-ingress-controller:nginx-0.32.0-rancher1
ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.5-rancher1
metrics_server: rancher/metrics-server:v0.3.6
windows_pod_infra_container: rancher/kubelet-pause:v0.1.4
ssh_key_path: ~/.ssh/id_rsa
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
mode: rbac
options: {}
ignore_docker_version: null
kubernetes_version: ""
private_registries: []
ingress:
provider: ""
options: {}
node_selector: {}
extra_args: {}
dns_policy: ""
extra_envs: []
extra_volumes: []
extra_volume_mounts: []
update_strategy: null
cluster_name: ""
cloud_provider:
name: ""
prefix_path: ""
win_prefix_path: ""
addon_job_timeout: 0
bastion_host:
address: ""
port: ""
user: ""
ssh_key: ""
ssh_key_path: ""
ssh_cert: ""
ssh_cert_path: ""
monitoring:
provider: ""
options: {}
node_selector: {}
update_strategy: null
replicas: null
restore:
restore: false
snapshot_name: ""
dns: null
Steps to Reproduce:
just a test, and i have copied the privated key id_rsa and the public key id_rsa.pub, but still got errors
Results:
ssh test ok:
ssh -i /root/.ssh/id_rsa [email protected] docker version
ssh -i /root/.ssh/id_rsa [email protected] docker version
but run rke up got errors:
WARN[0000] Failed to set up SSH tunneling for host [80.0.0.55]: Can't retrieve Docker Info: error during connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info: Failed to dial ssh using address [80.0.0.55:22]: Error configuring SSH: ssh: this private key is passphrase protected
INFO[0000] [dialer] Setup tunnel for host [80.0.0.54]
WARN[0000] Failed to set up SSH tunneling for host [80.0.0.54]: Can't retrieve Docker Info: error during connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info: Failed to dial ssh using address [80.0.0.54:22]: Error configuring SSH: ssh: this private key is passphrase protected
and the key is not protectd like this:
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAACmFlczI1Ni1jdHIAAAAGYmNyeXB0AAAAGAAAABBIxpKIEf
cSuZwvtpbajwcGAAAAEAAAAAEAAAGXAAAAB3NzaC1yc2EAAAADAQABAAABgQCSCBHe8p+g
1MefrFLFucxl9nQi50l7XEcNY1oookWIW8wDNUuWYSZ48M52ipyMmSrcN7n4bpT8WRy8d0
9+h979Eeuh1ZrY3JB6OjimEjb6YFp6ONDQS6BZFBS7C8Eg+jnwETZxoky7c1HMPkF5S9nu
MfVvA/qLyslnBOpiHSdZP4cgegQshESioij4IDhX2uCIoLxqpYT
...
z1hW+x40ikemrz/MhGA5px/tBy8=
-----END OPENSSH PRIVATE KEY-----
The OPENSSH format does not show it is password protected, the key file can be password protected in this case (and is indicated by the code). This can be easily verified by creating a new key file without password and see if that passes. If you still think this is an issue in the code, please provide the steps to reproduce (please include the commands you used to create the key, what you placed where so we can use the same to reproduce)
Same here, passphrase is added to ssh-agent. Shouldn't rke be able to use ssh-agent ?
➜ cat cluster.yml
nodes:
- address: 162.38.60.201
user: root
role:
- controlplane
- etcd
- address: 162.38.60.202
user: root
role:
- controlplane
- etcd
- address: 162.38.60.203
user: root
role:
- controlplane
- etcd
- address: 162.38.60.204
user: root
role:
- worker
- address: 162.38.60.205
user: root
role:
- worker
- address: 162.38.60.206
user: root
role:
- worker
➜ ssh [email protected] docker version 22:26:47
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:52 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.13
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:01:20 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.7
GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
➜ ./rke up
INFO[0000] Running RKE version: v1.2.1
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0000] [certificates] Generating Kubernetes API server certificates
INFO[0000] [certificates] Generating admin certificates and kubeconfig
INFO[0000] [certificates] Generating kube-etcd-162-38-60-201 certificate and key
INFO[0000] [certificates] Generating kube-etcd-162-38-60-202 certificate and key
INFO[0000] [certificates] Generating kube-etcd-162-38-60-203 certificate and key
INFO[0000] Successfully Deployed state file at [./cluster.rkestate]
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [162.38.60.206]
INFO[0000] [dialer] Setup tunnel for host [162.38.60.203]
INFO[0000] [dialer] Setup tunnel for host [162.38.60.204]
INFO[0000] [dialer] Setup tunnel for host [162.38.60.201]
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.206]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Failed to dial ssh using address [162.38.60.206:22]: Error configuring SSH: ssh: this private key is passphrase protected
INFO[0000] [dialer] Setup tunnel for host [162.38.60.202]
INFO[0000] [dialer] Setup tunnel for host [162.38.60.205]
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.201]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Failed to dial ssh using address [162.38.60.201:22]: Error configuring SSH: ssh: this private key is passphrase protected
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.204]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Failed to dial ssh using address [162.38.60.204:22]: Error configuring SSH: ssh: this private key is passphrase protected
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.202]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Failed to dial ssh using address [162.38.60.202:22]: Error configuring SSH: ssh: this private key is passphrase protected
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.205]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Failed to dial ssh using address [162.38.60.205:22]: Error configuring SSH: ssh: this private key is passphrase protected
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.203]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Failed to dial ssh using address [162.38.60.203:22]: Error configuring SSH: ssh: this private key is passphrase protected
WARN[0000] Removing host [162.38.60.206] from node lists
WARN[0000] Removing host [162.38.60.201] from node lists
WARN[0000] Removing host [162.38.60.204] from node lists
WARN[0000] Removing host [162.38.60.202] from node lists
WARN[0000] Removing host [162.38.60.205] from node lists
WARN[0000] Removing host [162.38.60.203] from node lists
FATA[0000] Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) []
➜ ./rke --version
rke version v1.2.1
Generating a specific key without passphrase with
ssh-keygen -t rsa -f id_rsa_rke -q -N ""
ssh-copy-id -i id_rsa_rke.pub [email protected]
# and five others
Then
➜ ssh -i id_rsa_rke [email protected] docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
➜ cat cluster.yml
nodes:
- address: 162.38.60.201
user: root
role:
- controlplane
- etcd
ssh_key_path: /home/user/dev/infra-kubernetes/ansible/rke/id_rsa_rke
- address: 162.38.60.202
user: root
role:
- controlplane
- etcd
ssh_key_path: /home/user/dev/infra-kubernetes/ansible/rke/id_rsa_rke
- address: 162.38.60.203
user: root
role:
- controlplane
- etcd
ssh_key_path: /home/user/dev/infra-kubernetes/ansible/rke/id_rsa_rke
- address: 162.38.60.204
user: root
role:
- worker
ssh_key_path: /home/user/dev/infra-kubernetes/ansible/rke/id_rsa_rke
- address: 162.38.60.205
user: root
role:
- worker
ssh_key_path: /home/user/dev/infra-kubernetes/ansible/rke/id_rsa_rke
- address: 162.38.60.206
user: root
role:
- worker
ssh_key_path: /home/user/dev/infra-kubernetes/ansible/rke/id_rsa_rke
➜ file /home/user/dev/infra-kubernetes/ansible/rke/id_rsa_rke
/home/user/dev/infra-kubernetes/ansible/rke/id_rsa_rke: OpenSSH private key
➜ ./rke up
INFO[0000] Running RKE version: v1.2.1
INFO[0000] Initiating Kubernetes cluster
INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates
INFO[0000] [certificates] Generating admin certificates and kubeconfig
INFO[0000] Successfully Deployed state file at [./cluster.rkestate]
INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [162.38.60.201]
INFO[0000] [dialer] Setup tunnel for host [162.38.60.206]
INFO[0000] [dialer] Setup tunnel for host [162.38.60.205]
INFO[0000] [dialer] Setup tunnel for host [162.38.60.203]
INFO[0000] [dialer] Setup tunnel for host [162.38.60.204]
INFO[0000] [dialer] Setup tunnel for host [162.38.60.202]
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.206]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Unable to access node with address [162.38.60.206:22] using SSH. Please check if you are able to SSH to the node using the specified SSH Private Key and if you have configured the correct SSH username. Error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.203]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Unable to access node with address [162.38.60.203:22] using SSH. Please check if you are able to SSH to the node using the specified SSH Private Key and if you have configured the correct SSH username. Error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.204]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Unable to access node with address [162.38.60.204:22] using SSH. Please check if you are able to SSH to the node using the specified SSH Private Key and if you have configured the correct SSH username. Error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.202]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Unable to access node with address [162.38.60.202:22] using SSH. Please check if you are able to SSH to the node using the specified SSH Private Key and if you have configured the correct SSH username. Error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.205]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Unable to access node with address [162.38.60.205:22] using SSH. Please check if you are able to SSH to the node using the specified SSH Private Key and if you have configured the correct SSH username. Error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
WARN[0000] Failed to set up SSH tunneling for host [162.38.60.201]: Can't retrieve Docker Info: error during connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Unable to access node with address [162.38.60.201:22] using SSH. Please check if you are able to SSH to the node using the specified SSH Private Key and if you have configured the correct SSH username. Error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
WARN[0000] Removing host [162.38.60.206] from node lists
WARN[0000] Removing host [162.38.60.203] from node lists
WARN[0000] Removing host [162.38.60.204] from node lists
WARN[0000] Removing host [162.38.60.202] from node lists
WARN[0000] Removing host [162.38.60.205] from node lists
WARN[0000] Removing host [162.38.60.201] from node lists
FATA[0000] Cluster must have at least one etcd plane host: failed to connect to the following etcd host(s) []
I just fixed the passphrase problem by adding ssh_agent_auth: true to my cluster.yml template.
nodes:
- address: 162.38.60.201
user: root
role:
- controlplane
- etcd
- address: 162.38.60.202
user: root
role:
- controlplane
- etcd
- address: 162.38.60.203
user: root
role:
- controlplane
- etcd
- address: 162.38.60.204
user: root
role:
- worker
- address: 162.38.60.205
user: root
role:
- worker
- address: 162.38.60.206
user: root
role:
- worker
ssh_agent_auth: true
This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.
This is a sort of funky problem for newcomers to RKE. It might be better to probe for the need to use agent auth within RKE rather than to expect users to notice the default of ssh_agent_auth in their yaml file. This tripped me up because rke config didn't cover this option.
Agreed. It's not at all clear. The suggestion by @kfiresmith is a good one.
The explanation is in this commit https://github.com/rancher/rke/commit/ad0bc6c0aa4b0e8fc623b0739b01646dab949c0f.
There will always be 2 sides of this, one expects to automatically pickup the environment variable (which may not work or is not expected), the other expects to specifically configure it. I will link https://github.com/rancher/rke/issues/2149 so this gets taken into consideration when refactoring rke config.
Most helpful comment
I just fixed the passphrase problem by adding
ssh_agent_auth: trueto mycluster.ymltemplate.