Kubespray: Create kubeadm token for joining nodes with 24h expiration - Fails
Environment:
On-Prem using Centos 7 VMs
- OS (
printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
- Version of Ansible (
ansible --version):
ansible 2.7.10
config file = None
configured module search path = [u'/home/hayashid/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Apr 9 2019, 14:30:50) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
Kubespray version (commit) (git rev-parse --short HEAD):
09fe95b
Network plugin used:
calico
Copy of your inventory file:
## configure a bastion host if your nodes are not directly reachable
bastion ansible_host=x.x.x.x ansible_user=some_user
[kube-master]
cvc-prod-nycdc-gcs01.air2gnd.net
cvc-prod-nycdc-gcs02.air2gnd.net
cvc-prod-nycdc-gcs03.air2gnd.net
[etcd]
cvc-prod-nycdc-gcs01.air2gnd.net
cvc-prod-nycdc-gcs02.air2gnd.net
cvc-prod-nycdc-gcs03.air2gnd.net
[kube-node]
cvn-prod-nycdc-gcs01.air2gnd.net
cvn-prod-nycdc-gcs02.air2gnd.net
cvn-prod-nycdc-gcs03.air2gnd.net
cvn-prod-nycdc-gcs04.air2gnd.net
cvn-prod-nycdc-gcs05.air2gnd.net
cvn-prod-nycdc-gcs06.air2gnd.net
[k8s-cluster:children]
kube-master
kube-node
Command used to invoke ansible:
ansible-playbook -i inventory/mycluster/hosts.ini cluster.yml -b -v --private-key=~/.ssh/id_rsa --become --become-user=root -vvv
Output of ansible run:
TASK [kubernetes/master : kubeadm | Initialize first master] *********************************************************
[1;30mtask path: /home/hayashid/code/kubespray/roles/kubernetes/master/tasks/kubeadm-setup.yml:91[0m
Monday 22 April 2019 18:45:36 +0000 (0:00:05.400) 0:09:01.573 ***
TASK [kubernetes/master : set kubeadm certificate key] ***********************************************************
[1;30mtask path: /home/hayashid/code/kubespray/roles/kubernetes/master/tasks/kubeadm-setup.yml:114[0m
Monday 22 April 2019 18:45:36 +0000 (0:00:00.133) 0:09:01.706 ***
TASK [kubernetes/master : Create kubeadm token for joining nodes with 24h expiration (default)] *********************************************
[1;30mtask path: /home/hayashid/code/kubespray/roles/kubernetes/master/tasks/kubeadm-setup.yml:124[0m
Monday 22 April 2019 18:45:37 +0000 (0:00:00.183) 0:09:01.890 **
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 2
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 4
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (5 retries left).Result was: {[0m
[1;30m "attempts": 1, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.027341", [0m
[1;30m "end": "2019-04-22 18:46:51.675574", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:45:36.648233", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (5 retries left).Result was: {[0m
[1;30m "attempts": 1, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.027818", [0m
[1;30m "end": "2019-04-22 18:46:51.677426", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:45:36.649608", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 6
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (5 retries left).Result was: {[0m
[1;30m "attempts": 1, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.023172", [0m
[1;30m "end": "2019-04-22 18:46:51.709866", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:45:36.686694", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34m
mote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 29271\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n')[0m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 4
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 3
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (4 retries left).Result was: {[0m
[1;30m "attempts": 2, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.028647", [0m
[1;30m "end": "2019-04-22 18:48:12.066067", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:46:57.037420", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (4 retries left).Result was: {[0m
[1;30m "attempts": 2, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.028052", [0m
[1;30m "end": "2019-04-22 18:48:12.066525", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:46:57.038473", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 6
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (4 retries left).Result was: {[0m
[1;30m "attempts": 2, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.025062", [0m
[1;30m "end": "2019-04-22 18:48:12.092094", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:46:57.067032", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 4
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 2
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (3 retries left).Result was: {[0m
[1;30m "attempts": 3, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.028209", [0m
[1;30m "end": "2019-04-22 18:49:32.455399", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:48:17.427190", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (3 retries left).Result was: {[0m
[1;30m "attempts": 3, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.030392", [0m
[1;30m "end": "2019-04-22 18:49:32.457584", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:48:17.427192", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 6
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (3 retries left).Result was: {[0m
[1;30m "attempts": 3, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.023233", [0m
[1;30m "end": "2019-04-22 18:49:32.478790", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:48:17.455557", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 2
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 4
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (2 retries left).Result was: {[0m
[1;30m "attempts": 4, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.028424", [0m
[1;30m "end": "2019-04-22 18:50:52.847516", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:49:37.819092", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (2 retries left).Result was: {[0m
[1;30m "attempts": 4, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.031326", [0m
[1;30m "end": "2019-04-22 18:50:52.850352", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:49:37.819026", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 6
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (2 retries left).Result was: {[0m
[1;30m "attempts": 4, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.024572", [0m
[1;30m "end": "2019-04-22 18:50:52.878642", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:49:37.854070", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 4
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 3
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (1 retries left).Result was: {[0m
[1;30m "attempts": 5, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.024972", [0m
[1;30m "end": "2019-04-22 18:52:13.240615", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:50:58.215643", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (1 retries left).Result was: {[0m
[1;30m "attempts": 5, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.031436", [0m
[1;30m "end": "2019-04-22 18:52:13.242397", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:50:58.210961", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 6
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[1;30mFAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (1 retries left).Result was: {[0m
[1;30m "attempts": 5, [0m
[1;30m "changed": true, [0m
[1;30m "cmd": [[0m
[1;30m "/usr/local/bin/kubeadm", [0m
[1;30m "--kubeconfig", [0m
[1;30m "/etc/kubernetes/admin.conf", [0m
[1;30m "token", [0m
[1;30m "create"[0m
[1;30m ], [0m
[1;30m "delta": "0:01:15.025239", [0m
[1;30m "end": "2019-04-22 18:52:13.266230", [0m
[1;30m "invocation": {[0m
[1;30m "module_args": {[0m
[1;30m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[1;30m "_uses_shell": false, [0m
[1;30m "argv": null, [0m
[1;30m "chdir": null, [0m
[1;30m "creates": null, [0m
[1;30m "executable": null, [0m
[1;30m "removes": null, [0m
[1;30m "stdin": null, [0m
[1;30m "warn": true[0m
[1;30m }[0m
[1;30m }, [0m
[1;30m "msg": "non-zero return code", [0m
[1;30m "rc": 1, [0m
[1;30m "retries": 6, [0m
[1;30m "start": "2019-04-22 18:50:58.240991", [0m
[1;30m "stderr": "timed out waiting for the condition", [0m
[1;30m "stderr_lines": [[0m
[1;30m "timed out waiting for the condition"[0m
[1;30m ], [0m
[1;30m "stdout": "", [0m
[1;30m "stdout_lines": [][0m
[1;30m}[0m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34m
[0;34mUsing module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py[0m
[0;34m
[0;34m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34mEscalation succeeded[0m
[0;34m
mote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 29271\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n')[0m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 2
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 4
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[0;31mfatal: [cvc-prod-nycdc-gcs01.air2gnd.net -> cvc-prod-nycdc-gcs01.air2gnd.net]: FAILED! => {[0m
[0;31m "attempts": 5, [0m
[0;31m "changed": true, [0m
[0;31m "cmd": [[0m
[0;31m "/usr/local/bin/kubeadm", [0m
[0;31m "--kubeconfig", [0m
[0;31m "/etc/kubernetes/admin.conf", [0m
[0;31m "token", [0m
[0;31m "create"[0m
[0;31m ], [0m
[0;31m "delta": "0:01:15.029431", [0m
[0;31m "end": "2019-04-22 18:53:33.634319", [0m
[0;31m "invocation": {[0m
[0;31m "module_args": {[0m
[0;31m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[0;31m "_uses_shell": false, [0m
[0;31m "argv": null, [0m
[0;31m "chdir": null, [0m
[0;31m "creates": null, [0m
[0;31m "executable": null, [0m
[0;31m "removes": null, [0m
[0;31m "stdin": null, [0m
[0;31m "warn": true[0m
[0;31m }[0m
[0;31m }, [0m
[0;31m "msg": "non-zero return code", [0m
[0;31m "rc": 1, [0m
[0;31m "start": "2019-04-22 18:52:18.604888", [0m
[0;31m "stderr": "timed out waiting for the condition", [0m
[0;31m "stderr_lines": [[0m
[0;31m "timed out waiting for the condition"[0m
[0;31m ], [0m
[0;31m "stdout": "", [0m
[0;31m "stdout_lines": [][0m
[0;31m}[0m
[0;31mfatal: [cvc-prod-nycdc-gcs02.air2gnd.net -> cvc-prod-nycdc-gcs01.air2gnd.net]: FAILED! => {[0m
[0;31m "attempts": 5, [0m
[0;31m "changed": true, [0m
[0;31m "cmd": [[0m
[0;31m "/usr/local/bin/kubeadm", [0m
[0;31m "--kubeconfig", [0m
[0;31m "/etc/kubernetes/admin.conf", [0m
[0;31m "token", [0m
[0;31m "create"[0m
[0;31m ], [0m
[0;31m "delta": "0:01:15.030900", [0m
[0;31m "end": "2019-04-22 18:53:33.635792", [0m
[0;31m "invocation": {[0m
[0;31m "module_args": {[0m
[0;31m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[0;31m "_uses_shell": false, [0m
[0;31m "argv": null, [0m
[0;31m "chdir": null, [0m
[0;31m "creates": null, [0m
[0;31m "executable": null, [0m
[0;31m "removes": null, [0m
[0;31m "stdin": null, [0m
[0;31m "warn": true[0m
[0;31m }[0m
[0;31m }, [0m
[0;31m "msg": "non-zero return code", [0m
[0;31m "rc": 1, [0m
[0;31m "start": "2019-04-22 18:52:18.604892", [0m
[0;31m "stderr": "timed out waiting for the condition", [0m
[0;31m "stderr_lines": [[0m
[0;31m "timed out waiting for the condition"[0m
[0;31m ], [0m
[0;31m "stdout": "", [0m
[0;31m "stdout_lines": [][0m
[0;31m}[0m
[0;34m
[0;34m
[0m
[0;34mdebug1: Reading configuration data /etc/ssh/ssh_config
[0m
[0;34mdebug1: /etc/ssh/ssh_config line 58: Applying options for *
[0m
[0;34mdebug1: auto-mux: Trying existing master
[0m
[0;34mdebug2: fd 3 setting O_NONBLOCK
[0m
[0;34mdebug2: mux_client_hello_exchange: master version 4
[0m
[0;34mdebug3: mux_client_forwards: request forwardings: 0 local, 0 remote
[0m
[0;34mdebug3: mux_client_request_session: entering
[0m
[0;34mdebug3: mux_client_request_alive: entering
[0m
[0;34mdebug3: mux_client_request_alive: done pid = 29271
[0m
[0;34mdebug3: mux_client_request_session: session request sent
[0m
[0;34mdebug1: mux_client_request_session: master session id: 6
[0m
[0;34mdebug3: mux_client_read_packet: read header failed: Broken pipe
[0m
[0;34mdebug2: Received exit status from master 1[0m
[0;31mfatal: [cvc-prod-nycdc-gcs03.air2gnd.net -> cvc-prod-nycdc-gcs01.air2gnd.net]: FAILED! => {[0m
[0;31m "attempts": 5, [0m
[0;31m "changed": true, [0m
[0;31m "cmd": [[0m
[0;31m "/usr/local/bin/kubeadm", [0m
[0;31m "--kubeconfig", [0m
[0;31m "/etc/kubernetes/admin.conf", [0m
[0;31m "token", [0m
[0;31m "create"[0m
[0;31m ], [0m
[0;31m "delta": "0:01:15.024689", [0m
[0;31m "end": "2019-04-22 18:53:33.659462", [0m
[0;31m "invocation": {[0m
[0;31m "module_args": {[0m
[0;31m "_raw_params": "/usr/local/bin/kubeadm --kubeconfig /etc/kubernetes/admin.conf token create", [0m
[0;31m "_uses_shell": false, [0m
[0;31m "argv": null, [0m
[0;31m "chdir": null, [0m
[0;31m "creates": null, [0m
[0;31m "executable": null, [0m
[0;31m "removes": null, [0m
[0;31m "stdin": null, [0m
[0;31m "warn": true[0m
[0;31m }[0m
[0;31m }, [0m
[0;31m "msg": "non-zero return code", [0m
[0;31m "rc": 1, [0m
[0;31m "start": "2019-04-22 18:52:18.634773", [0m
[0;31m "stderr": "timed out waiting for the condition", [0m
[0;31m "stderr_lines": [[0m
[0;31m "timed out waiting for the condition"[0m
[0;31m ], [0m
[0;31m "stdout": "", [0m
[0;31m "stdout_lines": [][0m
[0;31m}[0m
NO MORE HOSTS LEFT ************************************************************************
to retry, use: --limit @/home/hayashid/code/kubespray/cluster.retry
PLAY RECAP **************************************************************************
[0;31mcvc-prod-nycdc-gcs01.air2gnd.net[0m : [0;32mok=310 [0m [0;33mchanged=15 [0m unreachable=0 [0;31mfailed=1 [0m
[0;31mcvc-prod-nycdc-gcs02.air2gnd.net[0m : [0;32mok=287 [0m [0;33mchanged=15 [0m unreachable=0 [0;31mfailed=1 [0m
[0;31mcvc-prod-nycdc-gcs03.air2gnd.net[0m : [0;32mok=287 [0m [0;33mchanged=15 [0m unreachable=0 [0;31mfailed=1 [0m
[0;33mcvn-prod-nycdc-gcs01.air2gnd.net[0m : [0;32mok=239 [0m [0;33mchanged=8 [0m unreachable=0 failed=0
[0;33mcvn-prod-nycdc-gcs02.air2gnd.net[0m : [0;32mok=239 [0m [0;33mchanged=8 [0m unreachable=0 failed=0
[0;33mcvn-prod-nycdc-gcs03.air2gnd.net[0m : [0;32mok=239 [0m [0;33mchanged=8 [0m unreachable=0 failed=0
[0;33mcvn-prod-nycdc-gcs04.air2gnd.net[0m : [0;32mok=239 [0m [0;33mchanged=8 [0m unreachable=0 failed=0
[0;33mcvn-prod-nycdc-gcs05.air2gnd.net[0m : [0;32mok=239 [0m [0;33mchanged=8 [0m unreachable=0 failed=0
[0;33mcvn-prod-nycdc-gcs06.air2gnd.net[0m : [0;32mok=239 [0m [0;33mchanged=8 [0m unreachable=0 failed=0
[0;32mlocalhost[0m : [0;32mok=1 [0m changed=0 unreachable=0 failed=0
Monday 22 April 2019 18:53:34 +0000 (0:07:57.493) 0:16:59.383 **
kubernetes/master : Create kubeadm token for joining nodes with 24h expiration (default) ----------------------------------------------------------------------------------------------------------------------------------------- 477.49s
/home/hayashid/code/kubespray/roles/kubernetes/master/tasks/kubeadm-setup.yml:124 --------------------------------------------------------------------------------------------------------------------------------------------------------
kubernetes/preinstall : Update package management cache (YUM) --------------------------------------------------------------------------------------------------------------------------------------------------------------------- 31.79s
/home/hayashid/code/kubespray/roles/kubernetes/preinstall/tasks/0070-system-packages.yml:2 -----------------------------------------------------------------------------------------------------------------------------------------------
bootstrap-os : Check python-pip package ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 25.07s
/home/hayashid/code/kubespray/roles/bootstrap-os/tasks/bootstrap-centos.yml:44 -----------------------------------------------------------------------------------------------------------------------------------------------------------
download : container_download | download images for kubeadm config images --------------------------------------------------------------------------------------------------------------------------------------------------------- 22.79s
/home/hayashid/code/kubespray/roles/download/tasks/kubeadm_images.yml:23 -----------------------------------------------------------------------------------------------------------------------------------------------------------------
kubernetes/node : Write cloud-config ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 8.75s
/home/hayashid/code/kubespray/roles/kubernetes/node/tasks/main.yml:201 -------------------------------------------------------------------------------------------------------------------------------------------------------------------
gather facts from all instances ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 7.63s
/home/hayashid/code/kubespray/cluster.yml:39 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------------------------------------------------------------------------------------------- 7.33s
/home/hayashid/code/kubespray/roles/download/tasks/download_container.yml:30 -------------------------------------------------------------------------------------------------------------------------------------------------------------
container-engine/docker : Write docker dns systemd drop-in ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6.99s
/home/hayashid/code/kubespray/roles/container-engine/docker/tasks/systemd.yml:34 ---------------------------------------------------------------------------------------------------------------------------------------------------------
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------------------------------------------------------------------------------------------- 6.91s
/home/hayashid/code/kubespray/roles/download/tasks/download_container.yml:30 -------------------------------------------------------------------------------------------------------------------------------------------------------------
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------------------------------------------------------------------------------------------- 6.81s
/home/hayashid/code/kubespray/roles/download/tasks/download_container.yml:30 -------------------------------------------------------------------------------------------------------------------------------------------------------------
bootstrap-os : check python-httplib2 package --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.42s
/home/hayashid/code/kubespray/roles/bootstrap-os/tasks/bootstrap-centos.yml:61 -----------------------------------------------------------------------------------------------------------------------------------------------------------
kubernetes/master : Backup old certs and keys -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.40s
/home/hayashid/code/kubespray/roles/kubernetes/master/tasks/kubeadm-certificate.yml:2 ----------------------------------------------------------------------------------------------------------------------------------------------------
kubernetes/preinstall : Hosts | populate inventory into hosts file ----------------------------------------------------------------------------------------------------------------------------------------------------------------- 5.29s
/home/hayashid/code/kubespray/roles/kubernetes/preinstall/tasks/0090-etchosts.yml:2 ------------------------------------------------------------------------------------------------------------------------------------------------------
kubernetes/preinstall : Install packages requirements ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 4.97s
/home/hayashid/code/kubespray/roles/kubernetes/preinstall/tasks/0070-system-packages.yml:88 ----------------------------------------------------------------------------------------------------------------------------------------------
kubernetes/preinstall : Create kubernetes directories ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 4.19s
/home/hayashid/code/kubespray/roles/kubernetes/preinstall/tasks/0050-create_directories.yml:2 --------------------------------------------------------------------------------------------------------------------------------------------
kubernetes/node : nginx-proxy | Write nginx-proxy configuration -------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4.02s
/home/hayashid/code/kubespray/roles/kubernetes/node/tasks/nginx-proxy.yml:19 -------------------------------------------------------------------------------------------------------------------------------------------------------------
kubernetes/node : Write kubelet config file (kubeadm) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 4.00s
/home/hayashid/code/kubespray/roles/kubernetes/node/tasks/main.yml:43 --------------------------------------------------------------------------------------------------------------------------------------------------------------------
bootstrap-os : Install libselinux-python and yum-utils for bootstrap --------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.98s
/home/hayashid/code/kubespray/roles/bootstrap-os/tasks/bootstrap-centos.yml:34 -----------------------------------------------------------------------------------------------------------------------------------------------------------
bootstrap-os : Assign inventory name to unconfigured hostnames (non-CoreOS and Tumbleweed) ----------------------------------------------------------------------------------------------------------------------------------------- 3.83s
/home/hayashid/code/kubespray/roles/bootstrap-os/tasks/main.yml:42 -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
bootstrap-os : Gather nodes hostnames ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3.74s
/home/hayashid/code/kubespray/roles/bootstrap-os/tasks/main.yml:37 -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
]0;hayashid@cvc-prod-nycdc-mgmt01:~/code/kubespray[hayashid@cvc-prod-nycdc-mgmt01 kubespray]$
Anything else do we need to know:
https://gist.github.com/hayashid/84330ca13c90775a1a19023483ee13fe
hayashid
All 21 comments
Having same issue on Ubuntu as well.
ashishxooa
on 24 Apr 2019
+1
The problem for me looks to be in missing /etc/kubernetes/admin.conf
Found the problem.
TLDR; if cloud_provider is set to openstack, you must set OS_USER_DOMAIN_NAME environment variable during deploy. Otherwise this variable wont be provided to the kube master and wont be able to start kubelet.
hp197
on 3 May 2019
I solved that problem commenting theses conditions in roles/kubernetes/kubeadm/tasks/main.yml at lines 28 and 33 :
- name: Create kubeadm token for joining nodes with 24h expiration (default)
command: "{{ bin_dir }}/kubeadm token create"
register: temp_token
delegate_to: "{{ groups['kube-master'][0] }}"
# when: kubeadm_token is not defined
- name: Set kubeadm_token to generated token
set_fact:
kubeadm_token: "{{ temp_token.stdout }}"
# when: kubeadm_token is not defined
ledroide
on 29 May 2019
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
on 27 Aug 2019
I am facing the same error trying to launch the cluster on aws.
The error log goes as below:
TASK [kubernetes/master : kubeadm | Initialize first master] ************
task path: /Users/kamalikamajumder/CloudKata/Dkatalis/k8s-cluster-setup/ansible/roles/kubernetes/master/tasks/kubeadm-setup.yml:106
Saturday 31 August 2019 21:10:12 +0700 (0:00:03.140) 0:06:32.269 *
TASK [kubernetes/master : set kubeadm certificate key] **************
task path: /Users/kamalikamajumder/CloudKata/Dkatalis/k8s-cluster-setup/ansible/roles/kubernetes/master/tasks/kubeadm-setup.yml:133
Saturday 31 August 2019 21:10:12 +0700 (0:00:00.093) 0:06:32.363 *
TASK [kubernetes/master : Create hardcoded kubeadm token for joining nodes with 24h expiration (if defined)] *
task path: /Users/kamalikamajumder/CloudKata/Dkatalis/k8s-cluster-setup/ansible/roles/kubernetes/master/tasks/kubeadm-setup.yml:142
Saturday 31 August 2019 21:10:12 +0700 (0:00:00.106) 0:06:32.470 ***
TASK [kubernetes/master : Create kubeadm token for joining nodes with 24h expiration (default)]
task path: /Users/kamalikamajumder/CloudKata/Dkatalis/k8s-cluster-setup/ansible/roles/kubernetes/master/tasks/kubeadm-setup.yml:153
Saturday 31 August 2019 21:10:12 +0700 (0:00:00.107) 0:06:32.578 **
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (10 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (10 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (9 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (9 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (8 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (8 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (7 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (7 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (6 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (6 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (5 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (5 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (4 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (4 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (3 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (3 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (2 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (2 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (1 retries left).
FAILED - RETRYING: Create kubeadm token for joining nodes with 24h expiration (default) (1 retries left).
fatal: [ip-10-8-3-184.ap-southeast-1.compute.internal -> 10.8.3.191]: FAILED! => {"attempts": 10, "changed": true, "cmd": ["/opt/bin/kubeadm", "--kubeconfig", "/etc/kubernetes/admin.conf", "token", "create"], "delta": "0:01:15.033213", "end": "2019-08-31 14:24:52.702199", "msg": "non-zero return code", "rc": 1, "start": "2019-08-31 14:23:37.668986", "stderr": "timed out waiting for the condition", "stderr_lines": ["timed out waiting for the condition"], "stdout": "", "stdout_lines": []}
fatal: [ip-10-8-3-191.ap-southeast-1.compute.internal -> 10.8.3.191]: FAILED! => {"attempts": 10, "changed": true, "cmd": ["/opt/bin/kubeadm", "--kubeconfig", "/etc/kubernetes/admin.conf", "token", "create"], "delta": "0:01:15.033902", "end": "2019-08-31 14:24:52.702215", "msg": "non-zero return code", "rc": 1, "start": "2019-08-31 14:23:37.668313", "stderr": "timed out waiting for the condition", "stderr_lines": ["timed out waiting for the condition"], "stdout": "", "stdout_lines": []}
NO MORE HOSTS LEFT **************************
to retry, use: --limit @/Users/kamalikamajumder/CloudKata/Dkatalis/k8s-cluster-setup/ansible/cluster.retry
kamalika-majumder
on 31 Aug 2019
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
fejta-bot
on 30 Sep 2019
+1
lomori
on 5 Oct 2019
The same problem with CentOS 7 and Ansible 2.8.6
delmar74
on 23 Oct 2019
+1
digitaljedi2
on 30 Oct 2019
+1
vpohyla
on 1 Nov 2019
Same problem on coreos
davidq2q
on 4 Nov 2019
Is this sorted or still active ? It is still reproducible in latest master.
hprateek43
on 15 Nov 2019
I got the same issue using Openstack as provider on release-2.11
vblando
on 25 Nov 2019
I got same issue with using kubespray branch 2.11 and ansible 2.7.10 on CentOS 7.7.
I used configuration with 3 HA masters.
Any recommendations or workaround ?
mr-yaky
on 6 Dec 2019
@mr-yaky try change provided to external
vpohyla
on 6 Dec 2019
it didn't use OpenStack but just a regular baremetal installation.
I solved my problem with renewing all k8s certs by kubeadm and also /etc/kubernetes/admin.conf as well.
mr-yaky
on 6 Dec 2019
just run _reset.yml_
ansible-playbook --flush-cache -i inventory/mycluster/hosts.yml reset.yml --become -u root
and then _cluster.yml_
ansible-playbook --flush-cache -i inventory/mycluster/hosts.yml cluster.yml --become -u root
jborkowski
on 18 Dec 2019
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
fejta-bot
on 17 Jan 2020
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen.
Mark the issue as fresh with/remove-lifecycle rotten.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
k8s-ci-robot
on 17 Jan 2020
@mr-yaky :
I solved my problem with renewing all k8s certs by kubeadm and also
/etc/kubernetes/admin.confas well.
May you please detail operations you had run when solving this issue ?
ledroide
on 4 Sep 2020
Had similar issue running ansible-playbook with -u myuser --become. The problem is that kubeadm wants --kubeconfig specified explicitly. So, I've modified roles/kubernetes/kubeadm/tasks/main.yml like this:
24 │ - name: Create kubeadm token for joining nodes with 24h expiration (default)
25 ~ │ command: "{{ bin_dir }}/kubeadm token create --kubeconfig /etc/kubernetes/admin.conf"
26 │ register: temp_token
27 │ delegate_to: "{{ groups['kube-master'][0] }}"
28 │ when: kubeadm_token is not defined
and re-run ansible-playbook. It solved my issue.
pavdmyt
on 11 Nov 2020
Related issues
danielm0hr
·
4Comments
sermilrod
·
4Comments
mattdornfeld
·
4Comments
hellwen
·
4Comments
mjlshen
·
3Comments
Most helpful comment
I got same issue with using kubespray branch 2.11 and ansible 2.7.10 on CentOS 7.7.
I used configuration with 3 HA masters.
Any recommendations or workaround ?