Kubespray: fail at etcd : Install | Copy etcdctl binary from docker container

[LuckySB]

Environment:

• OS: centos 7

• Version of Ansible: 2.7.12

• Version of Python: 2.7

Kubespray version (commit) (git rev-parse --short HEAD): master

Network plugin used: flannel

inventory vars:

container_manager: containerd
etcd_deployment_type: docker

Output of ansible run:

included: /srv/kubespray/roles/etcd/tasks/install_docker.yml for master-1.s000005.slurm.io, master-2.s000005.slurm.io, master-3.s000005.slurm.io

TASK [etcd : Install | Copy etcdctl binary from docker container] ********************************************************************************************
Thursday 05 March 2020  12:15:05 +0000 (0:00:00.234)       0:10:40.110 ********
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (4 retries left).
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (4 retries left).
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (4 retries left).
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (3 retries left).
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (3 retries left).
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (3 retries left).
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (2 retries left).
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (2 retries left).
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (2 retries left).
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (1 retries left).
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (1 retries left).
FAILED - RETRYING: Install | Copy etcdctl binary from docker container (1 retries left).
fatal: [master-1.s000005.slurm.io]: FAILED! => {"attempts": 4, "changed": false, "cmd": ["sh", "-c", "/usr/bin/docker rm -f etcdctl-binarycopy; /usr/bin/docker create --name etcdctl-binarycopy quay.io/coreos/etcd:v3.3.10 && /usr/bin/docker cp etcdctl-binarycopy:/usr/local/bin/etcdctl /usr/local/bin/etcdctl && /usr/bin/docker rm -f etcdctl-binarycopy"], "delta": "0:00:00.038695", "end": "2020-03-05 12:15:34.687655", "msg": "non-zero return code", "rc": 127, "start": "2020-03-05 12:15:34.648960", "stderr": "sh: /usr/bin/docker: No such file or directory\nsh: /usr/bin/docker: No such file or directory", "stderr_lines": ["sh: /usr/bin/docker: No such file or directory", "sh: /usr/bin/docker: No such file or directory"], "stdout": "", "stdout_lines": []}

Anything else do we need to know:
no docker binary installed because use containerd

[LuckySB]

/assign @mattymo

[LuckySB]

use etcd_kubeadm_enabled: true or etcd_deployment_type: host

[EppO]

docker cp is a docker-only feature (as it's part of docker build code-base, it's not in containerd).
You can achieve the same thing in containerd but in a way more complicated way, and cri-o uses the same kind of hack using podman mount.
I think the easiest approach would be to create an alias on the master nodes that uses crictl exec on the etcd container with the correct arguments (etcd certs) and the user arguments on the alias command. If we're all ok with that approach, I will work on a PR

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[EppO]

/remove-lifecycle stale

6151 is trying to address the issue

[msteenhu]

With current master I had to change this to get kubespray containerd past this error.

--- a/roles/etcdctl/tasks/main.yml
+++ b/roles/etcdctl/tasks/main.yml
@@ -36,7 +36,7 @@
     when: container_manager ==  "docker"

   - name: Copy etcdctl script to host
-    shell: "crictl exec \"$(crictl ps -q --image {{ etcd_image_repo }}:{{ etcd_image_tag }})\" cp /usr/local/bin/etcdctl {{ etcd_data_dir }}"
+    shell: "crictl exec \"$(crictl ps -q --name etcd)\" cp /usr/local/bin/etcdctl {{ etcd_data_dir }}"
     when: container_manager in ['crio', 'containerd']

   - name: Copy etcdctl to {{ bin_dir }}

[anthr76]

@msteenhu

Instead you can do crictl exec \"$(crictl ps -q --image {{ etcd_image_repo }}:{{ etcd_image_tag }})\" cp /usr/local/bin/etcdctl {{ etcd_data_dir }}/etcdctl