Kubespray: vagrant up fails (rsync issue)

[Trefex]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG

When i run vagrant up, near the end the command fails with some weird rsync error.

I am not sure how to proceed.

Thanks,
T

Environment:

• Cloud provider or hardware configuration: vagrant

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"): macOS

• Version of Ansible (ansible --version): 2.7.2

Kubespray version (commit) (git rev-parse --short HEAD): 0cfcd39d

Network plugin used: default

Copy of your inventory file:

(openstack-tools) kubespray git:master ❯ cat inventory/mycluster/hosts.ini                                                                                         ✹ ✭
[all]
node1    ansible_host=10.10.1.3 ip=10.10.1.3
node2    ansible_host=10.10.1.4 ip=10.10.1.4
node3    ansible_host=10.10.1.5 ip=10.10.1.5

[kube-master]
node1
node2

[etcd]
node1
node2
node3

[kube-node]
node1
node2
node3

[k8s-cluster:children]
kube-master
kube-node

[calico-rr]

[vault]
node1
node2
node3

Command used to invoke ansible:
vagrant up

Output of ansible run:

```TASK [download : container_download | copy container images to ansible host] *
Thursday 22 November 2018 15:45:31 +0100 (0:00:03.759) 0:02:04.876 *
fatal: [k8s-1 -> localhost]: FAILED! => {"changed": false, "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --rsh=/usr/bin/ssh -S none -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null --out-format=<>%i %n%L /tmp/releases/containers/nginx:1.13.tar /tmp/releases/containers/nginx:1.13.tar", "msg": "rsync: link_stat \"/tmp/releases/containers/nginx:1.13.tar\" failed: No such file or directory (2)\nrsync: push_dir#3 \"/tmp/releases/containers\" failed: No such file or directory (2)\nrsync error: errors selecting input/output files, dirs (code 3) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-52.200.1/rsync/main.c(581) [receiver=2.6.9]\nrsync: connection unexpectedly closed (8 bytes received so far) [sender]\nrsync error: error in rsync protocol data stream (code 12) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-52.200.1/rsync/io.c(453) [sender=2.6.9]\n", "rc": 12}

NO MORE HOSTS LEFT *******************
to retry, use: --limit @/Users/christophe.trefois/dev/test/kubespray/cluster.retry

PLAY RECAP ***********************
k8s-1 : ok=203 changed=11 unreachable=0 failed=1
k8s-2 : ok=125 changed=7 unreachable=0 failed=0
k8s-3 : ok=124 changed=7 unreachable=0 failed=0
```

[ant31]

cc @jhalfmoon

[easel]

Seeing something vaguely similar running under libvirt on CentOS7:

fatal: [k8s-1 -> localhost]: FAILED! => {"changed": false, "cmd": "/bin/rsync --delay-updates -F --compress --archive --rsh=/bin/ssh -S none -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null --out-format=<<CHANGED>>%i %n%L [email protected]:/tmp/releases/containers/quay.io_coreos_etcd:v3.2.24.tar /tmp/releases/containers/quay.io_coreos_etcd:v3.2.24.tar", "msg": "Warning: Permanently added '192.168.121.180' (ECDSA) to the list of known hosts.\r\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic).\r\nrsync: connection unexpectedly closed (0 bytes received so far) [Receiver]\nrsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.2]\n", "rc": 255}

[Romern]

I'm also encountering something similar:

TASK [download : container_download | copy container images to ansible host] ***
Thursday 29 November 2018  15:50:32 +0000 (0:00:01.735)       0:04:29.884 ***** 
fatal: [k8s-1 -> localhost]: FAILED! => {"changed": false, "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --rsh=/usr/bin/ssh -S none -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null --out-format=<<CHANGED>>%i %n%L /tmp/releases/containers/gcr.io_google_containers_cluster-proportional-autoscaler-amd64:1.3.0.tar /tmp/releases/containers/gcr.io_google_containers_cluster-proportional-autoscaler-amd64:1.3.0.tar", "msg": "rsync: change_dir \"/tmp/releases/containers\" failed: No such file or directory (2)\nrsync: change_dir#3 \"/tmp/releases/containers\" failed: No such file or directory (2)\nrsync error: errors selecting input/output files, dirs (code 3) at main.c(713) [Receiver=3.1.2]\n", "rc": 3}                  

NO MORE HOSTS LEFT *************************************************************
        to retry, use: --limit @/ayy/kubespray/cluster.retry

PLAY RECAP *********************************************************************
k8s-1                      : ok=219  changed=42   unreachable=0    failed=1   
k8s-2                      : ok=132  changed=29   unreachable=0    failed=0   
k8s-3                      : ok=128  changed=29   unreachable=0    failed=0

[chenyg0911]

same problem encounted. I use vagrant too.
FAILED - RETRYING: container_download | upload container images to nodes (1 retries left). fatal: [k8s-3 -> localhost]: FAILED! => {"attempts": 4, "changed": false, "cmd": "/usr/bin/rsync --delay-updates -F --compress --archive --rsh=/usr/bin/ssh -S none -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null --out-format=<<CHANGED>>%i %n%L /tmp/releases/containers/registry.mycompany.com_kube_etcd:v3.2.24.tar /tmp/releases/containers/registry.mycompany.com_kube_etcd:v3.2.24.tar", "msg": "rsync: change_dir \"/tmp/releases/containers\" failed: No such file or directory (2)\nrsync: change_dir#3 \"/tmp/releases/containers\" failed: No such file or directory (2)\nrsync error: errors selecting input/output files, dirs (code 3) at main.c(713) [Receiver=3.1.2]\n", "rc": 3}

[chenyg0911]

update Vagrantfile to disable local rsync. Howerver, It works for every node download images from registry.
"download_run_once": "False",

[saedabdu]

Confirmed. Setting "download_run_once": "False" in Vagrantfile fixes the problem.

[iwankgb]

See #4218 - it seems to be another way to fix the problem.

[atrakic]

Why is this not updated on master? Pls update.

[Miouge1]

/bug
/help

[k8s-ci-robot]

@Miouge1:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/bug
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Miouge1]

/kind bug
/help

[QwertyJack]

Issue still exists with v2.11.0

[rakeysh17]

I am also facing the same issue with Vagrant v2.2.5.
Even "download_run_once": "False" does not work. Please help.

[chrisjowen]

+1 same issue