Kubespray: CoreOS 1800.4.0 breaks Kubespray
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT
Environment:
- Cloud provider or hardware configuration: any
OS (
printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
Linux 4.14.55-coreos x86_64
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1800.4.0
VERSION_ID=1800.4.0
BUILD_ID=2018-07-25-0020
PRETTY_NAME="Container Linux by CoreOS 1800.4.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"Version of Ansible (
ansible --version):
ansible 2.6.1
Kubespray version (commit) (git rev-parse --short HEAD):
v2.5.0
Network plugin used:
contiv
Copy of your inventory file:
Command used to invoke ansible:
ansible-playbook -i inventory/verwilst/hosts.ini cluster.yml
Output of ansible run:
TASK [bootstrap-os : Bootstrap | Run bootstrap.sh] ************************************************************************************************************************************************************
Thursday 26 July 2018 11:34:31 +0200 (0:00:00.294) 0:00:04.642 *********
fatal: [k8s001-api001]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 127, "stderr": "Shared connection to 172.25.0.6 closed.\r\n", "stderr_lines": ["Shared connection to 172.25.0.6 closed."], "stdout": "--2018-07-26 11:34:31-- https://bitbucket.org/pypy/pypy/downloads/pypy-5.1.0-linux64.tar.bz2\r\nResolving bitbucket.org... 104.192.143.3, 104.192.143.1, 104.192.143.2, ...\r\nConnecting to bitbucket.org|104.192.143.3|:443... connected.\r\nHTTP request sent, awaiting response... 302 Found\r\nLocation: https://bbuseruploads.s3.amazonaws.com/54220cd1-b139-4188-9455-1e13e663f1ac/downloads/b4819e39-76fe-4453-8315-9d76fcbec5a2/pypy-5.1.0-linux64.tar.bz2?Signature=Uj0GmvzfoD1%2BpJP2zw5HenuEYzU%3D&Expires=1532598675&AWSAccessKeyId=AKIAIQWXW6WLXMB5QZAQ&versionId=rbOyQ3EO5dhbTuz.gl60O6iXfntn7QM1&response-content-disposition=attachment%3B%20filename%3D%22pypy-5.1.0-linux64.tar.bz2%22 [following]\r\n--2018-07-26 11:34:38-- https://bbuseruploads.s3.amazonaws.com/54220cd1-b139-4188-9455-1e13e663f1ac/downloads/b4819e39-76fe-4453-8315-9d76fcbec5a2/pypy-5.1.0-linux64.tar.bz2?Signature=Uj0GmvzfoD1%2BpJP2zw5HenuEYzU%3D&Expires=1532598675&AWSAccessKeyId=AKIAIQWXW6WLXMB5QZAQ&versionId=rbOyQ3EO5dhbTuz.gl60O6iXfntn7QM1&response-content-disposition=attachment%3B%20filename%3D%22pypy-5.1.0-linux64.tar.bz2%22\r\nResolving bbuseruploads.s3.amazonaws.com... 52.216.134.35\r\nConnecting to bbuseruploads.s3.amazonaws.com|52.216.134.35|:443... connected.\r\nHTTP request sent, awaiting response... 200 OK\r\nLength: 23264958 (22M) [application/x-tar]\r\nSaving to: 'STDOUT'\r\n\r\n\r- 0%[ ] 0 --.-KB/s \r- 0%[ ] 41.52K 158KB/s \r- 0%[ ] 143.52K 272KB/s \r- 1%[ ] 279.52K 354KB/s \r- 2%[ ] 500.52K 475KB/s \r- 3%[ ] 806.52K 613KB/s \r- 5%[> ] 1.22M 791KB/s \r- 8%[> ] 1.83M 1020KB/s \r- 12%[=> ] 2.67M 1.31MB/s \r- 15%[==> ] 3.41M 1.50MB/s \r- 21%[===> ] 4.75M 1.91MB/s \r- 26%[====> ] 5.85M 2.18MB/s \r- 32%[=====> ] 7.10M 2.45MB/s \r- 38%[======> ] 8.45M 2.73MB/s eta 5s \r- 43%[=======> ] 9.63M 2.92MB/s eta 5s \r- 49%[========> ] 11.07M 3.17MB/s eta 5s \r- 58%[==========> ] 12.93M 3.66MB/s eta 5s \r- 65%[============> ] 14.60M 4.29MB/s eta 5s \r- 73%[=============> ] 16.39M 4.75MB/s eta 1s \r- 81%[===============> ] 18.17M 5.20MB/s eta 1s \r- 89%[================> ] 19.76M 5.82MB/s eta 1s \r- 95%[==================> ] 21.14M 6.12MB/s eta 1s \r- 100%[===================>] 22.19M 6.45MB/s in 4.8s \r\n\r\n2018-07-26 11:34:45 (4.59 MB/s) - written to stdout [23264958/23264958]\r\n\r\n/opt/bin/pypy/bin/pypy: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory\r\n", "stdout_lines": ["--2018-07-26 11:34:31-- https://bitbucket.org/pypy/pypy/downloads/pypy-5.1.0-linux64.tar.bz2", "Resolving bitbucket.org... 104.192.143.3, 104.192.143.1, 104.192.143.2, ...", "Connecting to bitbucket.org|104.192.143.3|:443... connected.", "HTTP request sent, awaiting response... 302 Found", "Location: https://bbuseruploads.s3.amazonaws.com/54220cd1-b139-4188-9455-1e13e663f1ac/downloads/b4819e39-76fe-4453-8315-9d76fcbec5a2/pypy-5.1.0-linux64.tar.bz2?Signature=Uj0GmvzfoD1%2BpJP2zw5HenuEYzU%3D&Expires=1532598675&AWSAccessKeyId=AKIAIQWXW6WLXMB5QZAQ&versionId=rbOyQ3EO5dhbTuz.gl60O6iXfntn7QM1&response-content-disposition=attachment%3B%20filename%3D%22pypy-5.1.0-linux64.tar.bz2%22 [following]", "--2018-07-26 11:34:38-- https://bbuseruploads.s3.amazonaws.com/54220cd1-b139-4188-9455-1e13e663f1ac/downloads/b4819e39-76fe-4453-8315-9d76fcbec5a2/pypy-5.1.0-linux64.tar.bz2?Signature=Uj0GmvzfoD1%2BpJP2zw5HenuEYzU%3D&Expires=1532598675&AWSAccessKeyId=AKIAIQWXW6WLXMB5QZAQ&versionId=rbOyQ3EO5dhbTuz.gl60O6iXfntn7QM1&response-content-disposition=attachment%3B%20filename%3D%22pypy-5.1.0-linux64.tar.bz2%22", "Resolving bbuseruploads.s3.amazonaws.com... 52.216.134.35", "Connecting to bbuseruploads.s3.amazonaws.com|52.216.134.35|:443... connected.", "HTTP request sent, awaiting response... 200 OK", "Length: 23264958 (22M) [application/x-tar]", "Saving to: 'STDOUT'", "", "", "- 0%[ ] 0 --.-KB/s ", "- 0%[ ] 41.52K 158KB/s ", "- 0%[ ] 143.52K 272KB/s ", "- 1%[ ] 279.52K 354KB/s ", "- 2%[ ] 500.52K 475KB/s ", "- 3%[ ] 806.52K 613KB/s ", "- 5%[> ] 1.22M 791KB/s ", "- 8%[> ] 1.83M 1020KB/s ", "- 12%[=> ] 2.67M 1.31MB/s ", "- 15%[==> ] 3.41M 1.50MB/s ", "- 21%[===> ] 4.75M 1.91MB/s ", "- 26%[====> ] 5.85M 2.18MB/s ", "- 32%[=====> ] 7.10M 2.45MB/s ", "- 38%[======> ] 8.45M 2.73MB/s eta 5s ", "- 43%[=======> ] 9.63M 2.92MB/s eta 5s ", "- 49%[========> ] 11.07M 3.17MB/s eta 5s ", "- 58%[==========> ] 12.93M 3.66MB/s eta 5s ", "- 65%[============> ] 14.60M 4.29MB/s eta 5s ", "- 73%[=============> ] 16.39M 4.75MB/s eta 1s ", "- 81%[===============> ] 18.17M 5.20MB/s eta 1s ", "- 89%[================> ] 19.76M 5.82MB/s eta 1s ", "- 95%[==================> ] 21.14M 6.12MB/s eta 1s ", "- 100%[===================>] 22.19M 6.45MB/s in 4.8s ", "", "2018-07-26 11:34:45 (4.59 MB/s) - written to stdout [23264958/23264958]", "", "/opt/bin/pypy/bin/pypy: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: No such file or directory"]}
NO MORE HOSTS LEFT ********************************************************************************************************************************************************************************************
to retry, use: --limit @/root/kubespray-2.5.0/cluster.retry
Anything else do we need to know:
CoreOS 1800.4.0 updates libncurses to version 6. which breaks python.
Container Linux by CoreOS stable (1800.4.0)
core@k8s001-api001 ~ $ ls -al /opt/bin/pypy/lib/libtinfo.so.5
lrwxrwxrwx. 1 root root 24 Jul 26 11:34 /opt/bin/pypy/lib/libtinfo.so.5 -> /lib64/libncurses.so.5.9
core@k8s001-api001 ~ $ ls -al /lib64/libncurses.so.*
lrwxrwxrwx. 1 root root 17 Jul 25 02:23 /lib64/libncurses.so.6 -> libncurses.so.6.1
-rwxr-xr-x. 1 root root 416392 Jul 24 22:04 /lib64/libncurses.so.6.1
core@k8s001-api001 ~ $
vs
Container Linux by CoreOS stable (1745.7.0)
core@k8s001-node002 ~ $ ls -al /opt/bin/pypy/lib/libtinfo.so.5
lrwxrwxrwx. 1 root root 24 Jul 25 16:29 /opt/bin/pypy/lib/libtinfo.so.5 -> /lib64/libncurses.so.5.9
core@k8s001-node002 ~ $ ls -al /lib64/libncurses.so.*
lrwxrwxrwx. 1 root root 17 Jun 14 11:12 /lib64/libncurses.so.5 -> libncurses.so.5.9
-rwxr-xr-x. 1 root root 371696 Jun 14 08:54 /lib64/libncurses.so.5.9
core@k8s001-node002 ~ $
verwilst
All 5 comments
As a workaround:
k8s001-api002 core # cgpt find -t coreos-usr | grep --invert-match "$(rootdev -s /usr)"
ERROR: cgpt find: Can't open /dev/sr0: No medium found
/dev/sda4
k8s001-api002 core # cgpt prioritize /dev/sda4
k8s001-api002 core # reboot
This brings you back to the older, working version for now.
verwilst
on 26 Jul 2018
for fast fix I changed:
-ln -snf /lib64/libncurses.so.5.9 $BINDIR/pypy/lib/libtinfo.so.5
+ln -snf /lib64/libncurses.so.6 $BINDIR/pypy/lib/libtinfo.so.5
targs08
on 26 Jul 2018
👍7
Maybe is better to use something like this:
ln -snf $(ls /lib64/* | grep "libncurses.so.[0-9]\+.\?[0-9]\?" | tail -1) $BINDIR/pypy/lib/libtinfo.so.5
rocko-n
on 27 Jul 2018
👍3
@rocko-n you should propose a PR
neith00
on 3 Aug 2018
PR is created. Awaiting CI.
woopstar
on 3 Aug 2018
Was this page helpful?
0 / 5 - 0 ratings
Related issues
lacebal
路
3Comments
servo1x
路
4Comments
mattdornfeld
路
4Comments
sermilrod
路
4Comments
TurboTim
路
4Comments
Most helpful comment
for fast fix I changed: