Scylla: epel update causes scylla_setup failed for ModuleNotFoundError: No module named 'yaml'

[amoskong]

Installation details
Scylla version (or git commit hash): 2.3.4-0.20190318.7818c63eb.el7
Cluster size: 4
OS (RHEL/CentOS/Ubuntu/AWS AMI): CentOS7

• Description: Two upgrade with same configuration, 199 failed in scylla_setup for lack of yaml python module
  
  
  
  • http://jenkins.cloudius-systems.com:8080/job/scylla-3.0/job/3.0-test/job/scylla-rolling-upgrade-centos-test/base_version=2.3,label=gce3-qavpc,test_sst3=true,version_desc=3-0/198/ PASS
  
  
  • http://jenkins.cloudius-systems.com:8080/job/scylla-3.0/job/3.0-test/job/scylla-rolling-upgrade-centos-test/base_version=2.3,label=gce3-qavpc,test_sst3=true,version_desc=3-0/199/ FAIL
  
  

scylla_setup log of node 2 in job 199:

Running 'sudo /usr/lib/scylla/scylla_setup --nic eth0 --disks /dev/nvme0n1'
/ssh -tt -a -x  -o ControlPath=/var/tmp/ssh-masterax2fiK/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/var/tmp/tmpnk6caG -o BatchMode=yes -o ConnectTimeout=300 -o ServerAliveInterval=300 -l scylla-test -p 22 -i /root/.ssh/scylla-test 10.142.0.20 "sudo /usr/lib/scylla/scylla_setup --nic eth0 --disks /dev/nvme0n1"'
  2019-04-04 00:57:32,552 [shard 0] iotune - /var/tmp/mnt passed sanity checks
This is a supported kernel version.
Created symlink from /etc/systemd/system/multi-user.target.wants/scylla-server.service to /usr/lib/systemd/system/scylla-server.service.
You current Scylla release is 2.3.4 the latest minor release is 3.0.3 go to http://www.scylladb.com for upgrade instructions
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.fileplanet.com
 * epel: reflector.westga.edu
 * extras: reflector.westga.edu
 * updates: mirror.vcu.edu
Package ntp-4.2.6p5-28.el7.centos.x86_64 already installed and latest version 
Package ntpdate-4.2.6p5-28.el7.centos.x86_64 already installed and latest version
Nothing to do
 4 Apr 00:57:40 ntpdate[31633]: adjust time server 169.254.169.254 offset 0.000609 sec
Creating XFS volume for scylla using 1 disk(s): /dev/nvme0n1
meta-data=/dev/nvme0n1           isize=512    agcount=4, agsize=24576000 blks 
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=98304000, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=48000, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e"
Traceback (most recent call last):
  File "/usr/lib/scylla/scylla_io_setup", line 25, in <module>
    import yaml
ModuleNotFoundError: No module named 'yaml'
IO configuration setup failed.

I checked the installation log, there is some difference. In job 199, some python3/python34 are updated, and python36 packages are installed. The packages are all from epel.

diff -urp install-job198-less.log  install-job199-less.log 
--- install-job198-less.log 2019-04-04 11:27:11.000000000 +0800
+++ install-job199-less.log 2019-04-04 11:26:14.000000000 +0800
@@ -1,12 +1,12 @@

-.142.0.68]: Running 'sudo yum install -y scylla'
-'/bin/ssh -tt -a -x  -o ControlPath=/var/tmp/ssh-masterszdsgN/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/var/tmp/tmpFuACzp -o BatchMode=yes -o ConnectTimeout=300 -o ServerAliveInterval=300 -l scylla-test -p 22 -i /root/.ssh/scylla-test 10.142.0.68 "sudo yum install -y scylla"'
+.142.0.20]: Running 'sudo yum install -y scylla'
+'/bin/ssh -tt -a -x  -o ControlPath=/var/tmp/ssh-masterax2fiK/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/var/tmp/tmpnk6caG -o BatchMode=yes -o ConnectTimeout=300 -o ServerAliveInterval=300 -l scylla-test -p 22 -i /root/.ssh/scylla-test 10.142.0.20 "sudo yum install -y scylla"'
  Loaded plugins: fastestmirror
  Loading mirror speeds from cached hostfile
-  * base: mirrors.advancedhosters.com
-  * epel: mirror.rackspace.com
-  * extras: mirrors.advancedhosters.com
-  * updates: mirrors.advancedhosters.com
+  * base: mirror.fileplanet.com
+  * epel: reflector.westga.edu
+  * extras: reflector.westga.edu
+  * updates: mirror.vcu.edu
  Resolving Dependencies
  --> Running transaction check
  ---> Package scylla.x86_64 0:2.3.4-0.20190318.7818c63eb.el7 will be installed
@@ -82,12 +82,15 @@
  ---> Package protobuf.x86_64 0:2.5.0-8.el7 will be installed
  ---> Package pyparsing.noarch 0:1.5.6-9.el7 will be installed
  ---> Package python-urwid.x86_64 0:1.1.1-3.el7 will be installed
- ---> Package python3-pyudev.noarch 0:0.21.0-1.el7 will be installed
- --> Processing Dependency: python34-six for package: python3-pyudev-0.21.0-1.el7.noarch
- ---> Package python34.x86_64 0:3.4.9-2.el7 will be installed
- --> Processing Dependency: python34-libs(x86-64) = 3.4.9-2.el7 for package: python34-3.4.9-2.el7.x86_64
- --> Processing Dependency: libpython3.4m.so.1.0()(64bit) for package: python34-3.4.9-2.el7.x86_64
- ---> Package python34-PyYAML.x86_64 0:3.11-3.el7 will be installed
+ ---> Package python3-pyudev.noarch 0:0.21.0-2.el7 will be installed
+ --> Processing Dependency: python36-six for package: python3-pyudev-0.21.0-2.el7.noarch
+ ---> Package python34.x86_64 0:3.4.9-3.el7 will be installed
+ --> Processing Dependency: python34-libs(x86-64) = 3.4.9-3.el7 for package: python34-3.4.9-3.el7.x86_64
+ --> Processing Dependency: libpython3.4m.so.1.0()(64bit) for package: python34-3.4.9-3.el7.x86_64
+ ---> Package python34-PyYAML.x86_64 0:3.11-4.el7 will be installed
+ ---> Package python36.x86_64 0:3.6.6-5.el7 will be installed
+ --> Processing Dependency: python36-libs(x86-64) = 3.6.6-5.el7 for package: python36-3.6.6-5.el7.x86_64
+ --> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python36-3.6.6-5.el7.x86_64
  ---> Package scylla-conf.x86_64 0:2.3.4-0.20190318.7818c63eb.el7 will be installed
  ---> Package scylla-env.noarch 0:1.1-1.el7 will be installed
  ---> Package snappy.x86_64 0:1.1.0-3.el7 will be installed
@@ -107,8 +110,9 @@
  ---> Package lksctp-tools.x86_64 0:1.0.17-2.el7 will be installed
  ---> Package nettle.x86_64 0:2.7.1-8.el7 will be installed
  ---> Package pcsc-lite-libs.x86_64 0:1.8.8-8.el7 will be installed
- ---> Package python34-libs.x86_64 0:3.4.9-2.el7 will be installed
- ---> Package python34-six.noarch 0:1.11.0-1.el7 will be installed
+ ---> Package python34-libs.x86_64 0:3.4.9-3.el7 will be installed
+ ---> Package python36-libs.x86_64 0:3.6.6-5.el7 will be installed
+ ---> Package python36-six.noarch 0:1.11.0-3.el7 will be installed
  ---> Package trousers.x86_64 0:0.3.14-2.el7 will be installed
  ---> Package tzdata-java.noarch 0:2019a-1.el7 will be installed
  --> Running transaction check
@@ -157,11 +161,13 @@
   python-javapackages   noarch 3.4.1-11.el7                    base         31 k
   python-lxml           x86_64 3.2.1-4.el7                     base        758 k
   python-urwid          x86_64 1.1.1-3.el7                     base        654 k
-  python3-pyudev        noarch 0.21.0-1.el7                    epel         77 k
-  python34              x86_64 3.4.9-2.el7                     epel         51 k
-  python34-PyYAML       x86_64 3.11-3.el7                      epel        150 k
-  python34-libs         x86_64 3.4.9-2.el7                     epel        8.3 M
-  python34-six          noarch 1.11.0-1.el7                    epel         33 k
+  python3-pyudev        noarch 0.21.0-2.el7                    epel         77 k
+  python34              x86_64 3.4.9-3.el7                     epel         51 k
+  python34-PyYAML       x86_64 3.11-4.el7                      epel        150 k
+  python34-libs         x86_64 3.4.9-3.el7                     epel        8.3 M
+  python36              x86_64 3.6.6-5.el7                     epel         66 k
+  python36-libs         x86_64 3.6.6-5.el7                     epel        8.5 M
+  python36-six          noarch 1.11.0-3.el7                    epel         33 k
   scylla-conf           x86_64 2.3.4-0.20190318.7818c63eb.el7  scylla-2.3   17 k
   scylla-env            noarch 1.1-1.el7                       scylladb-scylla-3rdparty
                                                                            3.1 k
@@ -184,7 +190,7 @@

  Transaction Summary
  ================================================================================
- Install  1 Package (+46 Dependent packages)
+ Install  1 Package (+48 Dependent packages)

- Total download size: 99 M
- Installed size: 284 M
+ Total download size: 108 M
+ Installed size: 320 M

/CC @roydahan @syuu1228 @tzach

[amoskong]

Installation log of one node:

• install-job198-less.log.txt
• install-job199-less.log.txt

full log (upgrade & installation & scylla_setup) of one node:

• install-job199.log.txt
• install-job198.log.txt

[amoskong]

In Job 199, python3.6 is installed, then the setup script will be run with python 3.6. but python36-PyYAML won't be installed automatically. We only have python-34 and python-34-PyYAML dependencies in spec file of scylla 2.3

In Job 198, only python 3.4 is installed, and python-34-PyYAML will be installed for dependency. So it works well.

[slivne]

workaround: https://docs.scylladb.com/troubleshooting/python-error-no-module-named-yaml/

[amoskong]

Brach-3.0 already applied a fix for this ISSUE:

• https://github.com/scylladb/scylla/commit/6e51a95 , [PATCH] dist/redhat: switch to python36

[avikivity]

2.3 also fixed - e888009f1257d86c9e4c4150e63bda373355e9ee