scylla_io_setup on GKE: found more than one disk mounted at root
Installation details
Scylla version: 4.3.0
Cluster size: 3
OS: GKE (over GCE) + scylla-operator using Helm
Platform (physical/VM/cloud instance type/docker): GKE cloud instance with K8s v1.16.15-gke.6000
Hardware: n1-standard-2 (vcpu=2 memory=7.5G)
Disks: SSD persistent disk (pd-ssd), no local SSD
Logs when starting the first Scylla instance:
│ {"L":"INFO","T":"2021-02-03T17:20:23.602Z","N":"member_controller","M":"Starting reconciliation...","_trace_id":"hhrl_QEtQcmJlorMijTAxQ"}
│ running: (['/opt/scylladb/scripts/scylla_dev_mode_setup', '--developer-mode', '0'],) │
│ running: (['/opt/scylladb/scripts/scylla_cpuset_setup', '--cpuset', '0-1'],) │
│ running: (['/opt/scylladb/scripts/scylla_io_setup'],) │
│ Problem when parsing disks from OS: │
│ found more than one disk mounted at root │
│ Traceback (most recent call last): │
│ File "/opt/scylladb/scripts/libexec/scylla_io_setup", line 239, in <module> │
│ if idata.is_recommended_instance(): │
│ File "/opt/scylladb/scripts/scylla_util.py", line 311, in is_recommended_instance │
│ diskSize = self.firstNvmeSize │
│ File "/opt/scylladb/scripts/scylla_util.py", line 290, in firstNvmeSize │
│ ephemeral_disks = self.getEphemeralOsDisks() │
│ File "/opt/scylladb/scripts/scylla_util.py", line 177, in getEphemeralOsDisks │
│ return self.os_disks[self.EPHEMERAL] │
│ File "/opt/scylladb/scripts/scylla_util.py", line 169, in os_disks │
│ nvmes_present = self._non_root_nvmes() │
│ File "/opt/scylladb/scripts/scylla_util.py", line 155, in _non_root_nvmes │
│ raise Exception("found more than one disk mounted at root ".format(root_dev_candidates)) │
│ Exception: found more than one disk mounted at root │
│ failed! │
│ Traceback (most recent call last): │
│ File "/docker-entrypoint.py", line 27, in <module> │
│ setup.io() │
│ File "/scyllasetup.py", line 67, in io │
│ self._run(['/opt/scylladb/scripts/scylla_io_setup']) │
│ File "/scyllasetup.py", line 37, in _run │
│ subprocess.check_call(*args, **kwargs) │
│ File "/opt/scylladb/python3/lib64/python3.8/subprocess.py", line 364, in check_call │
│ raise CalledProcessError(retcode, cmd) │
│ subprocess.CalledProcessError: Command '['/opt/scylladb/scripts/scylla_io_setup']' returned non-zero exit status 1.
Result of the mount command on the instance:
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/32O6Y3GND5YVHIDEYAYQALKJYW:/var/lib/docker/overlay2/l/DZOPRJ3YPUGL2LLOWS7HTYEPJJ:/var/lib/docker/overlay2/l/QVJYLMFUGLTNU5CFRYOIQ4H6CH:/var/lib/docker/overlay2/l/Y57AOZDPCTSZS6IJOMGJLEUODO:/var/lib/docker/overlay2/l/IOFSLW65XQUNHBD4EBVOQMQTH5:/var/lib/docker/overlay2/l/C3XIQX6HVNY5S7BWYRL5SLVJFJ:/var/lib/docker/overlay2/l/E4UXSM63JPMLKALZIZIDHZOPVW:/var/lib/docker/overlay2/l/IIN6QTH56OTAT7I2A3EOPIKKQ6:/var/lib/docker/overlay2/l/UVEO2RDQGFNMMWYADP3NFDSZMH:/var/lib/docker/overlay2/l/DWMKLR6XF7CNECA6MXMA7UJSZC:/var/lib/docker/overlay2/l/NMY32XTDDO5R5N22TA264CADUV:/var/lib/docker/overlay2/l/WABZOCGMAGPU766T7R26GYXVZD:/var/lib/docker/overlay2/l/O2FE76MNRKFNTB62MXESLTWDMU:/var/lib/docker/overlay2/l/GHPXZXQAXE32NDYDPXOQWFIEVR:/var/lib/docker/overlay2/l/U5P2HJWRPUDQL7ROXHEBTXQSCX:/var/lib/docker/overlay2/l/BIR2EN3J2LHADN3YMZKWGRXT77:/var/lib/docker/overlay2/l/XIE7AYJOVLPZJTAD5V4OQIGYFX:/var/lib/docker/overlay2/l/LK2AII4JXD6CSX2GSUKZTVWDCE:/var/lib/docker/overlay2/l/JXFOQHNX6HMTD25LWOPWFDAL5X:/var/lib/docker/overlay2/l/C3AOL3MSRJFGLPIMXTO7VUZLOE:/var/lib/docker/overlay2/l/PCVFAEQXIIFYYMQ4O4JZVRO52R:/var/lib/docker/overlay2/l/2KEODVSTBHQQUM6PO7CCD4XCVE,upperdir=/var/lib/docker/overlay2/a14b6435f3fd8e0d205f2fa6b5de13210d2f13d827c0a0a855e15c5dbeb2e572/diff,workdir=/var/lib/docker/overlay2/a14b6435f3fd8e0d205f2fa6b5de13210d2f13d827c0a0a855e15c5dbeb2e572/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (ro,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/rdma type cgroup (ro,nosuid,nodev,noexec,relatime,rdma)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/sda1 on /mnt/shared type ext4 (ro,relatime,commit=30)
/dev/sda1 on /mnt/scylla-config type ext4 (ro,relatime,commit=30)
tmpfs on /mnt/scylla-client-config type tmpfs (ro,relatime)
/dev/sda1 on /dev/termination-log type ext4 (rw,relatime,commit=30)
/dev/sda1 on /etc/resolv.conf type ext4 (rw,nosuid,nodev,relatime,commit=30)
/dev/sda1 on /etc/hostname type ext4 (rw,nosuid,nodev,relatime,commit=30)
/dev/sda1 on /etc/hosts type ext4 (rw,relatime,commit=30)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/sdb on /var/lib/scylla type ext4 (rw,relatime)
tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
proc on /proc/sysrq-trigger type proc (ro,relatime)
tmpfs on /proc/acpi type tmpfs (ro,relatime)
tmpfs on /proc/kcore type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime)
tmpfs on /sys/firmware type tmpfs (ro,relatime)
The line responsible for validating the number of disks is if len(root_dev_candidates) != 1:, the error is a bit misleading because in my case I suppose the number of disks found is actually 0 and not > 1.
Is running without local SSD unsupported on GKE?
etix
All 13 comments
Unexpectedly adding local SSDs trigger the exact same error so the problem lies somewhere else.
etix
on 4 Feb 2021
@penberg / @syuu1228 I am lost -
there is a /var/lib/scylla in the output above - so we should not be checking if there are ephemeral / ssd / nvmes we should use this disk and run iotune on it - what additional validation do we need when a directory exists ?
slivne
on 4 Feb 2021
Scylla operator cluster configuration:
cpuset: false
hostNetworking: false
developerMode: false
sysctls:
- "fs.aio-max-nr=2097152"
datacenter: us-central1
racks:
- name: us-central1-c
members: 3
storage:
capacity: 40G
resources:
limits:
cpu: 1
memory: 5G
requests:
cpu: 1
memory: 5G
placement:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
# for 1.17+ : topology.kubernetes.io/zone
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- us-central1-c
tolerations:
- key: role
operator: Equal
value: scylla-clusters
effect: NoSchedule
serviceMonitor:
create: true
alternator:
enabled: false
And of course, if I enable developer mode the nodes are starting since the scripts are bypassed but I'd like to avoid doing that.
etix
on 4 Feb 2021
Note: 4.2.0 Scylla image works fine, 4.3.0 image fails with error from the issue description.
zimnx
on 4 Feb 2021
Note: 4.2.0 Scylla image works fine, 4.3.0 image fails with error from the issue description.
I confirm that the 4.2.3 image works as expected with developer mode off. The regression is in 4.3.0
etix
on 4 Feb 2021
I think why it happens is because our script tries to run in GCP mode, but the mode not supported running in GKE (in container).
It should works finely if it run in normal mode, and that's how 4.2.x works, because the GCP support is introduced at Scylla 4.3.x.
syuu1228
on 5 Feb 2021
Seems like this code auto-detect GCP and enable GCP code in scylla_io_setup by default:
https://github.com/scylladb/scylla/blob/fd1dd0eac7a303ceaa9f3ff643c4848506b0c85c/dist/common/scripts/scylla_io_setup#L236
I suppose we should align behavior with AWS here:
On AWS, we won't enable aws mode until we specified --ami option.
Maybe we can reuse --ami option, or create another one something like --gcp.
syuu1228
on 5 Feb 2021
hmm, the check for root disk should be fixed too, I'll try to have a look at this and get it fixed, but I lack an env.
@zimnx @gnumoreno do you have any env with above problem reproduced by any chance?
tarzanek
on 5 Feb 2021
I can create an env, just let me know once you need it.
zimnx
on 5 Feb 2021
@slivne this looks like a regression because 4.2.x Scylla works just fine inside containers.
Any chance to have fix for it in 4.3.x? It's not possible to run any 4.3.0 cluster using Operator on GKE which is the only platform we officially support.
zimnx
on 2 Mar 2021
@tzach @slivne this is blocking us to test Scylla Operator with Scylla versions 4.3 and above on GKE
bentsi
on 13 May 2021
@penberg please prioritize
avikivity
on 27 May 2021
Queued @tarzanek's fix as https://github.com/scylladb/scylla/commit/cf6c09a08e1f665cadb4190e1c0a7f42329d847e
penberg
on 1 Jun 2021
Related issues
Felix-zhoux
·
3Comments
dimaqq
·
6Comments
pdziepak
·
6Comments
veramine
·
5Comments
duarten
·
5Comments