Scylla: Stuck server while upgrading to 4.4 from 4.3.2

[fggarcia]

This is Scylla's bug tracker, to be used for reporting bugs only.
If you have a question about Scylla, and not a bug, please ask it in
our mailing-list at [email protected] or in our slack channel.

• [] I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.

Installation details
Scylla version (or git commit hash): 4.3.2
Cluster size: 3
OS (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu

Hardware details (for performance issues) Delete if unneeded
Platform (physical/VM/cloud instance type/docker):
Hardware: sockets= cores= hyperthreading= memory=
Disks: (SSD/HDD, count)

Try to update to last version 4.4.0
When first intance updated start, it stucks at reading CF Keyspaces:

Mar 30 14:28:42 zoom-flights-scy-06 scylla[11238]: Scylla version 4.4.0-0.20210322.dffbcabbb with build-id 71a06d8290443f3abdd4f67911ab332db44d9a51 starting ...
Mar 30 14:28:42 zoom-flights-scy-06 scylla[11238]: command used: "/usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --lock-memory=1"
Mar 30 14:28:42 zoom-flights-scy-06 scylla[11238]: parsed command line options: [log-to-syslog: 1, log-to-stdout: 0, default-log-level: info, network-stack: posix, io-properties-file: /etc/scylla.d/io_properties.yaml, lock-memory: 1]
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - Unknown option : workdir
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - installing SIGHUP handler
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - Scylla version 4.4.0-0.20210322.dffbcabbb with build-id 71a06d8290443f3abdd4f67911ab332db44d9a51 starting ...
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - starting prometheus API server
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - starting tokens manager
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - starting migration manager notifier
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - creating tracing
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - creating snitch
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - determining DNS name
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - starting API server
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - Scylla API server listening on 127.0.0.1:10000 ...
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - initializing storage service
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - starting per-shard database core
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] init - creating and verifying directories
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Populating Keyspace system_schema
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF view_virtual_columns id=08843b63-45dc-3be2-9798-a0418295cfaa version=c777531c-15f7-326f-8ebe-39fd0265c8c9
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF computed_columns id=cc7c7069-3740-33c1-92a4-c3de78dbd2c4 version=2b8c4439-de76-31e0-807f-3b7290a975d7
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF aggregates id=924c5587-2e3a-345b-b10c-12f37c1ba895 version=4b53e92c-0368-3d5c-b959-2ec1bfd1a59f
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF functions id=96489b79-80be-3e14-a701-66a0b9159450 version=329ed804-55b3-3eee-ad61-d85317b96097
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF views id=9786ac1c-dd58-3201-a7cd-ad556410c985 version=5b58bb47-96e7-3f57-accf-0bfca4dbbc6e
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF indexes id=0feb57ac-311f-382f-ba6d-9024d305702f version=99c40462-8687-304e-abe3-2bdbef1f25aa
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF types id=5a8b1ca8-6602-3f77-a045-9273d308917a version=de51b2ce-5e4d-3b7d-a75f-2204332ce8d1
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF triggers id=4df70b66-6b05-3251-95a1-32b54005fd48 version=582d7071-1ef0-37c8-adc6-471a13636139
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF dropped_columns id=5e7583b5-f3f4-3af1-9a39-b7e1d6f5f11f version=7426bc6c-4c2f-3200-8ad8-4329610ed59a
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF columns id=24101c25-a2ae-3af7-87c1-b40ee1aca33f version=d33236d4-9bdd-3c09-abf0-a0bc5edc2526
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF scylla_tables id=5d912ff1-f759-3665-b2c8-8042ab5103dd version=38e5e56b-155d-39b5-a0d2-8e5dcad42a26
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF tables id=afddfb9d-bc1e-3068-8056-eed6c302ba09 version=b6240810-eeb7-36d5-9411-43b2d68dddab
Mar 30 14:28:43 zoom-flights-scy-06 scylla[11238]:  [shard 0] database - Keyspace system_schema: Reading CF keyspaces id=abac5682-dea6-31c5-b535-b3d6cffd0fb6 version=e79ca8ba-6556-3f7d-925a-7f20cf57938c

[avikivity]

Is there any disk activity? Anything in monitoring advanced dashboard?

[fggarcia]

[avikivity]

[fggarcia]

[fggarcia]

[fggarcia]

[fggarcia]

[slivne]

[slivne]

[fggarcia]

[slivne]

[fggarcia]

[krisdaniels]

[bhalevy]

[bhalevy]

[krisdaniels]

[bhalevy]

[xemul]

[fggarcia]

[fggarcia]