[x] I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.
Installation details
Scylla version (or git commit hash): 3.0.5
Cluster size: 21
OS (RHEL/CentOS/Ubuntu/AWS AMI): Debian Stretch
I have table global_stats with counter inside:
CREATE TABLE sync.global_stats (
name text PRIMARY KEY,
count counter
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
AND comment = ''
AND compaction = {'class': 'LeveledCompactionStrategy', 'tombstone_compaction_interval': '86400', 'tombstone_threshold': '0.2'}
AND compression = {}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 432000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
I'm using it for counting total number of users in my database. So every time user is added I increment the counter and if deleted - decrement.
Here's function I use (delta is always -1 or 1, I checked it):
def update_users_total(delta):
total_users = GlobalStats.objects.get(name='users_total')
if delta < 0:
total_users.count -= abs(delta)
else:
total_users.count += delta
total_users.update()
Unfortunately it's second time when counter value shows very unexpected results:
scylla@cqlsh:sync> select * from global_stats;
name | count
-------------+---------------------
users_total | 9179770327738884416
Looks like value of count overflowed somehow. Any idea why? The count value should be around 9.17M.
As you can see in attached screenshot counter value change drastically 2 times:
- 2019-05-14 at 21:51 UTC (23:51 CEST)
- 2019-05-14 at 22:51 UTC (00:51 CEST)
Around that times I was adding new node to the Cluster, but I was unable to do so because of this issue (I hope): https://github.com/scylladb/scylla/issues/4363
The most interesting things is in the logs of the node that wasn't able to join the cluster:
maj 14 21:51:37 ams-dbs12 scylla[37696]: [shard 16] compaction - Compacting [/var/lib/scylla/data/sync/bookmark-eab1b28094ac11e89e8000000000002b/mc-27846-big-Data.db:level=2, /var/lib/scylla/data/sync/bookmark-eab1b28094ac11e89e8000000000002b/mc-25960-big-Data.db:level=1, ]
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 0] boot_strapper - Error during bootstrap: streaming::stream_exception (Stream failed)
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 0] seastar - Exiting on unhandled exception: streaming::stream_exception (Stream failed)
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 0] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 1] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 12] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 19] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 3] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 23] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 15] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 27] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 19] compaction_manager - Stopped
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 40] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 20] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 26] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 7] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 31] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 8] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 38] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 33] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 36] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 14] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 6] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 18] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 39] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 14] compaction_manager - Stopped
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 9] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 42] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 13] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 30] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 25] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 3] compaction_manager - Stopped
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 22] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 41] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 5] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 37] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 24] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 28] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 8] compaction_manager - Stopped
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 0] compaction_manager - Stopped
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 45] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 32] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 10] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 21] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 45] compaction_manager - Stopped
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 44] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 2] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 16] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 34] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 35] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 17] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 4] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 1] compaction_manager - Stopped
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 11] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 43] compaction_manager - Asked to stop
maj 14 21:51:39 ams-dbs12 scylla[37696]: [shard 29] compaction_manager - Asked to stop
maj 14 21:51:40 ams-dbs12 scylla[37696]: [shard 33] storage_proxy - Failed to apply mutation from 10.210.3.129#14: seastar::gate_closed_exception (gate closed)
maj 14 21:51:40 ams-dbs12 scylla[37696]: [shard 20] storage_proxy - Failed to apply mutation from 10.195.15.212#3: seastar::gate_closed_exception (gate closed)
maj 14 21:51:40 ams-dbs12 scylla[37696]: [shard 43] storage_proxy - Failed to apply mutation from 10.195.15.207#24: seastar::gate_closed_exception (gate closed)
maj 14 21:51:40 ams-dbs12 scylla[37696]: [shard 20] storage_proxy - Failed to apply mutation from 10.210.20.127#4: seastar::gate_closed_exception (gate closed)
maj 14 21:51:40 ams-dbs12 scylla[37696]: [shard 2] storage_proxy - Failed to apply mutation from 10.210.20.128#20: seastar::gate_closed_exception (gate closed)
maj 14 21:51:40 ams-dbs12 scylla[37696]: [shard 35] storage_proxy - Failed to apply mutation from 10.210.20.126#16: seastar::gate_closed_exception (gate closed)
maj 14 21:51:40 ams-dbs12 scylla[37696]: [shard 44] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:40 ams-dbs12 scylla[37696]: [shard 44] compaction_manager - Stopped
maj 14 21:51:40 ams-dbs12 scylla[37696]: [shard 1] storage_proxy - Failed to apply mutation from 10.210.20.128#25: seastar::gate_closed_exception (gate closed)
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 20] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 20] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 21] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 21] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 28] compaction_manager - compaction info: Compaction for sync/typed_url was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 28] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 22] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 22] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 16] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 16] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 18] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 18] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 5] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 5] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 10] compaction_manager - compaction info: Compaction for sync/bookmark_by_id was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 10] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 30] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 15] storage_proxy - Failed to apply mutation from 10.210.20.127#13: seastar::gate_closed_exception (gate closed)
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 7] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 7] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 11] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 12] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 12] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 15] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 42] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 42] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 25] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 25] compaction_manager - Stopped
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 26] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 3] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 1] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 37] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:41 ams-dbs12 scylla[37696]: [shard 37] compaction_manager - Stopped
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 36] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 36] compaction_manager - Stopped
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 8] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 14] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 17] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 17] compaction_manager - Stopped
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 13] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 13] compaction_manager - Stopped
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 0] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x57c52d
0x2ff7cf
/lib/x86_64-linux-gnu/libc.so.6+0x202e0
0x372d89
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 2] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 2] compaction_manager - Stopped
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 27] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 27] compaction_manager - Stopped
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 34] compaction_manager - compaction info: Compaction for sync/bookmark_by_id was stopped due to shutdown.
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 34] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 34] compaction_manager - Stopped
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 39] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 39] compaction_manager - Stopped
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 19] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 38] storage_proxy - Failed to apply mutation from 10.210.20.84#43: seastar::gate_closed_exception (gate closed)
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 31] storage_proxy - Failed to apply mutation from 10.195.15.207#24: seastar::gate_closed_exception (gate closed)
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 11] storage_proxy - Failed to apply mutation from 10.195.15.210#12: seastar::gate_closed_exception (gate closed)
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 31] storage_proxy - Failed to apply mutation from 10.195.15.208#35: seastar::gate_closed_exception (gate closed)
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 30] storage_proxy - Failed to apply mutation from 10.195.15.208#13: seastar::gate_closed_exception (gate closed)
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 36] storage_proxy - Failed to apply mutation from 10.195.15.230#0: seastar::gate_closed_exception (gate closed)
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 37] storage_proxy - Failed to apply mutation from 10.210.20.126#30: seastar::gate_closed_exception (gate closed)
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 44] storage_proxy - Failed to apply mutation from 10.210.20.122#29: seastar::gate_closed_exception (gate closed)
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 44] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:42 ams-dbs12 scylla[37696]: [shard 45] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 9] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 9] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 9] compaction_manager - Stopped
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 28] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 12] storage_proxy - Failed to apply mutation from 10.210.3.84#30: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 20] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 22] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 21] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 34] storage_proxy - Failed to apply mutation from 10.210.20.128#21: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 15] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 15] compaction_manager - Stopped
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 33] compaction_manager - compaction info: Compaction for sync/bookmark_by_id was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 29] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 29] compaction_manager - Stopped
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 43] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 43] compaction_manager - Stopped
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 39] storage_proxy - Failed to apply mutation from 10.210.20.81#10: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 4] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 4] compaction_manager - Stopped
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 11] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 11] compaction_manager - Stopped
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 32] storage_proxy - Failed to apply mutation from 10.210.20.126#6: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 32] storage_proxy - Failed to apply mutation from 10.195.15.212#23: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 31] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 31] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 31] compaction_manager - Stopped
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 4] storage_proxy - Failed to apply mutation from 10.195.15.221#27: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 9] storage_proxy - Failed to apply mutation from 10.210.20.81#23: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 38] storage_proxy - Failed to apply mutation from 10.195.15.222#25: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 36] storage_proxy - Failed to apply mutation from 10.210.20.83#45: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 5] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 41] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 41] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 41] compaction_manager - Stopped
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 23] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 7] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 30] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 30] compaction_manager - Stopped
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 23] compaction_manager - compaction info: Compaction for sync/session was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 23] compaction_manager - Stopped
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 38] compaction_manager - compaction info: Compaction for sync/typed_url was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 33] storage_proxy - Failed to apply mutation from 10.210.20.122#3: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 38] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 38] compaction_manager - Stopped
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 11] storage_proxy - Failed to apply mutation from 10.210.20.128#22: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 32] storage_proxy - Failed to apply mutation from 10.195.15.222#21: seastar::gate_closed_exception (gate closed)
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 16] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 10] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:43 ams-dbs12 scylla[37696]: [shard 5] storage_proxy - Failed to apply mutation from 10.195.15.207#31: seastar::gate_closed_exception (gate closed)
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 36] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 17] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 13] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 12] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 38] storage_proxy - Failed to apply mutation from 10.210.3.129#16: seastar::gate_closed_exception (gate closed)
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 18] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 2] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 15] storage_proxy - Failed to apply mutation from 10.210.20.122#21: seastar::gate_closed_exception (gate closed)
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 36] storage_proxy - Failed to apply mutation from 10.195.15.222#15: seastar::gate_closed_exception (gate closed)
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 27] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 25] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 42] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 39] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 37] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 33] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 33] compaction_manager - Stopped
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 8] storage_proxy - Failed to apply mutation from 10.210.20.128#15: seastar::gate_closed_exception (gate closed)
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 34] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 43] storage_proxy - Failed to apply mutation from 10.210.3.84#27: seastar::gate_closed_exception (gate closed)
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 32] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 32] compaction_manager - compaction info: Compaction for sync/session_by_id was stopped due to shutdown.
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 0] storage_proxy - Failed to apply mutation from 10.210.20.126#8: seastar::gate_closed_exception (gate closed)
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 32] compaction_manager - compaction info: Compaction for sync/typed_url was stopped due to shutdown.
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 32] compaction_manager - Stopped
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 32] storage_proxy - Failed to apply mutation from 10.210.3.84#38: seastar::gate_closed_exception (gate closed)
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 35] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 6] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 6] compaction_manager - Stopped
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 35] compaction_manager - compaction info: Compaction for sync/bookmark_by_id was stopped due to shutdown.
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 35] compaction_manager - Stopped
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 9] storage_proxy - Failed to apply mutation from 10.210.20.128#37: seastar::gate_closed_exception (gate closed)
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 5] storage_proxy - Failed to apply mutation from 10.210.20.122#18: seastar::gate_closed_exception (gate closed)
maj 14 21:51:44 ams-dbs12 scylla[37696]: [shard 0] storage_proxy - Failed to apply mutation from 10.195.15.218#39: seastar::gate_closed_exception (gate closed)
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 24] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 24] compaction_manager - Stopped
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 26] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 26] compaction_manager - Stopped
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 16] storage_proxy - Failed to apply mutation from 10.210.3.84#15: seastar::gate_closed_exception (gate closed)
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 14] storage_proxy - Failed to apply mutation from 10.195.15.225#6: seastar::gate_closed_exception (gate closed)
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 28] storage_proxy - Failed to apply mutation from 10.210.20.126#9: seastar::gate_closed_exception (gate closed)
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 26] storage_proxy - Failed to apply mutation from 10.210.3.84#23: seastar::gate_closed_exception (gate closed)
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 43] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 9] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 15] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 32] storage_proxy - Failed to apply mutation from 10.195.15.225#37: seastar::gate_closed_exception (gate closed)
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 11] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 29] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 24] storage_proxy - Failed to apply mutation from 10.195.15.230#3: seastar::gate_closed_exception (gate closed)
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 30] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 41] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 4] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 31] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:45 ams-dbs12 scylla[37696]: [shard 38] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:46 ams-dbs12 scylla[37696]: [shard 23] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:46 ams-dbs12 scylla[37696]: [shard 6] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:46 ams-dbs12 scylla[37696]: [shard 40] compaction_manager - compaction info: Compaction for sync/bookmark was stopped due to shutdown.
maj 14 21:51:46 ams-dbs12 scylla[37696]: [shard 40] compaction_manager - Stopped
maj 14 21:51:47 ams-dbs12 scylla[37696]: [shard 35] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:47 ams-dbs12 scylla[37696]: [shard 32] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:47 ams-dbs12 scylla[37696]: [shard 33] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:47 ams-dbs12 scylla[37696]: [shard 9] storage_proxy - Failed to apply mutation from 10.195.15.230#1: seastar::gate_closed_exception (gate closed)
maj 14 21:51:47 ams-dbs12 scylla[37696]: [shard 26] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:47 ams-dbs12 scylla[37696]: [shard 24] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:51:49 ams-dbs12 scylla[37696]: [shard 40] seastar - Exceptional future ignored: seastar::timed_out_error (timedout), backtrace: 0x59b0be
0x37a2e8
0x117580e
0x48656f
0x3746f4
0x4491b9
0x46c6ba
0x52592d
/lib/x86_64-linux-gnu/libpthread.so.0+0x74a3
/lib/x86_64-linux-gnu/libc.so.6+0xe8d0e
maj 14 21:53:09 ams-dbs12 systemd[1]: scylla-server.service: Main process exited, code=exited, status=1/FAILURE
maj 14 21:53:09 ams-dbs12 systemd[1]: scylla-server.service: Unit entered failed state.
maj 14 21:53:09 ams-dbs12 systemd[1]: scylla-server.service: Failed with result 'exit-code'.
I'm not telling that this is somehow correlated for sure, but timing is perfect...
r4fek
All 29 comments
@r4fek - would it be possible to get the sstables for this table
counters are actually partitioned across all the nodes that may increment them - and looking at the raw data will allow us to at least find the nodes that may have caused the issue.
slivne
on 11 Jul 2019
@r4fek any update ?
slivne
on 25 Jul 2019
@slivne oh, sorry for not seeing this before.
Do you want sstables from all the nodes in the cluster? That can be tricky because I have 23 nodes right now.
r4fek
on 26 Jul 2019
FYI, I failed to reproduce with all kind of cluster topology change during Counter updates.
tzach
on 26 Aug 2019
@r4fek if this reproduces then it would help to get the sstables for a specific partition that has the issue.
The number of nodes that hold the data is according to RF so we do not need all the nodes only ones that have the partition.
slivne
on 8 Sep 2019
@r4fek - it will really help to get the sstables if this reproduces
slivne
on 24 Oct 2019
I can't reproduce it right now, sorry.
r4fek
on 24 Oct 2019
Hi, we have a similar issue, some of our counters are under-/overflowing.
We managed to reproduce the issue and I've created a gist and attached a snapshot of the table: Gist
To reproduce this issue I deleted and added nodes to the cluster while the attached python script was running on a few servers in the meantime.
_Installation details_
Scylla version (or git commit hash): 3.1.0
Cluster size: 5
OS (RHEL/CentOS/Ubuntu/AWS AMI): CentOS Linux release 7.7.1908 (Core)
We use the Docker image you provide: scylladb/scylla:3.1.0
If you have any questions feel free to ask.
wigginus
on 5 Nov 2019
Great work @wigginus !
Can you please add the full schema for the reproducer?
How many nodes in the cluster, and what RF and CL did you use?
tzach
on 5 Nov 2019
We have 5 nodes in the cluster and the RF is 3.
I haven't set the CL in this example, but as far as I can tell the default CL of the Python Cassandra driver is LOCAL_ONE.
The output of the desc dev command can be found below:
CREATE KEYSPACE dev WITH replication = {'class': 'NetworkTopologyStrategy', 'K8S': '3'} AND durable_writes = true;
CREATE TABLE dev.dummy_counter (
name text PRIMARY KEY,
count counter
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
AND comment = 'DummyCounter'
AND compaction = {'class': 'SizeTieredCompactionStrategy'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 3600
AND gc_grace_seconds = 300
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
@slivne can you please assign someone to look at this?
tzach
on 17 Nov 2019
@slivne can you please assign someone to look at this?
tzach
on 3 Dec 2019
Looks like we got hit by this too. Some of our tables have obviously "corrupted" values, something like 9179770327738884416 or -9179770327738884416. The use-case is similar - counting things, delta is always +1. Unfortunately we found this too late but I do remember we were restarting and rebuilding nodes not that long ago.
creker
on 12 Dec 2019
@bhalevy could this be related to #4074 ?
tzach
on 15 Dec 2019
@tzach I doubt it.
I'll try to reproduce with this gist
bhalevy
on 15 Dec 2019
@creker @wigginus @r4fek I'm on it.
I tried all night to reproduce it on a local 4-node cluster. Kept adding and removing nodes while running a script similar to the gist. Sadly, no overflows.
I need any info that'll help reproducing the error. Do you guys recall performing any other non-standard actions?
jul-stas
on 19 Dec 2019
Hi Juliusz,
Sorry, for the late answer, I was away from my computer during the last few weeks.
I don't recall performing any non-standard actions, just adding and removing nodes, sometimes even if the node had not yet joined.
My Scylla cluster was running in a Kubernetes / Docker setup, this is some additional information, which I haven't mentioned already.
wigginus
on 7 Jan 2020
Same for me. Nothing unusual, just adding, removing, rejoining nodes. My cluster is also running in docker on separate hosts. 3 nodes, all using official scylladb/scylla 3.1.0 image.
creker
on 7 Jan 2020
@wigginus @creker Thanks for the update. How did you remove the nodes?
@creker What RF did you use?
jul-stas
on 8 Jan 2020
@jul-stas RF is 3. The nodes were removed in different ways - properly with drain, terminated after flush, some even crashed. Unfortunately I don't know which of these might've caused the problem.
creker
on 9 Jan 2020
@jul-stas I've also used different ways (kill, decommission, drain) to remove the nodes, but I also don't remember which way might have caused the problem. I sometimes also didn't wait for the removal to complete to add new nodes.
wigginus
on 15 Jan 2020
Just an update: I prepared binaries of Scylla 3.1.2 with extra logging around counters. Big kudos to OP, who kindly offered to help us by running this build in his env. So, the hunt goes on.
Perhaps anyone else would be interested in test-running our custom release and contributing their results?
jul-stas
on 22 Jan 2020
Could this be related to: Merge "Fix empty counters handling in MC" from Piotr (c664615960930a6ec92750c3e876bfaf58912c98)?
3.0.5 lacks this fix and mc version is used.
The bug could result in counter cells being produced without any shards. Don't know how that could explain the over/underflow of counter value yet. @haaawk: could u please share ur thoughts?
I wrote this https://paste.centos.org/view/78e0be75 to reproduce the problem but failed miserably so far.
raphaelsc
on 13 Feb 2020
The patchset c664615960930a6ec92750c3e876bfaf58912c98 says "Before this patchset empty counters were incorrectly persisted for
MC format. No value was written to disk for them. The correct way
is to still write a header that informs the counter is empty.
We also need to make sure that reading wrongly persisted empty
counters works because customers may have sstables with wrongly
persisted empty counters."
Could it be that the parser is fetching bogus values when reading the wrongly persisted counters?
raphaelsc
on 13 Feb 2020
This could lead to the corruption we're seeing. It would be interesting to see whether the number of corrupted counters grows, shinks or stays the same now that a version with a fix is being run.
Thanks for bringing this up @raphaelsc
haaawk
on 13 Feb 2020
f2d2a9f5b874fa5a21b8de679d02c2b73b1c624d was backported to branch-3.0 and is available in version 3.0.7
bhalevy
on 13 Feb 2020
Good news: the cause is finally identified and the fix is already queued. For the curious, Tzach linked the full story above.
Now you just need to wait for backports.
jul-stas
on 18 Feb 2020
🎉6
@haaawk can we close this?
tzach
on 23 Feb 2020
yes fixed by 434347195416cc79211e4b7df001f90b0f1aded7
slivne
on 23 Feb 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
Felix-zhoux
路
3Comments
amoskong
路
5Comments
eyalgutkind
路
3Comments
fgelcer
路
6Comments
tgrabiec
路
6Comments
Most helpful comment
Good news: the cause is finally identified and the fix is already queued. For the curious, Tzach linked the full story above.
Now you just need to wait for backports.