Scylla: ERROR: filesystem error: open failed: Too many open files and coredump during resharding with 5000 tables
Installation details
Scylla version (or git commit hash): 4.2.rc5-0.20201012.a9109f068 with build-id 1ec6105c11985e83710995972d80ca60ea87faa9
Cluster size: 1 + 5 (Cluster started from 1 node, then it increased by adding 1 node sequentially. Total nodes 6
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-0dfe89db5c6c31b4f(eu-west-1)
instance_type_db: i3.8xlarge
Testid: 8db60178-8fa1-4f7b-a6f5-8e5ece6195a8
Monitoring: http://3.251.78.185:3000
Job: https://jenkins.scylladb.com/job/scylla-4.2/job/longevity/job/longevity-5000-tables-test/6/
Test 5000 tables create cluster with one node and generate ks with 5000 tables, after that it add with bootstrap a node sequentially while cluster will not have 6 nodes. After that test start c-s commands in batch for 200 tables with duration 55min. IN parallel it run various nemesis.
Nemesis RestartWIthResharding apply next steps:
- stops the scylla
- update scylla.yaml with murmur3_partitioner_ignore_msb_bits=15
- starts the scylla
- wait 5 minutes
- stops the scylla
- update scylla.yaml with murmur3_partitioner_ignore_msb_bits=12
7 starts the scylla
during this nemesis next errors happened on node2:
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 5] compaction - Resharding [/var/lib/scylla/data/feeds/table657-c5a99b600d4a11ebb700000000000005/mc-261-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657-c5a99b600d4a11ebb700000000000005/mc-281-big-Data.db:level=0, ]
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 25] compaction - Resharded 2 sstables to [/var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-695-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-663-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-661-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-662-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-694-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-726-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-660-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-727-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-734-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-702-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-737-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-739-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-703-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-736-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-735-big-Data.db:level=0, /var/lib/scylla/data/feeds/table656-c4f1bd100d4a11ebb700000000000005/mc-738-big-Data.db:level=0, ]. 884kB to 12MB (~1387% of original) in 532ms = 23MB/s. ~9600 total partitions merged to 9382.
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 9] compaction - Resharding [/var/lib/scylla/data/feeds/table701_field4_table701_index-e6b1b9010d4a11ebb700000000000005/mc-250-big-Data.db:level=0, /var/lib/scylla/data/feeds/table701_field4_table701_index-e6b1b9010d4a11ebb700000000000005/mc-227-big-Data.db:level=0, ]
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 11] compaction - Resharded 2 sstables to [/var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-463-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-462-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-496-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-460-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-461-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-497-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-465-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-464-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-506-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-507-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-385-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-502-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-384-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-531-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-503-big-Data.db:level=0, /var/lib/scylla/data/feeds/table657_field4_table657_index-c603f0610d4a11ebb700000000000005/mc-530-big-Data.db:level=0, ]. 597kB to 11MB (~1998% of original) in 554ms = 21MB/s. ~8576 total partitions merged to 8449.
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 25] compaction - Resharding [/var/lib/scylla/data/feeds/table640-b996f0700d4a11ebb700000000000005/mc-250-big-Data.db:level=0, /var/lib/scylla/data/feeds/table640-b996f0700d4a11ebb700000000000005/mc-171-big-Data.db:level=0, ]
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 11] compaction - Resharding [/var/lib/scylla/data/feeds/table441_field4_table441_index-4272c1910d4a11ebb700000000000005/mc-154-big-Data.db:level=0, /var/lib/scylla/data/feeds/table441_field4_table441_index-4272c1910d4a11ebb700000000000005/mc-188-big-Data.db:level=0, ]
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 7] compaction - Resharded 2 sstables to [/var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-250-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-279-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-254-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-255-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-222-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-278-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-251-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-223-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-351-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-319-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-317-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-318-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-350-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-316-big-Data.db:level=0, ]. 681kB to 10MB (~1560% of original) in 473ms = 22MB/s. ~6784 total partitions merged to 6666.
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 7] compaction - Resharding [/var/lib/scylla/data/feeds/table767_field4_table767_index-1be394e10d4b11ebb700000000000005/mc-198-big-Data.db:level=0, /var/lib/scylla/data/feeds/table767_field4_table767_index-1be394e10d4b11ebb700000000000005/mc-175-big-Data.db:level=0, ]
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 0] compaction - Resharded 2 sstables to [/var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-293-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-295-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-299-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-298-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-297-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-292-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-294-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-296-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-275-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-274-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-277-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-279-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-272-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-273-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-276-big-Data.db:level=0, /var/lib/scylla/data/feeds/table740-05298ed00d4b11ebb700000000000005/mc-278-big-Data.db:level=0, ]. 705kB to 11MB (~1589% of original) in 532ms = 21MB/s. ~7168 total partitions merged to 6994.
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 0] compaction - Resharding [/var/lib/scylla/data/feeds/table634-b56f8de00d4a11ebb700000000000005/mc-263-big-Data.db:level=0, /var/lib/scylla/data/feeds/table634-b56f8de00d4a11ebb700000000000005/mc-218-big-Data.db:level=0, ]
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 26] compaction - Resharded 2 sstables to [/var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-525-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-527-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-460-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-526-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-523-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-461-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-524-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-522-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-473-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-537-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-472-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-504-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-568-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-569-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-505-big-Data.db:level=0, /var/lib/scylla/data/feeds/table322-0d328dd00d4a11ebb700000000000005/mc-536-big-Data.db:level=0, ]. 863kB to 12MB (~1419% of original) in 534ms = 22MB/s. ~9216 total partitions merged to 9103.
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 26] compaction - Resharding [/var/lib/scylla/data/feeds/table494_field4_table494_index-5e57de910d4a11ebb700000000000005/mc-199-big-Data.db:level=0, /var/lib/scylla/data/feeds/table494_field4_table494_index-5e57de910d4a11ebb700000000000005/mc-161-big-Data.db:level=0, ]
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !ERR | scylla: [shard 1] sstable - Could not create SSTable component /var/lib/scylla/data/feeds/table358-1bc8fd700d4a11ebb700000000000005/0000000000000171.sstable/mc-171-big-Data.db. Found exception: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table358-1bc8fd700d4a11ebb700000000000005/0000000000000171.sstable/mc-171-big-Data.db])
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !ERR | scylla: [shard 1] sstable - Could not create SSTable component /var/lib/scylla/data/feeds/table358-1bc8fd700d4a11ebb700000000000005/0000000000000171.sstable/mc-171-big-Index.db. Found exception: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table358-1bc8fd700d4a11ebb700000000000005/0000000000000171.sstable/mc-171-big-Index.db])
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !WARNING | scylla: [shard 10] storage_service - Shutting down communications due to I/O errors until operator intervention
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !ERR | scylla: [shard 3] sstable - Could not create SSTable component /var/lib/scylla/data/feeds/table674-d21a6be00d4a11ebb700000000000005/0000000000000383.sstable/mc-383-big-Filter.db. Found exception: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table674-d21a6be00d4a11ebb700000000000005/0000000000000383.sstable/mc-383-big-Filter.db])
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !WARNING | scylla: [shard 10] storage_service - Disk error: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table286_field4_table286_index-ffff09910d4911ebb700000000000005])
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !WARNING | scylla: [shard 1] seastar - Exceptional future ignored: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table358-1bc8fd700d4a11ebb700000000000005/0000000000000171.sstable/mc-171-big-Data.db]), backtrace: 0x33485ad#012 0x33488c0#012 0x3348d49#012 0x2e3a507#012 0xf067a6#012 0x1262b57#012 0x2e8ea47#012 0x2e8edbe#012 0x2ec5e9d#012 0x2ed589a#012 0x2e59b1d#012 /opt/scylladb/libreloc/libpthread.so.0+0x9431#012 /opt/scylladb/libreloc/libc.so.6+0x101912#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::create_data()::{lambda(auto:1)#1}, seastar::future<std::tuple<sstables::sstable::create_data()::{lambda(auto:1)#1}<seastar::file>, seastar::file> >::then_impl_nrvo<{lambda(auto:1)#1}, sstables::sstable::create_data()::{lambda(auto:1)#1}<> >({lambda(auto:1)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda(auto:1)#1}&, seastar::future_state<sstables::sstable::create_data()::{lambda(auto:1)#1}<seastar::file> >&&)#1}, sstables::sstable::create_data()::{lambda(auto:1)#1}<seastar::file> >#012 --------#012 seastar::(anonymous namespace)::thread_wake_task#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEENS_6futureIJEEET_ENUl20flat_mutation_readerE_clESC_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISB_E4typeEDpNSH_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSB_DpOSK_EUlvE0_ZZNSA_14then_impl_nrvoISX_SA_EET0_SU_ENKUlvE_clEvEUlRS3_RSX_ONS_12future_stateIJEEEE_JEEE#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEES5_T_ENUl20flat_mutation_readerE_clESD_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISC_E4typeEDpNSI_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSC_DpOSL_EUlvE1_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_SZ_EENSG_ISC_E4typeEOT0_ENKUlvE_clEvEUlRS3_RSZ_ONS_12future_stateIJEEEE_JEEE
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 0] storage_service - Stop transport: starts
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: scylla: /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/file.cc:503: virtual seastar::append_challenged_posix_file_impl::~append_challenged_posix_file_impl(): Assertion `_q.empty() && _logical_size == _committed_size' failed.
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: Aborting on shard 7.
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: Backtrace:
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000002eed122
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000002e916a0
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000002e91945
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000002e91990
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x00007f25694aaa8f
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: /opt/scylladb/libreloc/libc.so.6+0x000000000003c9e4
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000025894
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000025768
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000034e75
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000002e067cf
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000000ee15b8
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000000f0677d
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000001262b57
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000002e8ea47
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000002e8edbe
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000002ec5e9d
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000002ed589a
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: 0x0000000002e59b1d
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: /opt/scylladb/libreloc/libpthread.so.0+0x0000000000009431
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000101912
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 0] storage_service - Stop transport: shutdown rpc and cql server done
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 0] gossip - gossip is already stopped
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 0] storage_service - Stop transport: stop_gossiping done
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !WARNING | scylla: [shard 10] seastar - Exceptional future ignored: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table286_field4_table286_index-ffff09910d4911ebb700000000000005/mc-375-big-Data.db]), backtrace: 0x33485ad#012 0x33488c0#012 0x3348d49#012 0x2e3a507#012 0xf067a6#012 0x126c2e9#012 0x2e8ea47#012 0x2e8edbe#012 0x2ec5e9d#012 0x2ed589a#012 0x2e59b1d#012 /opt/scylladb/libreloc/libpthread.so.0+0x9431#012 /opt/scylladb/libreloc/libc.so.6+0x101912#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda(auto:1)#1}, seastar::future<std::tuple<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file>, seastar::file> >::then_impl_nrvo<{lambda(auto:1)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<> >({lambda(auto:1)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda(auto:1)#1}&, seastar::future_state<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >&&)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda()#2}, seastar::future<>::then_impl_nrvo<{lambda()#2}, seastar::future>({lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#2}&, seastar::future_state<>&&)#1}>#012 --------#012 seastar::(anonymous namespace)::thread_wake_task#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEENS_6futureIJEEET_ENUl20flat_mutation_readerE_clESC_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISB_E4typeEDpNSH_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSB_DpOSK_EUlvE0_ZZNSA_14then_impl_nrvoISX_SA_EET0_SU_ENKUlvE_clEvEUlRS3_RSX_ONS_12future_stateIJEEEE_JEEE#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEES5_T_ENUl20flat_mutation_readerE_clESD_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISC_E4typeEDpNSI_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSC_DpOSL_EUlvE1_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_SZ_EENSG_ISC_E4typeEOT0_ENKUlvE_clEvEUlRS3_RSZ_ONS_12future_stateIJEEEE_JEEE
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !ERR | scylla: [shard 26] sstable - Could not create SSTable component /var/lib/scylla/data/feeds/table494_field4_table494_index-5e57de910d4a11ebb700000000000005/0000000000000598.sstable/mc-598-big-TOC.txt.tmp. Found exception: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table494_field4_table494_index-5e57de910d4a11ebb700000000000005/0000000000000598.sstable/mc-598-big-TOC.txt.tmp])
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !ERR | scylla: [shard 3] sstable - Could not create SSTable component /var/lib/scylla/data/feeds/table674-d21a6be00d4a11ebb700000000000005/0000000000000394.sstable/mc-394-big-Filter.db. Found exception: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table674-d21a6be00d4a11ebb700000000000005/0000000000000394.sstable/mc-394-big-Filter.db])
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !WARNING | scylla: [shard 17] seastar - Exceptional future ignored: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table670-cf2e21600d4a11ebb700000000000005/mc-534-big-Data.db]), backtrace: 0x33485ad#012 0x33488c0#012 0x3348d49#012 0x2e3a507#012 0xf067a6#012 0x126c2e9#012 0x2e8ea47#012 0x2e8edbe#012 0x2ec5e9d#012 0x2ed589a#012 0x2e59b1d#012 /opt/scylladb/libreloc/libpthread.so.0+0x9431#012 /opt/scylladb/libreloc/libc.so.6+0x101912#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda(auto:1)#1}, seastar::future<std::tuple<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file>, seastar::file> >::then_impl_nrvo<{lambda(auto:1)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<> >({lambda(auto:1)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda(auto:1)#1}&, seastar::future_state<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >&&)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda()#2}, seastar::future<>::then_impl_nrvo<{lambda()#2}, seastar::future>({lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#2}&, seastar::future_state<>&&)#1}>#012 --------#012 seastar::(anonymous namespace)::thread_wake_task#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEENS_6futureIJEEET_ENUl20flat_mutation_readerE_clESC_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISB_E4typeEDpNSH_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSB_DpOSK_EUlvE0_ZZNSA_14then_impl_nrvoISX_SA_EET0_SU_ENKUlvE_clEvEUlRS3_RSX_ONS_12future_stateIJEEEE_JEEE#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEES5_T_ENUl20flat_mutation_readerE_clESD_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISC_E4typeEDpNSI_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSC_DpOSL_EUlvE1_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_SZ_EENSG_ISC_E4typeEOT0_ENKUlvE_clEvEUlRS3_RSZ_ONS_12future_stateIJEEEE_JEEE
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !WARNING | scylla: [shard 10] seastar - Exceptional future ignored: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table286_field4_table286_index-ffff09910d4911ebb700000000000005/mc-299-big-Data.db]), backtrace: 0x33485ad#012 0x33488c0#012 0x3348d49#012 0x2e3a507#012 0xf067a6#012 0x126c2e9#012 0x2e8ea47#012 0x2e8edbe#012 0x2ec5e9d#012 0x2ed589a#012 0x2e59b1d#012 /opt/scylladb/libreloc/libpthread.so.0+0x9431#012 /opt/scylladb/libreloc/libc.so.6+0x101912#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda(auto:1)#1}, seastar::future<std::tuple<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file>, seastar::file> >::then_impl_nrvo<{lambda(auto:1)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<> >({lambda(auto:1)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda(auto:1)#1}&, seastar::future_state<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >&&)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda()#2}, seastar::future<>::then_impl_nrvo<{lambda()#2}, seastar::future>({lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#2}&, seastar::future_state<>&&)#1}>#012 --------#012 seastar::(anonymous namespace)::thread_wake_task#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEENS_6futureIJEEET_ENUl20flat_mutation_readerE_clESC_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISB_E4typeEDpNSH_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSB_DpOSK_EUlvE0_ZZNSA_14then_impl_nrvoISX_SA_EET0_SU_ENKUlvE_clEvEUlRS3_RSX_ONS_12future_stateIJEEEE_JEEE#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEES5_T_ENUl20flat_mutation_readerE_clESD_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISC_E4typeEDpNSI_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSC_DpOSL_EUlvE1_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_SZ_EENSG_ISC_E4typeEOT0_ENKUlvE_clEvEUlRS3_RSZ_ONS_12future_stateIJEEEE_JEEE#012 --------#012 seastar::parallel_for_each_state#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, seastar::future<>::finally_body<mutation_writer::feed_writer<mutation_writer::shard_based_splitting_mutation_writer>(flat_mutation_reader&&, mutation_writer::shard_based_splitting_mutation_writer&&)::{lambda(flat_mutation_reader&, mutation_writer::shard_based_splitting_mutation_writer&)#1}::operator()(flat_mutation_reader&, mutation_writer::shard_based_splitting_mutation_writer&) const::{lambda()#2}, true>::operator()(seastar::future<>&&)::{lambda(auto:1&&)#1}, seastar::future<>::then_wrapped_nrvo<seastar::future<>, seastar::future<>::finally_body<mutation_writer::feed_writer<mutation_writer::shard_based_splitting_mutation_writer>(flat_mutation_reader&&, mutation_writer::shard_based_splitting_mutation_writer&&)::{lambda(flat_mutation_reader&, mutation_writer::shard_based_splitting_mutation_writer&)#1}::operator()(flat_mutation_reader&, mutation_writer::shard_based_splitting_mutation_writer&) const::{lambda()#2}, true> >(seastar::future<>::finally_body<mutation_writer::feed_writer<mutation_writer::shard_based_splitting_mutation_writer>(flat_mutation_reader&&, mutation_writer::shard_based_splitting_mutation_writer&&)::{lambda(flat_mutation_reader&, mutation_writer::shard_based_splitting_mutation_writer&)#1}::operator()(flat_mutation_reader&, mutation_writer::shard_based_splitting_mutation_writer&) const::{lambda()#2}, true>&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, seastar::future<>::finally_body<mutation_writer::feed_writer<mutation_writer::shard_based_splitting_mutation_writer>(flat_mutation_reader&&, auto:1&&)::{lambda(flat_mutation_reader&, mutation_writer::shard_based_splitting_mutation_writer&)#1}::operator()(flat_mutation_reader&, mutation_writer::shard_based_splitting_mutation_writer&) const::{lambda()#2}, true>&, seastar::future_state<>&&)#1}>#012 --------#012 seastar::internal::do_with_state<std::tuple<flat_mutation_reader, mutation_writer::shard_based_splitting_mutation_writer>, seastar::future<> >#012 --------#012 seastar::(anonymous namespace)::thread_wake_task#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJN8sstables15compaction_infoEEEEZNS_5asyncIZNS3_10compaction3runI33noop_compacted_fragments_consumerEENS_6futureIJS4_EEESt10unique_ptrIS7_St14default_deleteIS7_EET_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISG_E4typeEDpNSK_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSG_DpOSN_EUlvE0_ZZNSA_IJEE14then_impl_nrvoIS10_SB_EET0_SX_ENKUlvE_clEvEUlRS5_RS10_ONS_12future_stateIJEEEE_JEEE#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<sstables::compaction_info>, seastar::future<sstables::compaction_info>::finally_body<seastar::async<sstables::compaction::run<noop_compacted_fragments_consumer>(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >, noop_compacted_fragments_consumer)::{lambda()#1}>(seastar::thread_attributes, sstables::compaction::run<noop_compacted_fragments_consumer>(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >, noop_compacted_fragments_consumer)::{lambda()#1}&&, (std::decay<sstables::compaction::run<noop_compacted_fragments_consumer>(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >, noop_compacted_fragments_consumer)::{lambda()#1}>::type&&)...)::{lambda()#3}, false>, seastar::future<sstables::compaction_info>::then_wrapped_nrvo<seastar::future<sstables::compaction_info>, {lambda()#3}>({lambda()#3}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<sstables::compaction_info>&, {lambda()#3}&, seastar::future_state<sstables::compaction_info>&&)#1}, sstables::compaction_info>#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable_directory::reshard(utils::chunked_vector<sstables::foreign_sstable_open_info, 131072ul>, compaction_manager&, table&, unsigned int, std::function<seastar::lw_shared_ptr<sstables::sstable> (unsigned int)>, seastar::io_priority_class const&)::{lambda(std::vector<std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >, std::allocator<std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > > >&)#1}::operator()(std::vector<std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >, std::allocator<std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > > >&)::{lambda()#2}::operator()()::{lambda(std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&)#1}::operator()({lambda()#2})::{lambda()#1}::operator()() const::{lambda(sstables::compaction_info)#1}, seastar::future<{lambda(std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&)#1}>::then_impl_nrvo<{lambda(std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&)#1}, {lambda()#1}<> >({lambda(std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&)#1}&&)::{lambda()#1}::operato
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !WARNING | scylla: [shard 17] seastar - Exceptional future ignored: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table670-cf2e21600d4a11ebb700000000000005/mc-511-big-Data.db]), backtrace: 0x33485ad#012 0x33488c0#012 0x3348d49#012 0x2e3a507#012 0xf067a6#012 0x126c2e9#012 0x2e8ea47#012 0x2e8edbe#012 0x2ec5e9d#012 0x2ed589a#012 0x2e59b1d#012 /opt/scylladb/libreloc/libpthread.so.0+0x9431#012 /opt/scylladb/libreloc/libc.so.6+0x101912#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda(auto:1)#1}, seastar::future<std::tuple<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file>, seastar::file> >::then_impl_nrvo<{lambda(auto:1)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<> >({lambda(auto:1)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda(auto:1)#1}&, seastar::future_state<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >&&)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda()#2}, seastar::future<>::then_impl_nrvo<{lambda()#2}, seastar::future>({lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#2}&, seastar::future_state<>&&)#1}>#012 --------#012 seastar::(anonymous namespace)::thread_wake_task#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEENS_6futureIJEEET_ENUl20flat_mutation_readerE_clESC_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISB_E4typeEDpNSH_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSB_DpOSK_EUlvE0_ZZNSA_14then_impl_nrvoISX_SA_EET0_SU_ENKUlvE_clEvEUlRS3_RSX_ONS_12future_stateIJEEEE_JEEE#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEES5_T_ENUl20flat_mutation_readerE_clESD_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISC_E4typeEDpNSI_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSC_DpOSL_EUlvE1_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_SZ_EENSG_ISC_E4typeEOT0_ENKUlvE_clEvEUlRS3_RSZ_ONS_12future_stateIJEEEE_JEEE
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !WARNING | scylla: [shard 10] seastar - Exceptional future ignored: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table286_field4_table286_index-ffff09910d4911ebb700000000000005/mc-270-big-Data.db]), backtrace: 0x33485ad#012 0x33488c0#012 0x3348d49#012 0x2e3a507#012 0xf067a6#012 0x126c2e9#012 0x2e8ea47#012 0x2e8edbe#012 0x2ec5e9d#012 0x2ed589a#012 0x2e59b1d#012 /opt/scylladb/libreloc/libpthread.so.0+0x9431#012 /opt/scylladb/libreloc/libc.so.6+0x101912#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda(auto:1)#1}, seastar::future<std::tuple<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file>, seastar::file> >::then_impl_nrvo<{lambda(auto:1)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<> >({lambda(auto:1)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda(auto:1)#1}&, seastar::future_state<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >&&)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda()#2}, seastar::future<>::then_impl_nrvo<{lambda()#2}, seastar::future>({lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#2}&, seastar::future_state<>&&)#1}>#012 --------#012 seastar::(anonymous namespace)::thread_wake_task#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEENS_6futureIJEEET_ENUl20flat_mutation_readerE_clESC_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISB_E4typeEDpNSH_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSB_DpOSK_EUlvE0_ZZNSA_14then_impl_nrvoISX_SA_EET0_SU_ENKUlvE_clEvEUlRS3_RSX_ONS_12future_stateIJEEEE_JEEE#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEES5_T_ENUl20flat_mutation_readerE_clESD_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISC_E4typeEDpNSI_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSC_DpOSL_EUlvE1_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_SZ_EENSG_ISC_E4typeEOT0_ENKUlvE_clEvEUlRS3_RSZ_ONS_12future_stateIJEEEE_JEEE
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !WARNING | scylla: [shard 28] seastar - Exceptional future ignored: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table610-a50bff600d4a11ebb700000000000005/mc-727-big-Data.db]), backtrace: 0x33485ad#012 0x33488c0#012 0x3348d49#012 0x2e3a507#012 0xf067a6#012 0x126c2e9#012 0x2e8ea47#012 0x2e8edbe#012 0x2ec5e9d#012 0x2ed589a#012 0x2e59b1d#012 /opt/scylladb/libreloc/libpthread.so.0+0x9431#012 /opt/scylladb/libreloc/libc.so.6+0x101912#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda(auto:1)#1}, seastar::future<std::tuple<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file>, seastar::file> >::then_impl_nrvo<{lambda(auto:1)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<> >({lambda(auto:1)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda(auto:1)#1}&, seastar::future_state<sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >&&)#1}, sstables::sstable::open_data()::{lambda(auto:1)#1}<seastar::file> >#012 --------#012 seastar::continuation<seastar::internal::promise_base_with_type<>, sstables::sstable::open_data()::{lambda()#2}, seastar::future<>::then_impl_nrvo<{lambda()#2}, seastar::future>({lambda()#2}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<>&, {lambda()#2}&, seastar::future_state<>&&)#1}>#012 --------#012 seastar::(anonymous namespace)::thread_wake_task#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEEZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEENS_6futureIJEEET_ENUl20flat_mutation_readerE_clESC_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISB_E4typeEDpNSH_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSB_DpOSK_EUlvE0_ZZNSA_14then_impl_nrvoISX_SA_EET0_SU_ENKUlvE_clEvEUlRS3_RSX_ONS_12future_stateIJEEEE_JEEE#012 --------#012 N7seastar12continuationINS_8internal22promise_base_with_typeIJEEENS_6futureIJEE12finally_bodyIZNS_5asyncIZZN8sstables10compaction5setupI33noop_compacted_fragments_consumerEES5_T_ENUl20flat_mutation_readerE_clESD_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayISC_E4typeEDpNSI_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSC_DpOSL_EUlvE1_Lb0EEEZZNS5_17then_wrapped_nrvoIS5_SZ_EENSG_ISC_E4typeEOT0_ENKUlvE_clEvEUlRS3_RSZ_ONS_12future_stateIJEEEE_JEEE
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !ERR | scylla: [shard 9] sstable - Could not create SSTable component /var/lib/scylla/data/feeds/table701_field4_table701_index-e6b1b9010d4a11ebb700000000000005/0000000000000289.sstable/mc-289-big-TOC.txt.tmp. Found exception: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table701_field4_table701_index-e6b1b9010d4a11ebb700000000000005/0000000000000289.sstable/mc-289-big-TOC.txt.tmp])
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !ERR | scylla: [shard 7] sstable - Could not create SSTable component /var/lib/scylla/data/feeds/table767_field4_table767_index-1be394e10d4b11ebb700000000000005/0000000000000292.sstable/mc-292-big-Index.db. Found exception: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table767_field4_table767_index-1be394e10d4b11ebb700000000000005/0000000000000292.sstable/mc-292-big-Index.db])
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !ERR | scylla: [shard 0] sstable - Could not create SSTable component /var/lib/scylla/data/feeds/table634-b56f8de00d4a11ebb700000000000005/0000000000000340.sstable/mc-340-big-TOC.txt.tmp. Found exception: std::filesystem::__cxx11::filesystem_error (error system:24, filesystem error: open failed: Too many open files [/var/lib/scylla/data/feeds/table634-b56f8de00d4a11ebb700000000000005/0000000000000340.sstable/mc-340-big-TOC.txt.tmp])
Decoded backtrace:
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/fmt/format.h:2188
seastar::backtrace_buffer::append_backtrace() at /usr/include/fmt/format.h:2188
(inlined by) print_with_backtrace at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:751
seastar::print_with_backtrace(char const*) at /usr/include/fmt/format.h:2188
sigabrt_action at /usr/include/fmt/format.h:2188
(inlined by) operator() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:3451
(inlined by) _FUN at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:3447
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
non-virtual thunk to seastar::append_challenged_posix_file_impl::~append_challenged_posix_file_impl() at /usr/include/c++/10/bits/basic_string.h:323
seastar::shared_ptr<seastar::file_impl>::~shared_ptr() at /usr/include/fmt/format.h:2188
(inlined by) seastar::file::~file() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/file.hh:158
(inlined by) checked_file_impl::~checked_file_impl() at /jenkins/workspace/scylla-4.2/next/scylla/checked-file-impl.hh:30
(inlined by) seastar::shared_ptr_count_for<checked_file_impl>::~shared_ptr_count_for() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/shared_ptr.hh:463
(inlined by) seastar::shared_ptr_count_for<checked_file_impl>::~shared_ptr_count_for() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/shared_ptr.hh:463
seastar::shared_ptr<seastar::file_impl>::~shared_ptr() at /usr/include/fmt/format.h:2188
(inlined by) seastar::file::~file() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/file.hh:158
(inlined by) std::_Head_base<0ul, seastar::file, false>::~_Head_base() at /usr/include/c++/10/tuple:124
(inlined by) std::_Tuple_impl<0ul, seastar::file>::~_Tuple_impl() at /usr/include/c++/10/tuple:341
(inlined by) std::tuple<seastar::file>::~tuple() at /usr/include/c++/10/tuple:516
(inlined by) seastar::future_state<seastar::file>::clear() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:521
(inlined by) seastar::future_state<seastar::file>::clear() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:519
(inlined by) seastar::future_state<seastar::file>::~future_state() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:526
(inlined by) seastar::future<seastar::file>::~future() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1371
std::_Head_base<1ul, seastar::future<seastar::file>, false>::~_Head_base() at /usr/include/fmt/format.h:2188
(inlined by) std::_Tuple_impl<1ul, seastar::future<seastar::file> >::~_Tuple_impl() at /usr/include/c++/10/tuple:341
(inlined by) std::_Tuple_impl<0ul, seastar::future<seastar::file>, seastar::future<seastar::file> >::~_Tuple_impl() at /usr/include/c++/10/tuple:191
(inlined by) std::tuple<seastar::future<seastar::file>, seastar::future<seastar::file> >::~tuple() at /usr/include/c++/10/tuple:887
(inlined by) __invoke_impl<void, sstables::sstable::create_data()::<lambda(auto:180)>, std::tuple<seastar::future<seastar::file>, seastar::future<seastar::file> > > at /usr/include/c++/10/bits/invoke.h:60
(inlined by) __invoke<sstables::sstable::create_data()::<lambda(auto:180)>&, std::tuple<seastar::future<seastar::file>, seastar::future<seastar::file> > > at /usr/include/c++/10/bits/invoke.h:95
(inlined by) __apply_impl<sstables::sstable::create_data()::<lambda(auto:180)>&, std::tuple<std::tuple<seastar::future<seastar::file>, seastar::future<seastar::file> > >, 0> at /usr/include/c++/10/tuple:1723
(inlined by) apply<sstables::sstable::create_data()::<lambda(auto:180)>&, std::tuple<std::tuple<seastar::future<seastar::file>, seastar::future<seastar::file> > > > at /usr/include/c++/10/tuple:1734
(inlined by) operator() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1525
(inlined by) satisfy_with_result_of<seastar::future<T>::then_impl_nrvo<sstables::sstable::create_data()::<lambda(auto:180)>, seastar::future<> >::<lambda()>::<lambda(pr_type&, sstables::sstable::create_data()::<lambda(auto:180)>&, seastar::future_state<std::tuple<seastar::future<seastar::file>, seastar::future<seastar::file> > >&&)> mutable::<lambda()> > at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1986
(inlined by) operator() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:1524
(inlined by) run_and_dispose at /jenkins/workspace/scylla-4.2/next/scylla/seastar/include/seastar/core/future.hh:647
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
(inlined by) seastar::reactor::run() at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/reactor.cc:2715
seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::{lambda()#3}::operator()() const at /usr/include/fmt/format.h:2188
std::function<void ()>::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
(inlined by) seastar::posix_thread::start_routine(void*) at /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/posix.cc:60
?? ??:0
?? ??:0
And next coredump
2020-10-14 00:42:54.000: (CoreDumpEvent Severity.ERROR): node=Node longevity-5000-tables-4-2-db-node-8db60178-2 [34.251.79.55 | 10.0.3.32] (seed: False)
corefile_url=
https://storage.cloud.google.com/upload.scylladb.com/core.scylla.997.4a38a3472c5544a6a2cf10d9d6b2e6bb.106487.1602636174000000/core.scylla.997.4a38a3472c5544a6a2cf10d9d6b2e6bb.106487.1602636174000000.gz
backtrace= PID: 106487 (scylla)
UID: 997 (scylla)
GID: 1001 (scylla)
Signal: 6 (ABRT)
Timestamp: Wed 2020-10-14 00:42:54 UTC (4min 31s ago)
Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-15,17-31 --lock-memory=1
Executable: /opt/scylladb/libexec/scylla
Control Group: /
Boot ID: 4a38a3472c5544a6a2cf10d9d6b2e6bb
Machine ID: 93f219319dd5bdb42d9f1c8f2e23d329
Hostname: longevity-5000-tables-4-2-db-node-8db60178-2
Coredump: /var/lib/systemd/coredump/core.scylla.997.4a38a3472c5544a6a2cf10d9d6b2e6bb.106487.1602636174000000
Message: Process 106487 (scylla) of user 997 dumped core.
Stack trace of thread 106494:
#0 0x00007f2568a5d9e5 raise (libc.so.6)
#1 0x00007f2568a4694d abort (libc.so.6)
#2 0x00007f2568a46769 __assert_fail_base.cold (libc.so.6)
#3 0x00007f2568a55e76 __assert_fail (libc.so.6)
#4 0x0000000002e067d0 _ZThn56_N7seastar33append_challenged_posix_file_implD0Ev (scylla)
#5 0x0000000000ee15b9 _ZN7seastar10shared_ptrINS_9file_implEED4Ev (scylla)
#6 0x0000000000f0677e _ZN7seastar10shared_ptrINS_9file_implEED4Ev (scylla)
#7 0x0000000001262b58 _ZNSt10_Head_baseILm1EN7seastar6futureIJNS0_4fileEEEELb0EED4Ev (scylla)
#8 0x0000000002e8ea48 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla)
#9 0x0000000002e8edbf _ZN7seastar7reactor14run_some_tasksEv (scylla)
#10 0x0000000002ec5e9e _ZN7seastar7reactor14run_some_tasksEv (scylla)
#11 0x0000000002ed589b _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
#12 0x0000000002e59b1e _ZNKSt8functionIFvvEEclEv (scylla)
#13 0x00007f256949f432 start_thread (libpthread.so.0)
#14 0x00007f2568b22913 __clone (libc.so.6)
Stack trace of thread 106539:
#0 0x00007f25694a99ac read (libpthread.so.0)
#1 0x000000000311d737 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x000000000311d998 operator() (scylla)
#3 0x0000000002e59b1e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f256949f432 start_thread (libpthread.so.0)
#5 0x00007f2568b22913 __clone (libc.so.6)
Stack trace of thread 106499:
#0 0x0000000002ecaf11 _ZN7seastar17smp_message_queue19flush_request_batchEv (scylla)
#1 0x0000000002efe3b6 _ZN7seastar7reactor10smp_pollfn4pollEv (scylla)
#2 0x0000000002e834bd _ZN7seastar7reactor9poll_onceEv (scylla)
#3 0x0000000002ec5ed1 _ZNKSt8functionIFbvEEclEv (scylla)
#4 0x0000000002ed589b _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
#5 0x0000000002e59b1e _ZNKSt8functionIFvvEEclEv (scylla)
#6 0x00007f256949f432 start_thread (libpthread.so.0)
#7 0x00007f2568b22913 __clone (libc.so.6)
Stack trace of thread 106547:
#0 0x00007f25694a99ac read (libpthread.so.0)
#1 0x000000000311d737 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x000000000311d998 operator() (scylla)
#3 0x0000000002e59b1e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f256949f432 start_thread (libpthread.so.0)
#5 0x00007f2568b22913 __clone (libc.so.6)
Stack trace of thread 106542:
#0 0x00007f25694a99ac read (libpthread.so.0)
#1 0x000000000311d737 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x000000000311d998 operator() (scylla)
#3 0x0000000002e59b1e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f256949f432 start_thread (libpthread.so.0)
#5 0x00007f2568b22913 __clone (libc.so.6)
Stack trace of thread 106543:
#0 0x00007f25694a99ac read (libpthread.so.0)
#1 0x000000000311d737 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x000000000311d998 operator() (scylla)
#3 0x0000000002e59b1e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f256949f432 start_thread (libpthread.so.0)
#5 0x00007f2568b22913 __clone (libc.so.6)
Stack trace of thread 106545:
#0 0x00007f25694a99ac read (libpthread.so.0)
#1 0x000000000311d737 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x000000000311d998 operator() (scylla)
#3 0x0000000002e59b1e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f256949f432 start_thread (libpthread.so.0)
#5 0x00007f2568b22913 __clone (libc.so.6)
Stack trace of thread 106501:
#0 0x0000000001f6fb53 _ZNK7seastar17future_state_base3any9availableEv (scylla)
#1 0x0000000002e8ea48 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla)
#2 0x0000000002e8edbf _ZN7seastar7reactor14run_some_tasksEv (scylla)
#3 0x0000000002ec5e9e _ZN7seastar7reactor14run_some_tasksEv (scylla)
#4 0x0000000002ed589b _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
#5 0x0000000002e59b1e _ZNKSt8functionIFvvEEclEv (scylla)
#6
download_instructions=
gsutil cp gs://upload.scylladb.com/core.scylla.997.4a38a3472c5544a6a2cf10d9d6b2e6bb.106487.1602636174000000/core.scylla.997.4a38a3472c5544a6a2cf10d9d6b2e6bb.106487.1602636174000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.997.4a38a3472c5544a6a2cf10d9d6b2e6bb.106487.1602636174000000.gz
after that node continue to restarting and trigger the coredumps and stay DN
Db node log:
longevity-5000-tables-4-2-db-node-8db60178-2.zip
aleksbykov
All 52 comments
Monitoring stack for that timeframe: http://3.251.78.185:3000/d/9bMwf-cGz/scylla-per-server-metrics-nemesis-master?orgId=1&from=1602633680312&to=1602637867781&var-by=instance&var-cluster=&var-dc=All&var-node=All&var-shard=All&var-sct_tags=DisruptionEvent&var-sct_tags=CoreDumpEvent
aleksbykov
on 14 Oct 2020
what's the compaction strategy of those tables? STCS or LCS?
looks like another problem caused by lack of off-strategy. tables potentially have lots of small sstables after bootstrapping, and resharding make it worse when those sstables are split. LCS tries to minimize this problem, but not STCS. So I assume STCS is being used here
raphaelsc
on 14 Oct 2020
the log seems to back up my theory, lots of resharding logs like this:
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: [shard 7] compaction - Resharded 2 sstables to [/var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-250-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-279-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-254-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-255-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-222-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-278-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-251-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-223-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-351-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-319-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-317-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-318-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-350-big-Data.db:level=0, /var/lib/scylla/data/feeds/table783-290c6ac00d4b11ebb700000000000005/mc-316-big-Data.db:level=0, ]. 681kB to 10MB (~1560% of original) in 473ms = 22MB/s. ~6784 total partitions merged to 6666.
lots of small SHARED sstables being resharded into many UNSHARED sstables
raphaelsc
on 14 Oct 2020
what's the compaction strategy of those tables? STCS or LCS?
SizeTieredCompactionStrategy
aleksbykov
on 15 Oct 2020
I think this is a regression ... probably related to offline strategy in some manner.
Do we reshard each table by itself or do we reshard tables together.
We used to allow shards to use non resharded sstables and in the background reshard
Have we moved to a mode we reshard everything in parallel.
slivne
on 15 Oct 2020
I pushed this to 4.3 and we can consider backport
We do have a test with 1000 Tables that passes - right @roydahan / @aleksbykov ?
slivne
on 15 Oct 2020
@slivne i didn't run with 1k tables. We have passed with 1k keyspaces( with 1 table per keyspace)
I am continue to investigate the job with 5000 tables. I think there are some more issues
- All 5 added nodes failed to run the nodetool command because could n't connect
- After 4th batch of c-s commands, A lot of errors in c-s commands
http://3.251.78.185:3000/d/9bMwf-cGz/scylla-per-server-metrics-nemesis-master?orgId=1&from=1602595235201&to=1602730927509&var-by=instance&var-cluster=&var-dc=All&var-node=All&var-shard=All&var-sct_tags=CoreDumpEvent
aleksbykov
on 15 Oct 2020
Don't turn the machines off yet.
I'd like to check the open file descriptors limit on them.
bhalevy
on 15 Oct 2020
There are 2 or 3 issues here:
- The assert tiggered - should be fixed by https://github.com/scylladb/scylla-seastar/commit/f760efe0a0f1fbcbdf19d843357adc26f4ba271d
(scylla-4.2 https://github.com/scylladb/scylla/commit/a9109f068c72ffa72e4413b11a3452f85713bba9) - Why did we run out of file descriptors?
- Is there a leak of open file descriptors? or maybe we just consume too much in a burst.
- If it's the latter, we may consider to increase the limit on the AMI.
bhalevy
on 15 Oct 2020
| longevity-5000-tables-4-2-db-node-8db60178-1 | eu-west-1a | 34.255.6.205 |
| longevity-5000-tables-4-2-monitor-node-8db60178-1 | eu-west-1a | 3.251.78.185 |
| longevity-5000-tables-4-2-db-node-8db60178-2 | eu-west-1a | 34.251.79.55 |
| longevity-5000-tables-4-2-db-node-8db60178-3 | eu-west-1a | 34.241.32.109 |
| longevity-5000-tables-4-2-db-node-8db60178-4 | eu-west-1a | 54.155.104.6 |
| longevity-5000-tables-4-2-db-node-8db60178-5 | eu-west-1a | 54.171.152.94 |
| longevity-5000-tables-4-2-db-node-8db60178-6 | eu-west-1a | 63.34.20.59 |
[centos@longevity-5000-tables-4-2-db-node-8db60178-1 ~]$ sysctl fs.file-max
fs.file-max = 25148194
[centos@longevity-5000-tables-4-2-db-node-8db60178-1 ~]$ ps -ef | grep bin/scyll[a]
scylla 3951 1 99 Oct13 ? 15-19:59:03 /usr/bin/scylla --blocked-reactor-notify-ms 500 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-15,17-31 --lock-memory=1
[centos@longevity-5000-tables-4-2-db-node-8db60178-1 ~]$ grep files /proc/3951/limits
Max open files 800000 800000 files
[centos@longevity-5000-tables-4-2-db-node-8db60178-1 ~]$ sudo lsof -p 3951 | wc -l
100424
that's pretty high, and those are mostly sstables files:
[centos@longevity-5000-tables-4-2-db-node-8db60178-1 ~]$ sudo lsof -p 3951 | grep -c '.*/mc-'
96678
and most of them are from user data:
[centos@longevity-5000-tables-4-2-db-node-8db60178-1 ~]$ sudo lsof -p 3951 | grep -c data/feeds
96480
@raphaelsc is it possible resharding keeps the output sstables open for too long, leading to having
#sstables * #shards opened at the same time? With 30 shards this could easily overwhelm the 800K limit with 100K sstables files.
bhalevy
on 15 Oct 2020
@raphaelsc is it possible resharding keeps the output sstables open for too long, leading to having
#sstables * #shardsopened at the same time? With 30 shards this could easily overwhelm the 800K limit with 100K sstables files.
the output sstables will be kept opened at their destination shards anyway. worth checking the amount of files stored across the 5000 tables. are the nodes still alive? it looks like lack of off-strategy compaction may lead to this as streaming could lead to creation of lots of small sstables
raphaelsc
on 15 Oct 2020
just noticed you shared it in another msg, @bhalevy
yes, 96k files is just too much, old and new resharding wouldn't be able to handle this scenario as a change in msb setting will cause the ssts to be potentially shared by all shards, so the amount of fds required after resharding is # of input sstables * # of shards.
what we can do to solve this is doing off-strategy compaction (#5226) on streaming, to coalesce those many tiny ssts into a large one.
raphaelsc
on 15 Oct 2020
@raphaelsc is it possible resharding keeps the output sstables open for too long, leading to having
#sstables * #shardsopened at the same time? With 30 shards this could easily overwhelm the 800K limit with 100K sstables files.the output sstables will be kept opened at their destination shards anyway. worth checking the amount of files stored across the 5000 tables. are the nodes still alive? it looks like lack of off-strategy compaction may lead to this as streaming could lead to creation of lots of small sstables
@raphaelsc
https://github.com/scylladb/scylla/issues/7439#issuecomment-709216167
aleksbykov
on 15 Oct 2020
what I noticed while reading the new resharding code is that we wait until all jobs, on a given shard, to complete before releasing all the input ssts, so preventing fds and disk space from being incrementally released. So we have a bug.
let's do some math:
100k fds opened means 50k ssts as both index and data is opened for each.
to limit memory consumption, up to 32 ssts are resharded together. so we have ~1562 resharding jobs producing up to 30 ssts (# of shards) each, producing a max of ~47k ssts (~94k fds).
I expect that the max amount of fds, without incrementally releasing the fds, would be ~200k (input + output fds). We probably have another unknown bug. It may be due to a combination of resharding and reshape. We may be incorrectly opening a fd of a shared sst in more than one shard, they're supposed to be shared. I'll investigate this further.
raphaelsc
on 15 Oct 2020
On Thu, Oct 15, 2020 at 8:59 PM Raphael Carvalho notifications@github.com
wrote:
what I noticed while reading the new resharding code is that we wait until
all jobs, on a given shard, completes before releasing all the input ssts,
so preventing fds and disk space from being incrementally released. So we
have a bug.Do we do the resharding while the node is accepting traffic or is it
"offline" till resharding ends ?
Do we do resharding of single old sstable all on its own - or do we mix
multiple old sstables and reshard them together and create a set of sharded
sstables for them all. Example 10 old sstables on a system of 32 shards
(that every input sstable is split on all 32 shards) - will they be
translated to 320 sstables or less ?
Is compaction enabled while resharding is ongoing ?
- do we compact new sharded sstables as resharding is progressing
- or accumulate them all and only then enable compaction
If we are doing both of the above - no compaction + resharding each sstable
on its own - its clear the issue of not releasing the original sstable is
not the problem.
let's do some math:
>
100k fds opened means 50k ssts as both index and data is opened for each.
to limit memory consumption, up to 32 ssts are resharded together. so we
have ~1562 resharding jobs producing up to 30 ssts (# of shards) each,
producing a max of ~47k ssts (~94k fds).I expect that the max amount of fds, without incrementally releasing the
fds, would be ~200k (input + output fds). We probably have another unknown
bug. It may be due to a combination of resharding and reshape. We may be
incorrectly opening a fd of a shared sst in more than one shard, they're
supposed to be shared. I'll investigate this further.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scylladb/scylla/issues/7439#issuecomment-709494343,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA2OCCABBVIO53QS7KQZAALSK4Z7PANCNFSM4SQ4PDAA
.
slivne
on 15 Oct 2020
Do we do the resharding while the node is accepting traffic or is it "offline" till resharding ends ?
Resharding is now always done off-strategy. On boot, node is only made online after resharding completes. On refresh, uploaded sstables will be resharded off-strategy before the output is made available to the sstable set.
Do we do resharding of single old sstable all on its own - or do we mix multiple old sstables and reshard them together and create a set of sharded sstables for them all. Example 10 old sstables on a system of 32 shards (that every input sstable is split on all 32 shards) - will they be translated to 320 sstables or less ?
Like I explained, resharding compacts 32 (max_threshold) sstables together, exactly to mitigate this problem of it spitting out lots of small ssts, which can potentially exhaust resources. So 10k shared ssts, on a 30 shard system, will be translated into ~9376 unshared ssts.
Is compaction enabled while resharding is ongoing ? - do we compact new sharded sstables as resharding is progressing - or accumulate them all and only then enable compaction If we are doing both of the above - no compaction + resharding each sstable on its own - its clear the issue of not releasing the original sstable is not the problem.
we only make new ssts, created by resharding, available on completion.
Not releasing resources of input ssts is clearly contributing significantly to this problem, as resharding is serialized on each shard, but we only release all consumed resources at the very end. But there's definitely something else that we're missing here. It may be a bad distribution of jobs among shards, or even resharding incorrectly opening fd of shared ssts at more than one shard.
raphaelsc
on 15 Oct 2020
On Thu, Oct 15, 2020 at 10:25 PM Raphael Carvalho notifications@github.com
wrote:
Do we do the resharding while the node is accepting traffic or is it
"offline" till resharding ends ?Resharding is now always done off-strategy. On boot, node is only made
online after resharding completes. On refresh, uploaded sstables will be
resharded off-strategy before the output is made available to the sstable
set.Do we do resharding of single old sstable all on its own - or do we mix
multiple old sstables and reshard them together and create a set of sharded
sstables for them all. Example 10 old sstables on a system of 32 shards
(that every input sstable is split on all 32 shards) - will they be
translated to 320 sstables or less ?Like I explained, resharding compacts 32 (max_threshold) sstables
together, exactly to mitigate this problem of it spitting out lots of small
ssts, which can potentially exhaust resources. So 10k shared ssts, on a 30
shard system, will be translated into ~9376 unshared ssts.Is compaction enabled while resharding is ongoing ? - do we compact new
sharded sstables as resharding is progressing - or accumulate them all and
only then enable compaction If we are doing both of the above - no
compaction + resharding each sstable on its own - its clear the issue of
not releasing the original sstable is not the problem.we only make new ssts, created by resharding, available on completion.
Not releasing resources of input ssts is clearly contributing
significantly to this problem, as resharding is serialized on each shard,
but we only release all consumed resources at the very end. But there's
definitely something else that we're missing here. It may be a bad
distribution of jobs among shards, or even resharding incorrectly opening
fd of shared ssts are more than one shard.
Thanks - BTW - we only need a single node to try and reproduce this :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scylladb/scylla/issues/7439#issuecomment-709542213,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA2OCCFHM6SZU44PJOKJIZ3SK5EEFANCNFSM4SQ4PDAA
.
slivne
on 15 Oct 2020
😄1
Could i terminate the cluster?
aleksbykov
on 16 Oct 2020
Raphael,
If you want to check something on the nodes - please comment or it will be
scratched
If it helps you - you can actually use a single node and restart it with
additional logging with a different msb setting (forcing resharding) and
see if the issue reproduces / and get logs helping you to debug this
@Raphael S. Carvalho raphaelsc@scylladb.com please comment
Shlomi
On Fri, Oct 16, 2020 at 10:48 AM aleksbykov notifications@github.com
wrote:
Could i terminate the cluster?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scylladb/scylla/issues/7439#issuecomment-709885763,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA2OCCHISGXZEKPAMGPLMZDSK73EJANCNFSM4SQ4PDAA
.
slivne
on 16 Oct 2020
Could i terminate the cluster?
yes, please.
raphaelsc
on 16 Oct 2020
i'll try to reproduce the issue now...
raphaelsc
on 16 Oct 2020
figured it out from the log itself...
$ cat messages.log | grep "data/feeds" | grep "Resharded *[0-9] sstables to" | awk '{s[$11]+=1; ssts+=$11 } END { print "input ssts: " ssts; for (i in s) {print i,s[i]; output+=(s[i]*30)} print "estimated output ssts: " output; }'
input ssts: 30233
1 12113
2 8087
3 173
4 147
5 113
6 35
7 8
8 1
estimated output ssts: 620310
620k ssts translates into ~1.24m fds (index + data), which explains the fd exhaustion.
the problem indeed happens because we're resharding too little ssts together, which means:
- bug in the new resharding code when creating jobs across shards
- or max threshold is set to a low value. anyway we possibly not trust table's max threshold to avoid running into resource exhaustion like this one.
I am suspecting we do really have a bug.
For comparison, if we had resharded the 30k ssts in groups of 32 ssts, we'd only have needed ~56k fds instead.
raphaelsc
on 16 Oct 2020
[centos@longevity-5000-tables-4-2-db-node-8db60178-2 feeds]$ ls -lh * | grep "Data" | wc -l
548257
This problem is triggered with a high number of tables, each containing only a couple of ssts.
In this test, there are 5000 tables, averaging 6 ssts in each.
To take advantage of all shards, resharding tries to fairly spread the work among all shards, meaning those 6 ssts will each end up with in a different shard to be resharded at. On a 30 shard system, resharding could produce 180 ssts from those initial 6 ssts. Multiply 180 by 5000, and resharding will potentially produce ~900k ssts.
Resharding optimizes for the case where a table contains many ssts, but for this scenario where user has thousands of tables, each containing only a couple of ssts, we end up with this problem.
If we had resharded those 6 ssts in a single shard, we'd end up with 150k ssts rather than 900k.
I'll work a solution to fix this efficiently, without compromising resharding performance.
raphaelsc
on 17 Oct 2020
@avikivity this change of resharding is part of the offline strategy and it was introduced in 4.2 - any view on this ?
@raphaelsc can you elaborate on what is the solution you are working on - lets agree on it - before you code
slivne
on 18 Oct 2020
IMO this is a blocker. While large numbers of tables are rare, and resharding is even more rare, the impact is too severe.
avikivity
on 18 Oct 2020
On Sun, Oct 18, 2020 at 4:35 AM Shlomi Livne notifications@github.com
wrote:
@avikivity https://github.com/avikivity this change of resharding is
part of the offline strategy and it was introduced in 4.2 - any view on
this ?this issue exists prior to offstrategy changes, as old resharding would
also spread the work in a similar fashion. so probably not a blocker for 4.2
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scylladb/scylla/issues/7439#issuecomment-711129435,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKYA4YEMHGH73T7CA7AORLSLKLC3ANCNFSM4SQ4PDAA
.
raphaelsc
on 19 Oct 2020
If this is so then it's not a blocker. @aleksbykov is this a new test? Or did it pass in 4.1?
avikivity
on 19 Oct 2020
It is not new, it didn't passed on 4.1
aleksbykov
on 19 Oct 2020
Did it ever pass?
avikivity
on 19 Oct 2020
This specific "resharding" nemesis failed on a different issue in 4.1 (the service got stuck during stop), so it never reached to actual resharding.
In 4.0 we didn't have this nemesis running and it was the first version of this test that included nemesis.
roydahan
on 19 Oct 2020
So, we had a successful run of this test?
avikivity
on 19 Oct 2020
If you mean resharding of 5000 tables, so no we don't have a successful run of it.
roydahan
on 19 Oct 2020
Ok. @slivne we can degrade this from blocker status.
avikivity
on 19 Oct 2020
@raphaelsc we still need to solve it for 4.3 :)
Please provide the logic of what the change you will introduce so we can make sure we are on an agreed path to a solution
slivne
on 19 Oct 2020
👍1
Solution is based on this:
- The more tables you have, the less shards you want to reshard them at.
- Determine upfront the amount of shards each table will be resharded at.
- # of shards for each table:
S / T
where T is # of tables and
S is # of shards.
NOTE: formula needs some tweak for sure, keeping it simple here.
We know that tables are loaded (therefore resharded) in parallel (throttled with a semaphore though).
So, when resharding a table T1, reshard T1 at N shards (S / T) only.
For example,
with S=20 and T=10, each table will be resharded at 2 shards
with S=10 and T=1, that unique table will be resharded at all 10 shards
with S=10 and T=100, all tables will be resharded at a single shard
Which shard(s) will a table be resharded at?
- To achieve fairness:
1) That will be based on the amount of resharding work each shard is currently operating
2) Each table will choose the shard(s) with the least amount of work
- To achieve fairness:
So if T=1000 and S=30, each table with 1 sstable per shard, this solution will reduce the amount of ssts after resharding from
1000(T)30(ssts)30(S) = 900k sstables
to
1000(T)*30(ssts) = 30k sstables
That's because each table is resharded at 1 shard (S / T) only, which produces at most S sstables.
raphaelsc
on 19 Oct 2020
Thanks @raphaelsc . I like the overall direction.
I'd like to point it that you calculation about the number of additional sstables assumes a small number of sstables per table.
If we have more of them we may end up with #shards new sstables for each group of up to max_threshold sstables, and with #shards==30 and max_threshold==32 this would roughly double the number of sstables, unless we can delete the retired sstables incrementally as we go.
Also, for resharding purposes, it might make sense to consider #shards rather than max_threshold for grouping together sstables for compaction.
bhalevy
on 20 Oct 2020
Thanks @raphaelsc . I like the overall direction.
I'd like to point it that you calculation about the number of additional sstables assumes a small number of sstables per table.
If we have more of them we may end up with#shardsnew sstables for each group of up tomax_thresholdsstables, and with#shards==30andmax_threshold==32this would roughly double the number of sstables, unless we can delete the retired sstables incrementally as we go.
With T=1, each with 100 sstables per shard, S=30,
- that unique table is assigned all shards
- but that's not a problem because each shard will reshard sstables in groups of 32, which cancels the S factor (if 32 >= S) into the # of files.
What led to the problem described in this issue is high # of tables with a small # of sstables. High # of sstables has never been a problem because we're able to reshard them in groups, cancelling the S factor which would otherwise increase the # of files by a factor of S.
BTW, not incrementally deleting resharded ssts as we go is a regression, AFAICT.
Also, for resharding purposes, it might make sense to consider
#shardsrather thanmax_thresholdfor grouping together sstables for compaction.
Makes sense. That needs to be # of shards at least.
raphaelsc
on 20 Oct 2020
@avikivity please review suggestion
slivne
on 21 Oct 2020
with S=20 and T=10, each table will be resharded at 2 shards with S=10 and T=1, that unique table will be resharded at all 10 shards with S=10 and T=100, all tables will be resharded at a single shard
For the first example, if one table is large and nine are small, we'll run out of work on the the 18 shards quickly, and two shards will keep working. This results in the process taking 10X as long.
So we need to take the table size into account.
avikivity
on 25 Oct 2020
Resharding a large table on many shards isn't a problem, because there can only be a small number of large tables. So it's okay to treat small and large tables differently.
avikivity
on 25 Oct 2020
So we can adjust the formula:
N =
S if the table size is larger than 10% of the total storage used
ceil(S/T) otherwise
avikivity
on 25 Oct 2020
Proposal looks good to me.
avikivity
on 25 Oct 2020
@bhalevy lets queue this to be fixed in 4.4
slivne
on 20 Dec 2020
seeing very similar scenario on 4.3.rc3 during RestartNodeWithResharding for job 5000 tables:
2020-12-26T09:33:16+00:00 longevity-5000-tables-4-3-db-node-b3a128e5-2 !INFO | scylla: Scylla version 4.3.rc3-0.20201217.5bd52e4db with build-id fbc87a9195ba296dbec0f662bc159d0117d63e2c starting ...
2020-12-26T09:33:16+00:00 longevity-5000-tables-4-3-db-node-b3a128e5-2 !INFO | scylla: command used: "/usr/bin/scylla --blocked-reactor-notify-ms 500 --log-to-syslog 1 --log-to-stdout 0 --default-log
-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-15,17-31 --lock-memory=1"
2020-12-26T09:33:16+00:00 longevity-5000-tables-4-3-db-node-b3a128e5-2 !INFO | scylla: parsed command line options: [blocked-reactor-notify-ms: 500, log-to-syslog: 1, log-to-stdout: 0, default-log-le
vel: info, network-stack: posix, io-properties-file: /etc/scylla.d/io_properties.yaml, cpuset: 1-15,17-31, lock-memory: 1]
2020-12-26T09:33:20+00:00 longevity-5000-tables-4-3-db-node-b3a128e5-2 !CRIT | systemd-coredump: Process 29313 (scylla) of user 997 dumped core.#012#012Stack trace of thread 29341:#012#0 0x000000000
12a47af _ZNSt15_Deque_iteratorIN8sstables11compression17segmented_offsets6bucketERS3_PS3_EpLEl (scylla)#012#1 0x00000000012a5d92 _ZN8sstables11compression17segmented_offsets6writer9push_backEm (scylla)#
012#2 0x0000000000e2304a _ZN7seastar9data_sink3putENS_16temporary_bufferIcEE (scylla)#012#3 0x0000000000e236bc _ZN7seastar13output_streamIcE5flushEv (scylla)#012#4 0x0000000000e23b5a _ZN7seastar13outp
ut_streamIcE5closeEv (scylla)#012#5 0x0000000001178edd _ZN8sstables11file_writer5closeEv (scylla)#012#6 0x0000000001272e26 operator()<std::unique_ptr<sstables::file_writer> > (scylla)#012#7 0x00000000
01273739 _ZN8sstables2mc6writerD0Ev (scylla)#012#8 0x000000000131e3c2 _ZN8sstables17compaction_writerD4Ev (scylla)#012#9 0x000000000133ffdd _ZN20flat_mutation_reader4impl17consume_in_threadI35stable_fl
attened_mutations_consumerI22compact_for_compactionIN8sstables25compacting_sstable_writerE33noop_compacted_fragments_consumerEENS_9no_filterEEEDaT_T0_NSt6chrono10time_pointIN7seastar12lowres_clockENSC_8d
urationIlSt5ratioILl1ELl1000EEEEEE (scylla)#012#10 0x0000000002df514d _ZNK7seastar20noncopyable_functionIFvvEEclEv (scylla)#012#012Stack trace of thread 29407:#012#0 0x00007f2f77a5f9ac read (libpthread.
so.0)#012#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)#012#2 0x0000000002dc7e58 operator() (scylla)#012#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (
scylla)#012#4 0x00007f2f77a55432 start_thread (libpthread.so.0)#012#5 0x00007f2f76dd1913 __clone (libc.so.6)#012#012Stack trace of thread 29436:#012#0 0x00007f2f77a5f9ac read (libpthread.so.0)#012#1
0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)#012#2 0x0000000002dc7e58 operator() (scylla)#012#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)#012#4
0x00007f2f77a55432 start_thread (libpthread.so.0)#012#5 0x00007f2f76dd1913 __clone (libc.so.6)#012#012Stack trace of thread 29438:#012#0 0x00007f2f77a5f9ac read (libpthread.so.0)#012#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)#012#2 0x0000000002dc7e58 operator() (scylla)#012#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)#012#4 0x00007f2f77a55432 start_thread (libpthread.so.0)#012#5 0x00007f2f76dd1913 __clone (libc.so.6)#012#012Stack trace of thread 29428:#012#0 0x00007f2f77a5f9ac read (libpthread.so.0)#012#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)#012#2 0x0000000002dc7e58 operator() (scylla)#012#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)#012#4 0x00007f2f77a55432 start_thread (libpthread.so.0)#012#5 0x00007f2f76dd1913 __clone (libc.so.6)#012#012Stack trace of thread 29416:#012#0 0x00007f2f77a5f9ac read (libpthread.so.0)#012#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)#012#2 0x0000000002dc7e58 operator() (scylla)#012#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)#012#4 0x00007f2f77a55432 start_thread (libpthread.so.0)#012#5 0x00007f2f76dd1913 __clone (libc.so.6)#012#012Stack trace of thread 29425:#012#0 0x00007f2f77a5f9ac read (libpthread.so.0)#012#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)#012#2 0x0000000002dc7e58 operator() (scylla)#012#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)#012#4 0x00007f2f77a55432 start_thread (libpthread.so.0)#012#5 0x00007f2f76dd1913 __clone (libc.so.6)#012#012Stack trace of thread 29320:#012#0 0x0000000002b4c9f2 process_queue<2, seastar::smp_message_queue::process_incoming()::<lambda(seastar::smp_message_queue::work_item*)> > (scylla)#012#1 0x0000000002b9ab6e _ZN7seastar17smp_message_queue16process_incomingEv (scylla)#012#2 0x0000000002b9ac16 _ZN7seastar7reactor10smp_pollfn4pollEv (scylla)#012#3 0x0000000002b458bd _ZN7seastar7reactor9poll_onceEv (scylla)#012#4 0x0000000002b97139 _ZNKSt8functionIFbvEEclEv (scylla)#012#5 0x0000000002ba88bb _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)#012#6 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)#012#7 0x00007f2f77a55432 start_thread (libpthread.so.0)#012#8 0x00007f2f76dd1913 __clone (libc.so.6)#012#012Stack trace of thread 29385:#012#0 0x0000000002b9abde _ZN7seastar3smp11poll_queuesEv (scylla)#012#1 0x0000000002b9ac16 _ZN7seastar7reactor10smp_pollfn4pollEv (scylla)#012#2 0x0000000002b458bd _ZN7seastar7reactor9poll_onceEv (scylla)#012#3 0x0000000002b97139 _ZNKSt8functionIFbvEEclEv (scylla)#012#4 0x0000000002ba88bb _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)#012#5 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)#012#6 0x00007f2f77a55432 start_thread (libpthread.so.0)#012#7 0x00007f2f76dd1913 __clone (libc.so.6)#012#012Stack trace of thread 29363:#012#0 0x0000000002b06b6e _ZN7seastar6memory10small_pool16add_more_objectsEv (scylla)#012#1 0x0000000002b08d8d _ZN7seastar6memory10small_pool8allocateEv (scylla)#012#2 0x0000000000efbda4 _ZSt11make_uniqueIN17mutation_fragment4dataEJ13reader_permitEENSt9_MakeUniqIT_E15__single_objectEDpOT0_ (scylla)#012#3 0x00000000012caeb1 _ZN8sstables17mp_row_consumer_m15consume_row_endEv (scylla)#012#4 0x00000000012d7eb7 _ZN8sstables27data_consume_rows_context_m16do_process_stateERN7seastar16temporary_bufferIcEE (scylla)#012#5 0x00000000012f103a _ZN8sstables27data_consume_rows_context_m13process_stateERN7seastar16temporary_bufferIcEE (scylla)#012#6 0x00000000012f21cb _ZZZN8sstables23sstable_mutation_readerINS_27data_consume_rows_context_mENS_17mp_row_consumer_mEE11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS4_8durationIlSt5ratioILl1ELl1000EEEEEEENKUlvE1_clEvENKUlvE0_clEv (scylla)#012#7 0x00000000012f5464 _ZZN8sstables23sstable_mutation_readerINS_27data_consume_rows_context_mENS_17mp_row_consumer_mEE11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS4_8durationIlSt5ratioILl1ELl1000EEEEEEENKUlvE1_clEv (scylla)#012#8 0x00000000012f6168 _ZN7seastar8futurizeINS_6futureIvEEE6invokeIRZN8sstables23sstable_mutation_readerINS5_27data_consume_rows_context_mENS5_17mp_row_consumer_mEE11fill_bufferENSt6chrono10time_pointINS_12lowres_clockENSA_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_JEEES2_OT_DpOT0_ (scylla)#012#9 0x00000000010f41a9 _ZN20flat_mutation_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS0_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)#012#10 0x00000000010f5f24 _ZN24mutation_fragment_mergerI22mutation_reader_mergerE5fetchENSt6chrono10time_pointIN7seastar12lowres_clockENS2_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)#012#11 0x00000000010f6337 repeat<combined_mutation_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > (scylla)#012#12 0x0000000001308dbf _ZN20flat_mutation_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS0_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)#012#13 0x000000000256ad42 _ZZZZN15mutation_writer11feed_writerINS_37shard_based_splitting_mutation_writerEEEN7seastar6futureIvEEO20flat_mutation_readerOT_ENKUlRS5_RS1_E_clES9_SA_ENKUlvE_clEvENKUlvE0_clEv (scylla)#012#14 0x000000000256b48d _ZN7seastar8internal14do_until_stateIZZZN15mutation_writer11feed_writerINS2_37shard_based_splitting_mutation_writerEEENS_6futureIvEEO20flat_mutation_readerOT_ENKUlRS7_RS4_E_clESB_SC_ENKUlvE_clEvEUlvE_ZZZNS3_IS4_EES6_S8_SA_ENKSD_clESB_SC_ENKSE_clEvEUlvE0_E15run_and_disposeEv (scylla)#012#15 0x0000000002b57568 _ZN7seastar7reactor9run_tasksERNS0_10task_queueE (scylla)#012#16 0x0000000002b5781f _ZN7seastar7reactor14run_some_tasksEv (scylla)#012#17 0x0000000002b97106 _ZN7seastar7reactor14run_some_tasksEv (scylla)#012#18 0x0000000002ba88bb _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)#012#19 0x0000000002b2148e _ZNK
2020-12-26T09:33:23+00:00 longevity-5000-tables-4-3-db-node-b3a128e5-2 !INFO | scylla: [shard 0] init - installing SIGHUP handler
and the coredump:
PID: 2274 (bash)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Thu 2020-12-24 15:42:20 UTC (2 days ago)
Command Line: /bin/bash /tmp/tmp4xcfyddo
Executable: /usr/bin/bash
Control Group: /system.slice/scylla-image-setup.service
Unit: scylla-image-setup.service
Slice: system.slice
Boot ID: e820ea02e266416ba6569522084ed3c7
Machine ID: cc2c86fe566741e6a2ff6d399c5d5daa
Hostname: longevity-5000-tables-4-3-db-node-b3a128e5-2
Message: Process 2274 (bash) of user 0 dumped core.
Stack trace of thread 2274:
#0 0x00007f4c9de5b657 kill (libc.so.6)
#1 0x0000000000440846 kill_pid (bash)
#2 0x000000000047309e kill_builtin (bash)
#3 0x000000000042f35f execute_builtin.isra.2 (bash)
#4 0x0000000000431449 execute_simple_command (bash)
#5 0x0000000000432353 execute_command_internal (bash)
#6 0x0000000000433d3e execute_command (bash)
#7 0x000000000041e375 reader_loop (bash)
#8 0x000000000041c9de main (bash)
#9 0x00007f4c9de47555 __libc_start_main (libc.so.6)
#10 0x000000000041d47a _start (bash)
PID: 25805 (scylla)
UID: 997 (scylla)
GID: 1001 (scylla)
Signal: 11 (SEGV)
Timestamp: Sat 2020-12-26 09:11:38 UTC (24h ago)
Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-15,17-31 --lock-memory=1
Executable: /opt/scylladb/libexec/scylla
Control Group: /
Boot ID: e820ea02e266416ba6569522084ed3c7
Machine ID: cc2c86fe566741e6a2ff6d399c5d5daa
Hostname: longevity-5000-tables-4-3-db-node-b3a128e5-2
Message: Process 25805 (scylla) of user 997 dumped core.
Stack trace of thread 25833:
#0 0x00000000012a47af _ZNSt15_Deque_iteratorIN8sstables11compression17segmented_offsets6bucketERS3_PS3_EpLEl (scylla)
#1 0x00000000012a5d92 _ZN8sstables11compression17segmented_offsets6writer9push_backEm (scylla)
#2 0x0000000000e2304a _ZN7seastar9data_sink3putENS_16temporary_bufferIcEE (scylla)
#3 0x0000000000e236bc _ZN7seastar13output_streamIcE5flushEv (scylla)
#4 0x0000000000e23b5a _ZN7seastar13output_streamIcE5closeEv (scylla)
#5 0x0000000001178edd _ZN8sstables11file_writer5closeEv (scylla)
#6 0x0000000001272e26 operator()<std::unique_ptr<sstables::file_writer> > (scylla)
#7 0x0000000001273739 _ZN8sstables2mc6writerD0Ev (scylla)
#8 0x000000000131e3c2 _ZN8sstables17compaction_writerD4Ev (scylla)
#9 0x000000000133ffdd _ZN20flat_mutation_reader4impl17consume_in_threadI35stable_flattened_mutations_consumerI22compact_for_compactionIN8sstables25compacting_sstable_writerE33noop_compacted_fragments_consumerEENS_9no_filterEEEDaT_T0_NSt6chrono10time_pointIN7seastar12lowres_clockENSC_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
#10 0x0000000002df514d _ZNK7seastar20noncopyable_functionIFvvEEclEv (scylla)
Stack trace of thread 25815:
#0 0x00007f4911ab25bb pthread_sigmask (libpthread.so.0)
#1 0x0000000002b54845 _ZN7seastar7reactor13signal_pollfn24try_enter_interrupt_modeEv (scylla)
#2 0x0000000002b579da _ZN7seastar7reactor5sleepEv (scylla)
#3 0x0000000002b975da _ZN7seastar7reactor3runEv (scylla)
#4 0x0000000002ba88bb _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
#5 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#6 0x00007f4911aaa432 start_thread (libpthread.so.0)
#7 0x00007f4910e26913 __clone (libc.so.6)
Stack trace of thread 25820:
#0 0x00007f4911ab25bb pthread_sigmask (libpthread.so.0)
#1 0x0000000002b54845 _ZN7seastar7reactor13signal_pollfn24try_enter_interrupt_modeEv (scylla)
#2 0x0000000002b579da _ZN7seastar7reactor5sleepEv (scylla)
#3 0x0000000002b975da _ZN7seastar7reactor3runEv (scylla)
#4 0x0000000002ba88bb _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
#5 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#6 0x00007f4911aaa432 start_thread (libpthread.so.0)
#7 0x00007f4910e26913 __clone (libc.so.6)
Stack trace of thread 25834:
#0 0x000000000117f2d0 _ZN7seastar13output_streamIcE5writeEPKcm (scylla)
#1 0x00000000011806e0 _ZN8sstables5writeINS_11file_writerEhNS_11disk_stringItEEJEEEvNS_21sstable_version_typesERT_RKT0_RKT1_DpOT2_ (scylla)
#2 0x000000000118ecd5 _ZNK8sstables24disk_set_of_tagged_unionINS_20scylla_metadata_typeEJNS_24disk_tagged_union_memberIS1_LS1_1ENS_17sharding_metadataEEENS2_IS1_LS1_2ENS_24sstable_enabled_featuresEEENS2_IS1_LS1_3ENS_9disk_hashIjNS_11disk_stringIjEES9_EEEENS2_IS1_LS1_4ENS_14run_identifierEEEEE6serdes16lookup_and_writeENS_21sstable_version_typesERNS_11file_writerES1_RKN5boost7variantIS4_JS6_SB_SD_EEE (scylla)
#3 0x00000000012192d6 _ZNK7seastar20noncopyable_functionIFvN8sstables21sstable_version_typesERNS1_11file_writerEEEclES2_S4_ (scylla)
#4 0x0000000001219f21 _ZN8sstables7sstable12write_simpleILNS_14component_typeE11ENS_15scylla_metadataEEEvRKT0_RKN7seastar17io_priority_classE (scylla)
#5 0x0000000001279677 _ZN8sstables2mc6writer21consume_end_of_streamEv (scylla)
#6 0x0000000001307f7d _ZN8sstables10compaction18finish_new_sstableEPNS_17compaction_writerE (scylla)
#7 0x000000000133e01c _ZN8sstables25compacting_sstable_writer21consume_end_of_streamEv (scylla)
#8 0x0000000002df514d _ZNK7seastar20noncopyable_functionIFvvEEclEv (scylla)
Stack trace of thread 25817:
#0 0x00007f4911ab25bb pthread_sigmask (libpthread.so.0)
#1 0x0000000002b54845 _ZN7seastar7reactor13signal_pollfn24try_enter_interrupt_modeEv (scylla)
#2 0x0000000002b579da _ZN7seastar7reactor5sleepEv (scylla)
#3 0x0000000002b975da _ZN7seastar7reactor3runEv (scylla)
#4 0x0000000002ba88bb _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEEN
PID: 29313 (scylla)
UID: 997 (scylla)
GID: 1001 (scylla)
Signal: 11 (SEGV)
Timestamp: Sat 2020-12-26 09:28:29 UTC (24h ago)
Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-15,17-31 --lock-memory=1
Executable: /opt/scylladb/libexec/scylla
Control Group: /
Boot ID: e820ea02e266416ba6569522084ed3c7
Machine ID: cc2c86fe566741e6a2ff6d399c5d5daa
Hostname: longevity-5000-tables-4-3-db-node-b3a128e5-2
Message: Process 29313 (scylla) of user 997 dumped core.
Stack trace of thread 29341:
#0 0x00000000012a47af _ZNSt15_Deque_iteratorIN8sstables11compression17segmented_offsets6bucketERS3_PS3_EpLEl (scylla)
#1 0x00000000012a5d92 _ZN8sstables11compression17segmented_offsets6writer9push_backEm (scylla)
#2 0x0000000000e2304a _ZN7seastar9data_sink3putENS_16temporary_bufferIcEE (scylla)
#3 0x0000000000e236bc _ZN7seastar13output_streamIcE5flushEv (scylla)
#4 0x0000000000e23b5a _ZN7seastar13output_streamIcE5closeEv (scylla)
#5 0x0000000001178edd _ZN8sstables11file_writer5closeEv (scylla)
#6 0x0000000001272e26 operator()<std::unique_ptr<sstables::file_writer> > (scylla)
#7 0x0000000001273739 _ZN8sstables2mc6writerD0Ev (scylla)
#8 0x000000000131e3c2 _ZN8sstables17compaction_writerD4Ev (scylla)
#9 0x000000000133ffdd _ZN20flat_mutation_reader4impl17consume_in_threadI35stable_flattened_mutations_consumerI22compact_for_compactionIN8sstables25compacting_sstable_writerE33noop_compacted_fragments_consumerEENS_9no_filterEEEDaT_T0_NSt6chrono10time_pointIN7seastar12lowres_clockENSC_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
#10 0x0000000002df514d _ZNK7seastar20noncopyable_functionIFvvEEclEv (scylla)
Stack trace of thread 29407:
#0 0x00007f2f77a5f9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f2f77a55432 start_thread (libpthread.so.0)
#5 0x00007f2f76dd1913 __clone (libc.so.6)
Stack trace of thread 29436:
#0 0x00007f2f77a5f9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f2f77a55432 start_thread (libpthread.so.0)
#5 0x00007f2f76dd1913 __clone (libc.so.6)
Stack trace of thread 29438:
#0 0x00007f2f77a5f9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f2f77a55432 start_thread (libpthread.so.0)
#5 0x00007f2f76dd1913 __clone (libc.so.6)
Stack trace of thread 29428:
#0 0x00007f2f77a5f9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f2f77a55432 start_thread (libpthread.so.0)
#5 0x00007f2f76dd1913 __clone (libc.so.6)
Stack trace of thread 29416:
#0 0x00007f2f77a5f9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f2f77a55432 start_thread (libpthread.so.0)
#5 0x00007f2f76dd1913 __clone (libc.so.6)
Stack trace of thread 29425:
#0 0x00007f2f77a5f9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f2f77a55432 start_thread (libpthread.so.0)
#5 0x00007f2f76dd1913 __clone (libc.so.6)
Stack trace of thread 29320:
#0 0x0000000002b4c9f2 process_queue<2, seastar::smp_message_queue::process_incoming()::<lambda(seastar::smp_message_queue::work_item*)> > (scylla)
#1 0x0000000002b9ab6e _ZN7seastar17smp_message_queue16process_incomingEv (scylla)
#2 0x0000000002b9ac16 _ZN7seastar7reactor10smp_pollfn4pollEv (scylla)
#3 0x0000000002b458bd _ZN7seastar7reactor9poll_onceEv (scylla)
#4 0x0000000002b97139 _ZNKSt8functionIFbvEEclEv (scylla)
#5 0x0000000002ba88bb _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
#6 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#7 0x00007f2f77a55432 star
PID: 33645 (scylla)
UID: 997 (scylla)
GID: 1001 (scylla)
Signal: 11 (SEGV)
Timestamp: Sat 2020-12-26 09:45:26 UTC (24h ago)
Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-15,17-31 --lock-memory=1
Executable: /opt/scylladb/libexec/scylla
Control Group: /
Boot ID: e820ea02e266416ba6569522084ed3c7
Machine ID: cc2c86fe566741e6a2ff6d399c5d5daa
Hostname: longevity-5000-tables-4-3-db-node-b3a128e5-2
Message: Process 33645 (scylla) of user 997 dumped core.
Stack trace of thread 33651:
#0 0x00000000012a47af _ZNSt15_Deque_iteratorIN8sstables11compression17segmented_offsets6bucketERS3_PS3_EpLEl (scylla)
#1 0x00000000012a5d92 _ZN8sstables11compression17segmented_offsets6writer9push_backEm (scylla)
#2 0x0000000000e2304a _ZN7seastar9data_sink3putENS_16temporary_bufferIcEE (scylla)
#3 0x0000000000e236bc _ZN7seastar13output_streamIcE5flushEv (scylla)
#4 0x0000000000e23b5a _ZN7seastar13output_streamIcE5closeEv (scylla)
#5 0x0000000001178edd _ZN8sstables11file_writer5closeEv (scylla)
#6 0x0000000001272e26 operator()<std::unique_ptr<sstables::file_writer> > (scylla)
#7 0x0000000001273739 _ZN8sstables2mc6writerD0Ev (scylla)
#8 0x000000000131e3c2 _ZN8sstables17compaction_writerD4Ev (scylla)
#9 0x000000000133ffdd _ZN20flat_mutation_reader4impl17consume_in_threadI35stable_flattened_mutations_consumerI22compact_for_compactionIN8sstables25compacting_sstable_writerE33noop_compac:ted_fragments_consumerEENS_9no_filterEEEDaT_T0_NSt6chrono10time_pointIN7seastar12lowres_clockENSC_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
#10 0x0000000002df514d _ZNK7seastar20noncopyable_functionIFvvEEclEv (scylla)
Stack trace of thread 33680:
#0 0x00007fed182af9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007fed182a5432 start_thread (libpthread.so.0)
#5 0x00007fed17621913 __clone (libc.so.6)
Stack trace of thread 33648:
#0 0x0000000002dcd940 try_reap_events (scylla)
#1 0x0000000002dc9b54 _ZN7seastar19reactor_backend_aio12await_eventsEiPK10__sigset_t (scylla)
#2 0x0000000002dc9c12 _ZN7seastar19reactor_backend_aio23reap_kernel_completionsEv (scylla)
#3 0x0000000002b458bd _ZN7seastar7reactor9poll_onceEv (scylla)
#4 0x0000000002b97139 _ZNKSt8functionIFbvEEclEv (scylla)
#5 0x0000000002ba88bb _ZZN7seastar3smp9configureEN5boost15program_options13variables_mapENS_14reactor_configEENKUlvE1_clEv (scylla)
#6 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#7 0x00007fed182a5432 start_thread (libpthread.so.0)
#8 0x00007fed17621913 __clone (libc.so.6)
Stack trace of thread 33677:
#0 0x00007fed182af9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007fed182a5432 start_thread (libpthread.so.0)
#5 0x00007fed17621913 __clone (libc.so.6)
Stack trace of thread 33690:
#0 0x00007fed182af9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007fed182a5432 start_thread (libpthread.so.0)
#5 0x00007fed17621913 __clone (libc.so.6)
Stack trace of thread 33694:
#0 0x00007fed182af9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007fed182a5432 start_thread (libpthread.so.0)
#5 0x00007fed17621913 __clone (libc.so.6)
Stack trace of thread 33695:
#0 0x00007fed182af9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007fed182a5432 start_thread (libpthread.so.0)
#5 0x00007fed17621913 __clone (libc.so.6)
Stack trace of thread 33693:
#0 0x00007fed182af9ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007fed182a5432 start_thread (libpthread.so.0)
#5 0x00007fed17621913 __clone (libc.so.
PID: 37690 (scylla)
UID: 997 (scylla)
GID: 1001 (scylla)
Signal: 11 (SEGV)
Timestamp: Sat 2020-12-26 10:02:17 UTC (23h ago)
Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 1-15,17-31 --lock-memory=1
Executable: /opt/scylladb/libexec/scylla
Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
Unit: scylla-server.service
Slice: scylla-server.slice
Boot ID: e820ea02e266416ba6569522084ed3c7
Machine ID: cc2c86fe566741e6a2ff6d399c5d5daa
Hostname: longevity-5000-tables-4-3-db-node-b3a128e5-2
Coredump: /var/lib/systemd/coredump/core.scylla.997.e820ea02e266416ba6569522084ed3c7.37690.1608976937000000
Message: Process 37690 (scylla) of user 997 dumped core.
Stack trace of thread 37690:
#0 0x00000000012a47af _ZNSt15_Deque_iteratorIN8sstables11compression17segmented_offsets6bucketERS3_PS3_EpLEl (scylla)
#1 0x00000000012a5d92 _ZN8sstables11compression17segmented_offsets6writer9push_backEm (scylla)
#2 0x0000000000e2304a _ZN7seastar9data_sink3putENS_16temporary_bufferIcEE (scylla)
#3 0x0000000000e236bc _ZN7seastar13output_streamIcE5flushEv (scylla)
#4 0x0000000000e23b5a _ZN7seastar13output_streamIcE5closeEv (scylla)
#5 0x0000000001178edd _ZN8sstables11file_writer5closeEv (scylla)
#6 0x0000000001272e26 operator()<std::unique_ptr<sstables::file_writer> > (scylla)
#7 0x0000000001273739 _ZN8sstables2mc6writerD0Ev (scylla)
#8 0x000000000131e3c2 _ZN8sstables17compaction_writerD4Ev (scylla)
#9 0x000000000133ffdd _ZN20flat_mutation_reader4impl17consume_in_threadI35stable_flattened_mutations_consumerI22compact_for_compactionIN8sstables25compacting_sstable_writerE33noop_compacted_fragments_consumerEENS_9no_filterEEEDaT_T0_NSt6chrono10time_pointIN7seastar12lowres_clockENSC_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
#10 0x0000000002df514d _ZNK7seastar20noncopyable_functionIFvvEEclEv (scylla)
Stack trace of thread 37725:
#0 0x00007f23b56229ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f23b5618432 start_thread (libpthread.so.0)
#5 0x00007f23b4994913 __clone (libc.so.6)
Stack trace of thread 37737:
#0 0x00007f23b56229ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f23b5618432 start_thread (libpthread.so.0)
#5 0x00007f23b4994913 __clone (libc.so.6)
Stack trace of thread 37721:
#0 0x00007f23b56229ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f23b5618432 start_thread (libpthread.so.0)
#5 0x00007f23b4994913 __clone (libc.so.6)
Stack trace of thread 37747:
#0 0x00007f23b56229ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f23b5618432 start_thread (libpthread.so.0)
#5 0x00007f23b4994913 __clone (libc.so.6)
Stack trace of thread 37724:
#0 0x00007f23b56229ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f23b5618432 start_thread (libpthread.so.0)
#5 0x00007f23b4994913 __clone (libc.so.6)
Stack trace of thread 37731:
#0 0x00007f23b56229ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f23b5618432 start_thread (libpthread.so.0)
#5 0x00007f23b4994913 __clone (libc.so.6)
Stack trace of thread 37729:
#0 0x00007f23b56229ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f23b5618432 start_thread (libpthread.so.0)
#5 0x00007f23b4994913 __clone (libc.so.6)
Stack trace of thread 37726:
#0 0x00007f23b56229ac read (libpthread.so.0)
#1 0x0000000002dc7be7 _ZN7seastar11thread_pool4workENS_13basic_sstringIcjLj15ELb1EEE (scylla)
#2 0x0000000002dc7e58 operator() (scylla)
#3 0x0000000002b2148e _ZNKSt8functionIFvvEEclEv (scylla)
#4 0x00007f23b5618432 start_th
coredump files can be found here:
download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.997.e820ea02e266416ba6569522084ed3c7.25805.1608973898000000/core.scylla.997.e820ea02e266416ba6569522084ed3c7.25805.1608973898000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.997.e820ea02e266416ba6569522084ed3c7.25805.1608973898000000.gz
download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.997.e820ea02e266416ba6569522084ed3c7.29313.1608974909000000/core.scylla.997.e820ea02e266416ba6569522084ed3c7.29313.1608974909000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.997.e820ea02e266416ba6569522084ed3c7.29313.1608974909000000.gz
md5-15e179d9efc61d4022df2e8e0ece2917
download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.997.e820ea02e266416ba6569522084ed3c7.33645.1608975926000000/core.scylla.997.e820ea02e266416ba6569522084ed3c7.33645.1608975926000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.997.e820ea02e266416ba6569522084ed3c7.33645.1608975926000000.gz
md5-15e179d9efc61d4022df2e8e0ece2917
download_instructions=gsutil cp gs://upload.scylladb.com/core.scylla.997.e820ea02e266416ba6569522084ed3c7.37690.1608976937000000/core.scylla.997.e820ea02e266416ba6569522084ed3c7.37690.1608976937000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.997.e820ea02e266416ba6569522084ed3c7.37690.1608976937000000.gz
md5-8f1264ffd8c82adc32ca8dcc2817fdd0
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Log links for testrun with test id b3a128e5-6563-459d-aeb4-eae163a7bedf |
+-----------------+-------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Date | Log type | Link |
+-----------------+-------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 20201227_093353 | grafana | https://cloudius-jenkins-test.s3.amazonaws.com/b3a128e5-6563-459d-aeb4-eae163a7bedf/20201227_093353/grafana-screenshot-overview-20201227_093353-longevity-5000-tables-4-3-monitor-node-b3a128e5-1.png |
| 20201227_093353 | grafana | https://cloudius-jenkins-test.s3.amazonaws.com/b3a128e5-6563-459d-aeb4-eae163a7bedf/20201227_093353/grafana-screenshot-scylla-per-server-metrics-nemesis-20201227_093807-longevity-5000-tables-4-3-monitor-node-b3a128e5-1.png |
| 20201227_094728 | db-cluster | https://cloudius-jenkins-test.s3.amazonaws.com/b3a128e5-6563-459d-aeb4-eae163a7bedf/20201227_094728/db-cluster-b3a128e5.zip |
| 20201227_094728 | loader-set | https://cloudius-jenkins-test.s3.amazonaws.com/b3a128e5-6563-459d-aeb4-eae163a7bedf/20201227_094728/loader-set-b3a128e5.zip |
| 20201227_094728 | monitor-set | https://cloudius-jenkins-test.s3.amazonaws.com/b3a128e5-6563-459d-aeb4-eae163a7bedf/20201227_094728/monitor-set-b3a128e5.zip |
| 20201227_094728 | sct-runner | https://cloudius-jenkins-test.s3.amazonaws.com/b3a128e5-6563-459d-aeb4-eae163a7bedf/20201227_094728/sct-runner-b3a128e5.zip |
+-----------------+-------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
fgelcer
on 27 Dec 2020
Probably fixed by 8a745a0ee0cd53b047079ef4ad9f63a50b4712ff
/cc @bhalevy
avikivity
on 27 Dec 2020
👍1
There's also:
2020-10-14T00:42:54+00:00 longevity-5000-tables-4-2-db-node-8db60178-2 !INFO | scylla: scylla: /jenkins/workspace/scylla-4.2/next/scylla/seastar/src/core/file.cc:503: virtual seastar::append_challenged_posix_file_impl::~append_challenged_posix_file_impl(): Assertion `_q.empty() && _logical_size == _committed_size' failed.
It looks like it's caused by truncating the file to the sloppy_size_hint in append_challenged_posix_file_impl constructor
(scylladb/seastar@9359188020a5d0ec3ed53a978e2a7ebc370471a0)
and to f5f58b46c7346a36db309aa3b65079f29343684a setting opt.sloppy_size = true; in sstable::create_data()
So scylladb/seastar@35c255dcd39c235712681c404d2a1aa783423127 wasn't enough.
bhalevy
on 28 Dec 2020
Sent [seastar-dev] [PATCH 1/1] append_challenged_posix_file_impl: adjust sloppy_size only in optimize_queue to the mailing list
bhalevy
on 28 Dec 2020
@bhalevy does this require a backport? Or was it a regression introduced and fixed on master only?
avikivity
on 8 Feb 2021
@bhalevy does this require a backport? Or was it a regression introduced and fixed on master only?
@avikivity the issue has been probably lurking since 4.0 (that's the first release that has https://github.com/scylladb/seastar/commit/35c255dcd39c235712681c404d2a1aa783423127)
Therefor we may want to consider it for 4.4 but I don't think it's worth backporting to earlier releases.
bhalevy
on 8 Feb 2021
Was this page helpful?
0 / 5 - 0 ratings
Related issues
pdziepak
·
6Comments
fgelcer
·
6Comments
LouayKamel
·
6Comments
dimaqq
·
6Comments
mattcobley
·
6Comments