Scylla: Segmentation fault and coredump on the few nodes during reading from materialized view

Created on 8 Jun 2020 · 61Comments · Source: scylladb/scylla

Installation details
Scylla version (or git commit hash): 666.development-0.20200607.a1e
235b1a with build-id f71dfc93f549d44c958d52d11ffc78757e085386
Cluster size: 4 nodes (i3.2xlarge)
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-0b53d2c2e5f6eb7d6 (eu-west-1)

Test: longevity-lwt-3h

Test schema:

one table

CREATE TABLE IF NOT EXISTS blogposts (
        domain int,
        published_date int,
        lwt_indicator int,
        url text,
        author text,
        title text,
        PRIMARY KEY(domain, published_date)
  ) WITH compaction = { 'class':'LeveledCompactionStrategy' }
    AND comment='A table to hold blog posts'

5 materialized views:

  - create MATERIALIZED VIEW blogposts_update_one_column_lwt_indicator as select domain, lwt_indicator, author from blogposts where domain is not null and published_date is not null and lwt_indicator < 0 PRIMARY KEY(lwt_indicator, domain, published_date);
  - create MATERIALIZED VIEW blogposts_update_one_column_lwt_indicator_after_update as select lwt_indicator, author from blogposts where domain is not null and published_date is not null and lwt_indicator = 30000000 PRIMARY KEY(lwt_indicator, domain, published_date);
  - create MATERIALIZED VIEW blogposts_update_2_columns_lwt_indicator as select lwt_indicator, author from blogposts where domain is not null and published_date is not null and lwt_indicator > 0 and lwt_indicator <= 1000000 PRIMARY KEY(lwt_indicator, domain, published_date);
  - create MATERIALIZED VIEW blogposts_update_2_columns_lwt_indicator_after_update as select lwt_indicator, author from blogposts where domain is not null and published_date is not null and lwt_indicator = 20000000 PRIMARY KEY(lwt_indicator, domain, published_date);
  - create MATERIALIZED VIEW blogposts_not_updated_lwt_indicator as select lwt_indicator, author from blogposts where domain is not null and published_date is not null and lwt_indicator > 1000000 and lwt_indicator < 20000000 PRIMARY KEY(lwt_indicator, domain, published_date);

First was run c-s command to prepare data:

cassandra-stress user profile=/tmp/c-s_lwt_basic.yaml n=10000000 ops'(insert=1)' cl=QUORUM -mode native cql3 -rate threads=30

Next new table was created and start fetch all rows from materialized views:

Jun 08 10:53:30 longevity-lwt-3h-master-db-node-9199cde4-4 scylla[3898]:  [shard 0] schema_tables - Creating cqlstress_lwt_example.blogposts_update_one_column_lwt_indicator_expect id=4a448120-a976-11ea-9a87-000000000002 version=2b61c1ea-316e-30c1-9343-7c3735f16cf8
092 bytes to 50637 (~90% of original) in 30ms = 1.61MB/s. ~256 total partitions merged to 2.
Jun 08 10:53:30 longevity-lwt-3h-master-db-node-9199cde4-4 scylla[3898]:  [shard 0] database - Schema version changed to 1f5189ef-ffb2-3dfc-8713-125fafd27f15

< t:2020-06-08 10:53:31,158 f:tester.py       l:1499 c:sdcm.tester          p:DEBUG > Fetch all rows by statement: 
SELECT domain,published_date FROM cqlstress_lwt_example.blogposts_update_one_column_lwt_indicator

During reading data node 4 got Segmentation fault and coredump:

Jun 08 10:53:30 longevity-lwt-3h-master-db-node-9199cde4-4 scylla[3898]:  [shard 7] compaction - Compacted 2 sstables to [/var/lib/scylla/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/mc-143-big-Data.db:level=0, ]. 
10833 bytes to 5631 (~51% of original) in 32ms = 0.17MB/s. ~256 total partitions merged to 1.
Jun 08 10:53:30 longevity-lwt-3h-master-db-node-9199cde4-4 scylla[3898]:  [shard 7] compaction - Compacted 2 sstables to [/var/lib/scylla/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/mc-159-big-Data.db:level=0, ]. 54894 
bytes to 49548 (~90% of original) in 34ms = 1.39MB/s. ~256 total partitions merged to 2.
Jun 08 10:53:30 longevity-lwt-3h-master-db-node-9199cde4-4 scylla[3898]:  [shard 7] compaction - Compacted 2 sstables to [/var/lib/scylla/data/system_schema/computed_columns-cc7c7069374033c192a4c3de78dbd2c4/mc-143-big-Data.db:level=0, ].
 10550 bytes to 5399 (~51% of original) in 28ms = 0.18MB/s. ~256 total partitions merged to 1.
Jun 08 10:53:30 longevity-lwt-3h-master-db-node-9199cde4-4 scylla[3898]:  [shard 7] compaction - Compacted 2 sstables to [/var/lib/scylla/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/mc-143-big-Data.db:level=0, ]. 60499 byt
es to 53585 (~88% of original) in 27ms = 1.89MB/s. ~256 total partitions merged to 2.

2020-06-08T10:53:53+00:00  longevity-lwt-3h-master-db-node-9199cde4-4 !INFO    | scylla: Segmentation fault on shard 1.
scylla: Backtrace:
scylla: 0x0000000002f27312
scylla: 0x0000000002eca930
scylla: 0x0000000002ecabd5
scylla: 0x0000000002ecac70
scylla: 0x00007f18500a6a8f
scylla: 0x0000000000de19f1
scylla: 0x0000000000fc543f
scylla: 0x0000000000fc8dba
scylla: 0x0000000000fcdc4b
scylla: 0x0000000000fcf971
scylla: 0x0000000000fd009b
scylla: 0x0000000000fcfa7c
scylla: 0x000000000116fb77
scylla: 0x0000000000fbf208
scylla: 0x0000000000fbf7cd
scylla: 0x000000000112edbc
scylla: 0x000000000112fc91
scylla: 0x000000000113a8dc
scylla: 0x000000000113cbfd
scylla: 0x000000000113d110
scylla: 0x000000000110a5dc
scylla: 0x0000000001131490
scylla: 0x0000000001132628
scylla: 0x00000000011356f6
scylla: 0x0000000001137d54
scylla: 0x0000000001137f4d
scylla: 0x000000000113840b
scylla: 0x0000000001138534
scylla: 0x0000000001096430
scylla: 0x000000000109f2bd
scylla: 0x0000000002ec7ee7
scylla: 0x0000000002ec826e
scylla: 0x0000000002f0280d
scylla: 0x0000000002f0f87a
scylla: 0x0000000002e9473d
scylla: /opt/scylladb/libreloc/libpthread.so.0+0x0000000000009431
scylla: /opt/scylladb/libreloc/libc.so.6+0x00000000001019d2
kernel: traps: reactor-1[3899] general protection fault ip:de19f2 sp:7f184cfe5500 error:0 in scylla[c79000+2721000]

Decoded:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/fmt/format.h:2188
seastar::backtrace_buffer::append_backtrace() at /usr/include/fmt/format.h:2188
 (inlined by) print_with_backtrace at ./build/release/seastar/./seastar/src/core/reactor.cc:763
seastar::print_with_backtrace(char const*) at /usr/include/fmt/format.h:2188
sigsegv_action at /usr/include/fmt/format.h:2188
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3460
 (inlined by) _FUN at ./build/release/seastar/./seastar/src/core/reactor.cc:3456
?? ??:0
managed_bytes::managed_bytes(managed_bytes const&) at /usr/include/fmt/core.h:606
compound_wrapper<clustering_key_prefix, clustering_key_prefix_view>::compound_wrapper(compound_wrapper<clustering_key_prefix, clustering_key_prefix_view> const&) at ./keys.hh:165
 (inlined by) prefix_compound_wrapper<clustering_key_prefix, clustering_key_prefix_view, clustering_key_prefix>::prefix_compound_wrapper(prefix_compound_wrapper<clustering_key_prefix, clustering_key_prefix_view, clustering_key_prefix> const&) at ./keys.hh:558
 (inlined by) clustering_key_prefix::clustering_key_prefix(clustering_key_prefix const&) at ./keys.hh:788
 (inlined by) std::_Optional_payload_base<clustering_key_prefix>::_Storage<clustering_key_prefix, false>::_Storage<clustering_key_prefix const&>(std::in_place_t, clustering_key_prefix const&) at /usr/include/c++/10/optional:226
 (inlined by) std::_Optional_payload_base<clustering_key_prefix>::_Optional_payload_base<clustering_key_prefix const&>(std::in_place_t, clustering_key_prefix const&) at /usr/include/c++/10/optional:118
 (inlined by) _ZNSt17_Optional_payloadI21clustering_key_prefixLb1ELb0ELb0EECI4St22_Optional_payload_baseIS0_EIJRKS0_EEESt10in_place_tDpOT_ at /usr/include/c++/10/optional:359
 (inlined by) _ZNSt17_Optional_payloadI21clustering_key_prefixLb0ELb0ELb0EECI4St22_Optional_payload_baseIS0_EIJRKS0_EEESt10in_place_tDpOT_ at /usr/include/c++/10/optional:393
 (inlined by) std::_Optional_base<clustering_key_prefix, false, false>::_Optional_base<clustering_key_prefix const&, false>(std::in_place_t, clustering_key_prefix const&) at /usr/include/c++/10/optional:485
 (inlined by) std::optional<clustering_key_prefix>::optional<clustering_key_prefix const&, true>(clustering_key_prefix const&) at /usr/include/c++/10/optional:702
 (inlined by) position_in_partition::position_in_partition(position_in_partition::range_tag_t, bound_view) at ././position_in_partition.hh:249
 (inlined by) position_in_partition::for_range_start(nonwrapping_range<clustering_key_prefix> const&) at ././position_in_partition.hh:516
 (inlined by) cache::cache_flat_mutation_reader::move_to_range(__gnu_cxx::__normal_iterator<nonwrapping_range<clustering_key_prefix> const*, std::vector<nonwrapping_range<clustering_key_prefix>, std::allocator<nonwrapping_range<clustering_key_prefix> > > >) at ./cache_flat_mutation_reader.hh:552
cache::cache_flat_mutation_reader::move_to_next_range() at ./cache_flat_mutation_reader.hh:546
 (inlined by) cache::cache_flat_mutation_reader::copy_from_cache_to_buffer() at ./cache_flat_mutation_reader.hh:529
 (inlined by) cache::cache_flat_mutation_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}::operator()() const at ./cache_flat_mutation_reader.hh:297
_ZZZN5cache11lsa_manager19run_in_read_sectionIZNS_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_ENKUlvE_clEvENKUlvE_clEv at ./row_cache.hh:590
 (inlined by) _Z29with_linearized_managed_bytesIZZN5cache11lsa_manager19run_in_read_sectionIZNS0_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS4_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_ENKUlvE_clEvEUlvE_ENSt9result_ofIFSE_vEE4typeEOSE_ at ./utils/managed_bytes.hh:428
 (inlined by) _ZZN5cache11lsa_manager19run_in_read_sectionIZNS_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_ENKUlvE_clEv at ./row_cache.hh:591
 (inlined by) _ZN8logalloc18allocating_section24with_reclaiming_disabledIRZN5cache11lsa_manager19run_in_read_sectionIZNS2_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS6_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_EUlvE_EEDcRNS_6regionEOSG_ at ././utils/logalloc.hh:746
 (inlined by) _ZZN8logalloc18allocating_sectionclIZN5cache11lsa_manager19run_in_read_sectionIZNS2_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS6_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_EUlvE_EEDcRNS_6regionEOSG_ENKUlvE_clEv at ././utils/logalloc.hh:768
 (inlined by) _ZN8logalloc18allocating_section12with_reserveIZNS0_clIZN5cache11lsa_manager19run_in_read_sectionIZNS3_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS7_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_EUlvE_EEDcRNS_6regionEOSH_EUlvE_EEDcSN_ at ././utils/logalloc.hh:718
 (inlined by) _ZN8logalloc18allocating_sectionclIZN5cache11lsa_manager19run_in_read_sectionIZNS2_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS6_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_EUlvE_EEDcRNS_6regionEOSG_ at ././utils/logalloc.hh:769
 (inlined by) _ZN5cache11lsa_manager19run_in_read_sectionIZNS_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_ at ./row_cache.hh:592
 (inlined by) cache::cache_flat_mutation_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./cache_flat_mutation_reader.hh:306
cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}::operator()() const at ./cache_flat_mutation_reader.hh:237
 (inlined by) seastar::future<> seastar::futurize<seastar::future<> >::invoke<cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&>(cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&) at ././seastar/include/seastar/core/future.hh:1861
 (inlined by) auto seastar::futurize_invoke<cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&>(cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&) at ././seastar/include/seastar/core/future.hh:1945
 (inlined by) seastar::future<> seastar::do_until<cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}, cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}>(cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}, cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}) at ././seastar/include/seastar/core/future-util.hh:525
 (inlined by) cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./cache_flat_mutation_reader.hh:238
cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const at ./cache_flat_mutation_reader.hh:227
cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./cache_flat_mutation_reader.hh:232
flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /usr/include/fmt/format.h:2188
 (inlined by) seastar::future<bool> flat_mutation_reader::impl::fill_buffer_from<flat_mutation_reader>(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./flat_mutation_reader.cc:196
scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() const at ./row_cache.cc:697
seastar::future<> seastar::futurize<seastar::future<> >::invoke<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&) at ././seastar/include/seastar/core/future.hh:1861
 (inlined by) auto seastar::futurize_invoke<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&) at ././seastar/include/seastar/core/future.hh:1945
 (inlined by) seastar::future<> seastar::do_until<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}, scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}) at ././seastar/include/seastar/core/future-util.hh:525
 (inlined by) scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./row_cache.cc:703
mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at mutation_reader.cc:?
 (inlined by) flat_mutation_reader::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./flat_mutation_reader.hh:319
 (inlined by) mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at ./mutation_reader.cc:361
mutation_reader_merger::advance_galloping_reader(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&>(combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&) at mutation_reader.cc:?
 (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./mutation_reader.cc:108
 (inlined by) operator() at ./mutation_reader.cc:578
 (inlined by) invoke<combined_mutation_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at ././seastar/include/seastar/core/future.hh:1861
combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./mutation_reader.cc:589
(anonymous namespace)::shard_reader::remote_reader::do_fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) do_fill_buffer at ./mutation_reader.cc:1258
auto seastar::internal::do_with_impl<flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(nonwrapping_range<dht::ring_position> const&, bool, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)#1}>(flat_mutation_reader&&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&&, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(nonwrapping_range<dht::ring_position> const&, bool, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)#1}&&) at mutation_reader.cc:?
 (inlined by) operator() at ./mutation_reader.cc:1355
 (inlined by) __invoke_impl<seastar::future<(anonymous namespace)::shard_reader::fill_buffer_result>, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&> at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) __invoke<(anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&> at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) __apply_impl<(anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, std::tuple<flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> > >&, 0, 1> at /usr/include/c++/10/tuple:1724
 (inlined by) apply<(anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, std::tuple<flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> > >&> at /usr/include/c++/10/tuple:1736
 (inlined by) do_with_impl<flat_mutation_reader, seastar::circular_buffer<mutation_fragment>, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)> > at ././seastar/include/seastar/core/do_with.hh:106
(anonymous namespace)::shard_reader::remote_reader::fill_buffer(nonwrapping_range<dht::ring_position> const&, bool, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) futurize_invoke<seastar::future<(anonymous namespace)::shard_reader::fill_buffer_result> (*&)(flat_mutation_reader&&, seastar::circular_buffer<mutation_fragment>&&, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&&), flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)> > at ././seastar/include/seastar/core/future.hh:1945
 (inlined by) do_with<flat_mutation_reader, seastar::circular_buffer<mutation_fragment>, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)> > at ././seastar/include/seastar/core/do_with.hh:141
 (inlined by) fill_buffer at ./mutation_reader.cc:1360
(anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) invoke<(anonymous namespace)::shard_reader::do_fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at ././seastar/include/seastar/core/future.hh:1861
 (inlined by) submit_to<(anonymous namespace)::shard_reader::do_fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at ././seastar/include/seastar/core/smp.hh:333
 (inlined by) submit_to<(anonymous namespace)::shard_reader::do_fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at ././seastar/include/seastar/core/smp.hh:360
 (inlined by) do_fill_buffer at ./mutation_reader.cc:1404
(anonymous namespace)::shard_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
multishard_combining_reader::handle_empty_reader_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() const at mutation_reader.cc:?
multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) futurize_invoke<multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at ././seastar/include/seastar/core/future.hh:1945
 (inlined by) do_until<multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>, multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at ././seastar/include/seastar/core/future-util.hh:525
 (inlined by) multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./mutation_reader.cc:1629
seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() at /usr/include/fmt/format.h:2188
seastar::internal::do_until_state<seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}>::run_and_dispose() at /usr/include/fmt/format.h:2188
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2724
seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::{lambda()#3}::operator()() const at /usr/include/fmt/format.h:2188
std::function<void ()>::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

Coredump:

PID: 3898 (scylla)
           UID: 996 (scylla)
           GID: 1001 (scylla)
        Signal: 11 (SEGV)
     Timestamp: Mon 2020-06-08 10:53:53 UTC (1min 35s ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 
--default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 0-7 --lock-memory=1
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /
       Boot ID: d36fdba55cb44fc98409492539496158
    Machine ID: df877a200226bc47d06f26dae0736ec9
      Hostname: longevity-lwt-3h-master-db-node-9199cde4-4
      Coredump: /var/lib/systemd/coredump/core.scylla.996.d36fdba55cb44fc98409492539496158.3898.1591613633000000
       Message: Process 3898 (scylla) of user 996 dumped core.

                Stack trace of thread 3899:
                #0  0x0000000000de19f2 _ZN13managed_bytesC2ERKS_ (scylla)
                #1  0x0000000000fc5440 _ZN16compound_wrapperI21clustering_key_prefix26clustering_key_prefix_viewEC4ERKS2_ (scylla)
                #2  0x0000000000fc8dbb _ZZN5cache26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEEENKUlvE1_clEv (scylla)
                #3  0x0000000000fcdc4c _ZZZN5cache11lsa_manager19run_in_read_sectionIZNS_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_E
NKUlvE_clEvENKUlvE_clEv (scylla)
                #4  0x0000000000fcf972 _ZZN5cache26cache_flat_mutation_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEEENKUlvE1_clEv (scylla)
                #5  0x0000000000fd009c _ZZN5cache26cache_flat_mutation_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEEENKUlvE_clEv (scylla)
                #6  0x0000000000fcfa7d _ZN5cache26cache_flat_mutation_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #7  0x000000000116fb78 _ZN20flat_mutation_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS0_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #8  0x0000000000fbf209 _ZZN30scanning_and_populating_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS0_8durationIlSt5ratioILl1ELl1000EEEEEEENKUlvE0_clEv (scylla)
                #9  0x0000000000fbf7ce _ZN7seastar8futurizeINS_6futureIJEEEE6invokeIRZN30scanning_and_populating_reader11fill_bufferENSt6chrono10time_pointINS_12lowres_clockENS6_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE0_JEEES2_OT_DpOT0_
 (scylla)
                #10 0x000000000112edbd _ZN20flat_mutation_reader4implclENSt6chrono10time_pointIN7seastar12lowres_clockENS1_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #11 0x000000000112fc92 _ZN22mutation_reader_merger24advance_galloping_readerENSt6chrono10time_pointIN7seastar12lowres_clockENS0_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #12 0x000000000113a8dd _ZN22mutation_reader_mergerclENSt6chrono10time_pointIN7seastar12lowres_clockENS0_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #13 0x000000000113cbfe _ZN24mutation_fragment_mergerI22mutation_reader_mergerE5fetchENSt6chrono10time_pointIN7seastar12lowres_clockENS2_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #14 0x000000000113d111 repeat<combined_mutation_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > (scylla)
                #15 0x000000000110a5dd _ZN20flat_mutation_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS0_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #16 0x0000000001131491 fill_buffer (scylla)
                #17 0x0000000001132629 invoke<seastar::future<(anonymous namespace)::shard_reader::fill_buffer_result> (*&)(flat_mutation_reader&&, seastar::circular_buffer<mutation_fragment>&&, (anonymous namespace)::shard_reader::remot
e_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&&), flat_mutation_reader, seastar::circular_buffer<mutation_fragment, s
td::allocator<mutation_fragment> >, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragme
nt, std::allocator<mutation_fragment> >&)> > (scylla)
                #18 0x00000000011356f7 operator() (scylla)
                #19 0x0000000001137d55 fill_buffer (scylla)
                #20 0x0000000001137f4e _ZN27multishard_combining_reader26handle_empty_reader_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS0_8durationIlSt5ratioILl1ELl1000EEEEEE (scylla)
                #21 0x000000000113840c _ZZN27multishard_combining_reader11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS0_8durationIlSt5ratioILl1ELl1000EEEEEEENKUlvE0_clEv (scylla)
                #22 0x0000000001138535 invoke<multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_

download instruction:

gsutil cp gs://upload.scylladb.com/core.scylla.996.d36fdba55cb44fc98409492539496158.3898.1591613633000000/core.scylla.996.d36fdba55cb44fc98409492539496158.3898.1591613633000000.gz .

gunzip /var/lib/systemd/coredump/core.scylla.996.d36fdba55cb44fc98409492539496158.3898.1591613633000000.gz

Node was restarted. Ther are alot of commitlog_files for replaying
Similar segmentation fault and coredump on the nodes 3 and 2

The monitor and nodes are alive meanwhile

Test logs:
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Log links for testrun with test id 9199cde4-5a74-4002-88fc-c4d8ff97ea80 |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Date | Log type | Link |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 20200608_114528 | db-cluster | https://cloudius-jenkins-test.s3.amazonaws.com/9199cde4-5a74-4002-88fc-c4d8ff97ea80/20200608_114528/db-cluster-9199cde4.zip |
| 20200608_114528 | sct-runner | https://cloudius-jenkins-test.s3.amazonaws.com/9199cde4-5a74-4002-88fc-c4d8ff97ea80/20200608_114528/sct-runner-9199cde4.zip |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

bug high

Source

juliayakovlev

All 61 comments

@psarna I think you had a look - since @denesb was out on holiday on monday - nut lets get the right person checking this.

@psarna can you please update the issue if you found anything

@bhalevy / @denesb I expect you need to look at this

slivne on 9 Jun 2020

I spent the day on trying to run the core + binary under gdb properly, but I ultimately discovered that the binary from this AMI is corrupt in a weird way - it doesn't have a proper dynamic linker path and it crashes gdb when trying to print a backtrace. Eventually, Rafael pointed me to this link, and that's what I was planning to start with tomorrow: https://github.com/scylladb/scylla-pkg/#looking-for-a-build . Rafael reported that the binary found via this link works properly with the core, and he was able to see the backtrace without issues. My next planned step was to go to the problematic frame and see at which point we started using freed memory, since that sounds like a likely cause, judging from the backtrace.

psarna on 9 Jun 2020

The package is here: http://downloads.scylladb.com.s3.amazonaws.com/relocatable/unstable/master/2020-06-07T17%3A36%3A22Z//scylla-package.tar.gz

Note that due to https://github.com/scylladb/scylla-pkg/issues/1365 there are two consecutive slashes (//) in the url.
and that confuses the scylla-pkg scripts/scylla-s3-reloc tool.

bhalevy on 10 Jun 2020

This issue seems to reoccur in the latest lwt-3h job, where two segmentation faults occurred in two nodes.
@bhalevy Scylla version: 666.development-0.20200610.65d3e3992 with build-id e42da3a7b88f607f972cc487f08ce35faaa60fe5
node#1:

2020-06-11 02:27:10.000: (CoreDumpEvent Severity.ERROR): node=Node longevity-lwt-3h-master-db-node-ac2871f9-1 [3.250.108.226 | 10.0.99.128] (seed: True)
corefile_url=
https://storage.cloud.google.com/upload.scylladb.com/core.scylla.996.43f871bf672840c891c9973e36edc9ea.3014.1591842430000000/core.scylla.996.43f871bf672840c891c9973e36edc9ea.3014.1591842430000000.gz
backtrace=           PID: 3014 (scylla)
           UID: 996 (scylla)
           GID: 1001 (scylla)
        Signal: 11 (SEGV)
     Timestamp: Thu 2020-06-11 02:27:10 UTC (1min 37s ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 0-7 --lock-memory=1
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
          Unit: scylla-server.service
         Slice: scylla-server.slice
       Boot ID: 43f871bf672840c891c9973e36edc9ea
    Machine ID: df877a200226bc47d06f26dae0736ec9
      Hostname: longevity-lwt-3h-master-db-node-ac2871f9-1
      Coredump: /var/lib/systemd/coredump/core.scylla.996.43f871bf672840c891c9973e36edc9ea.3014.1591842430000000
       Message: Process 3014 (scylla) of user 996 dumped core.
...
download_instructions=
gsutil cp gs://upload.scylladb.com/core.scylla.996.43f871bf672840c891c9973e36edc9ea.3014.1591842430000000/core.scylla.996.43f871bf672840c891c9973e36edc9ea.3014.1591842430000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.996.43f871bf672840c891c9973e36edc9ea.3014.1591842430000000.gz

Backtrace:

2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: Segmentation fault on shard 0.
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: Backtrace:
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000002f6b982
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000002f0ef90
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000002f0f235
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000002f0f2d0
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x00007fe8da44da8f
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000001f4ee60
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000000fa70dd
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000000fa83cc
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000000fbfbe7
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000000fcbc89
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000000fccb7b
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000000fccefd
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x000000000113c02c
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x000000000113d2ca
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x00000000011478f5
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000001149e6d
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x000000000114a380
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x000000000111784c
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x000000000113e700
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x000000000113f898
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x000000000114200d
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x000000000114351d
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000001144fc4
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x00000000011451bd
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x000000000114567b
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x00000000011457a4
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x00000000010a9a5f
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x00000000027003da
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000002707b28
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x000000000270d26c
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000002f0c547
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000002f0c8ce
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000002f46e6d
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000002ea1db5
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000002ea24ae
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000000d70ab0
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000027041
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: 0x0000000000c8662d
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | kernel: scylla[3014]: segfault at 17 ip 0000000001f4ee61 sp 00007ffdd3d0a9d0 error 4 in scylla[c86000+2758000]
2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | kernel: Code: 89 54 24 08 48 89 0c 24 e8 fc 9b f3 ff 48 8b 54 24 08 48 8b 0c 24 49 89 c0 44 0f be 72 0f 45 84 f6 79 07 48 8b 02 44 8b 70 08 <41> 0f b6 44 24 0f 44 89 f5 4c 89 e7 84 c0 0f 89 9b 00 00 00 49 8b

Translated:

2020-06-11T02:27:10+00:00  longevity-lwt-3h-master-db-node-ac2871f9-1 !INFO    | scylla: Segmentation fault on shard 0.

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/fmt/format.h:2188
seastar::backtrace_buffer::append_backtrace() at /usr/include/fmt/format.h:2188
 (inlined by) print_with_backtrace at ./build/release/seastar/./seastar/src/core/reactor.cc:763
seastar::print_with_backtrace(char const*) at /usr/include/fmt/format.h:2188
sigsegv_action at /usr/include/fmt/format.h:2188
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3460
 (inlined by) _FUN at ./build/release/seastar/./seastar/src/core/reactor.cc:3456
?? ??:0
managed_bytes::size() const at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) managed_bytes::operator std::basic_string_view<signed char, std::char_traits<signed char> >() const at ./utils/managed_bytes.hh:341
 (inlined by) compound_wrapper<partition_key, partition_key_view>::equal(schema const&, partition_key const&) const at ./keys.hh:304
 (inlined by) query::specific_ranges::range_for(schema const&, partition_key const&) const at ./query-request.hh:101
 (inlined by) query::partition_slice::row_ranges(schema const&, partition_key const&) const at ./query.cc:162
query::clustering_key_filter_ranges::get_ranges(schema const&, query::partition_slice const&, partition_key const&) at /usr/include/fmt/core.h:323
 (inlined by) cache_entry::do_read(row_cache&, cache::read_context&) at ./row_cache.cc:1267
cache_entry::read(row_cache&, cache::read_context&) at /usr/include/fmt/core.h:323
scanning_and_populating_reader::read_from_entry(cache_entry&) at /usr/include/fmt/core.h:323
 (inlined by) scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const at ./row_cache.cc:615
std::result_of<scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const::{lambda()#1} ()>::type with_linearized_managed_bytes<scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const::{lambda()#1}>(scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const::{lambda()#1}&&) at ./utils/managed_bytes.hh:428
 (inlined by) scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const at ./row_cache.cc:640
 (inlined by) _ZN8logalloc18allocating_section24with_reclaiming_disabledIRZN30scanning_and_populating_reader20do_read_from_primaryENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_EEDcRNS_6regionEOT_ at ././utils/logalloc.hh:746
 (inlined by) _ZZN8logalloc18allocating_sectionclIZN30scanning_and_populating_reader20do_read_from_primaryENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_EEDcRNS_6regionEOT_ENKUlvE_clEv at ././utils/logalloc.hh:768
 (inlined by) _ZN8logalloc18allocating_section12with_reserveIZNS0_clIZN30scanning_and_populating_reader20do_read_from_primaryENSt6chrono10time_pointIN7seastar12lowres_clockENS4_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_EEDcRNS_6regionEOT_EUlvE_EEDcSH_ at ././utils/logalloc.hh:718
 (inlined by) _ZN8logalloc18allocating_sectionclIZN30scanning_and_populating_reader20do_read_from_primaryENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_EEDcRNS_6regionEOT_ at ././utils/logalloc.hh:769
 (inlined by) scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./row_cache.cc:641
 (inlined by) scanning_and_populating_reader::read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./row_cache.cc:645
scanning_and_populating_reader::read_next_partition(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./row_cache.cc:669
 (inlined by) scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() const at ./row_cache.cc:695
seastar::future<> seastar::futurize<seastar::future<> >::invoke<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&) at ././seastar/include/seastar/core/future.hh:1865
 (inlined by) auto seastar::futurize_invoke<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&) at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) seastar::future<> seastar::do_until<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}, scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}) at ././seastar/include/seastar/core/future-util.hh:525
 (inlined by) scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./row_cache.cc:703
mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at mutation_reader.cc:?
 (inlined by) flat_mutation_reader::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./flat_mutation_reader.hh:319
 (inlined by) mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at ./mutation_reader.cc:361
mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at ././seastar/include/seastar/core/future.hh:1865
 (inlined by) futurize_invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) parallel_for_each<mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at ././seastar/include/seastar/core/future-util.hh:159
 (inlined by) parallel_for_each_impl<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at ././seastar/include/seastar/core/future-util.hh:204
 (inlined by) invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at ././seastar/include/seastar/core/future.hh:1865
 (inlined by) futurize_invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) parallel_for_each<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at ././seastar/include/seastar/core/future-util.hh:216
 (inlined by) mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./mutation_reader.cc:353
mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&>(combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&) at mutation_reader.cc:?
 (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./mutation_reader.cc:108
 (inlined by) operator() at ./mutation_reader.cc:578
 (inlined by) invoke<combined_mutation_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at ././seastar/include/seastar/core/future.hh:1865
combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./mutation_reader.cc:589
(anonymous namespace)::shard_reader::remote_reader::do_fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) do_fill_buffer at ./mutation_reader.cc:1258
auto seastar::internal::do_with_impl<flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(nonwrapping_range<dht::ring_position> const&, bool, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)#1}>(flat_mutation_reader&&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&&, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(nonwrapping_range<dht::ring_position> const&, bool, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)#1}&&) at mutation_reader.cc:?
 (inlined by) operator() at ./mutation_reader.cc:1355
 (inlined by) __invoke_impl<seastar::future<(anonymous namespace)::shard_reader::fill_buffer_result>, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&> at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) __invoke<(anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&> at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) __apply_impl<(anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, std::tuple<flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> > >&, 0, 1> at /usr/include/c++/10/tuple:1724
 (inlined by) apply<(anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, std::tuple<flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> > >&> at /usr/include/c++/10/tuple:1736
 (inlined by) do_with_impl<flat_mutation_reader, seastar::circular_buffer<mutation_fragment>, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)> > at ././seastar/include/seastar/core/do_with.hh:106
(anonymous namespace)::shard_reader::remote_reader::fill_buffer(nonwrapping_range<dht::ring_position> const&, bool, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) futurize_invoke<seastar::future<(anonymous namespace)::shard_reader::fill_buffer_result> (*&)(flat_mutation_reader&&, seastar::circular_buffer<mutation_fragment>&&, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&&), flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)> > at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) do_with<flat_mutation_reader, seastar::circular_buffer<mutation_fragment>, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)> > at ././seastar/include/seastar/core/do_with.hh:141
 (inlined by) fill_buffer at ./mutation_reader.cc:1360
(anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const at mutation_reader.cc:?
(anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) submit_to<(anonymous namespace)::shard_reader::do_fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at ././seastar/include/seastar/core/smp.hh:333
 (inlined by) submit_to<(anonymous namespace)::shard_reader::do_fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at ././seastar/include/seastar/core/smp.hh:360
 (inlined by) do_fill_buffer at ./mutation_reader.cc:1397
(anonymous namespace)::shard_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
multishard_combining_reader::handle_empty_reader_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() const at mutation_reader.cc:?
multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) futurize_invoke<multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) do_until<multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>, multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at ././seastar/include/seastar/core/future-util.hh:525
 (inlined by) multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./mutation_reader.cc:1629
flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /usr/include/fmt/format.h:2188
 (inlined by) flat_mutation_reader::peek(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./flat_mutation_reader.hh:457
 (inlined by) auto query::consume_page<(emit_only_live_rows)0, reconcilable_result_builder>(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >, query::partition_slice const&, reconcilable_result_builder&&, unsigned int, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, unsigned long) at ./querier.hh:87
auto seastar::internal::do_with_impl<flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >, do_query_mutations(seastar::sharded<database>&, seastar::lw_shared_ptr<schema const>, query::read_command const&, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, query::result_memory_accounter&&)::{lambda(seastar::shared_ptr<read_context>&)#1}::operator()(seastar::shared_ptr<read_context>&)::{lambda()#1}::operator()()::{lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)#2}>(flat_mutation_reader&&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&&, do_query_mutations(seastar::sharded<database>&, seastar::lw_shared_ptr<schema const>, query::read_command const&, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, query::result_memory_accounter&&)::{lambda(seastar::shared_ptr<read_context>&)#1}::operator()(seastar::shared_ptr<read_context>&)::{lambda()#1}::operator()()::{lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)#2}&&) at multishard_mutation_query.cc:?
 (inlined by) __invoke_impl<seastar::future<(anonymous namespace)::page_consume_result>, do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)>&, flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&> at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) __invoke<do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)>&, flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&> at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) __apply_impl<do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)>&, std::tuple<flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> > >&, 0, 1> at /usr/include/c++/10/tuple:1724
 (inlined by) apply<do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)>&, std::tuple<flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> > >&> at /usr/include/c++/10/tuple:1736
 (inlined by) do_with_impl<flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >, do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)> > at ././seastar/include/seastar/core/do_with.hh:106
do_query_mutations(seastar::sharded<database>&, seastar::lw_shared_ptr<schema const>, query::read_command const&, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, query::result_memory_accounter&&)::{lambda(seastar::shared_ptr<read_context>&)#1}::operator()(seastar::shared_ptr<read_context>&)::{lambda()#1}::operator()() at multishard_mutation_query.cc:?
 (inlined by) futurize_invoke<seastar::future<(anonymous namespace)::page_consume_result> (*&)(flat_mutation_reader&&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&&, do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)>&&), flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >, do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)> > at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) do_with<flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >, do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)> > at ././seastar/include/seastar/core/do_with.hh:141
 (inlined by) operator() at ./multishard_mutation_query.cc:620
seastar::continuation<seastar::internal::promise_base_with_type<reconcilable_result>, seastar::future<reconcilable_result> seastar::future<>::then_impl_nrvo<do_query_mutations(seastar::sharded<database>&, seastar::lw_shared_ptr<schema const>, query::read_command const&, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, query::result_memory_accounter&&)::{lambda(seastar::shared_ptr<read_context>&)#1}::operator()(seastar::shared_ptr<read_context>&)::{lambda()#1}, seastar::future<reconcilable_result> >(do_query_mutations(seastar::sharded<database>&, seastar::lw_shared_ptr<schema const>, query::read_command const&, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, query::result_memory_accounter&&)::{lambda(seastar::shared_ptr<read_context>&)#1}::operator()(seastar::shared_ptr<read_context>&)::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<reconcilable_result>&, seastar::future_state<>&&)#1}>::run_and_dispose() at multishard_mutation_query.cc:?
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2724
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /usr/include/boost/program_options/variables_map.hpp:146
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at /usr/include/boost/program_options/variables_map.hpp:146
main at ./main.cc:485
?? ??:0
_start at ??:?

Node#4:

2020-06-11 02:30:15.000: (CoreDumpEvent Severity.ERROR): node=Node longevity-lwt-3h-master-db-node-ac2871f9-4 [54.171.242.120 | 10.0.218.22] (seed: False)
corefile_url=
https://storage.cloud.google.com/upload.scylladb.com/core.scylla.996.ed8705e0038b4ba1bc625289e0303fce.3957.1591842615000/core.scylla.996.ed8705e0038b4ba1bc625289e0303fce.3957.1591842615000000
backtrace=           PID: 3957 (scylla)
           UID: 996 (scylla)
           GID: 1001 (scylla)
        Signal: 11 (SEGV)
     Timestamp: Thu 2020-06-11 02:30:15 UTC (3min 1s ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 0-7 --lock-memory=1
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
          Unit: scylla-server.service
         Slice: scylla-server.slice
       Boot ID: ed8705e0038b4ba1bc625289e0303fce
    Machine ID: df877a200226bc47d06f26dae0736ec9
      Hostname: longevity-lwt-3h-master-db-node-ac2871f9-4
      Coredump: /var/lib/systemd/coredump/core.scylla.996.ed8705e0038b4ba1bc625289e0303fce.3957.1591842615000000
       Message: Process 3957 (scylla) of user 996 dumped core.
...
download_instructions=
gsutil cp gs://upload.scylladb.com/core.scylla.996.ed8705e0038b4ba1bc625289e0303fce.3957.1591842615000/core.scylla.996.ed8705e0038b4ba1bc625289e0303fce.3957.1591842615000000 .
gunzip /var/lib/systemd/coredump/core.scylla.996.ed8705e0038b4ba1bc625289e0303fce.3957.1591842615000000

Backtrace:

2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: Segmentation fault on shard 0.
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: Backtrace:
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000002f6b982
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000002f0ef90
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000002f0f235
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000002f0f2d0
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x00007ffb5e93ba8f
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000001f4ee60
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x000000000101724f
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000001018b5d
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x000000000113c02c
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x000000000113d2ca
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x00000000011478f5
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000001149e6d
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x000000000114a380
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x000000000111784c
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x000000000113e700
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x000000000113f898
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x000000000114200d
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x000000000114351d
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000001144fc4
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x00000000011451bd
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x000000000114567b
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x00000000011457a4
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x00000000010a9a5f
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x00000000027003da
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000002707b28
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x000000000270d26c
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000002f0c547
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000002f0c8ce
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000002f46e6d
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000002ea1db5
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000002ea24ae
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000000d70ab0
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000027041
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: 0x0000000000c8662d
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | kernel: scylla[3957]: segfault at 19e ip 0000000001f4ee61 sp 00007ffc758efcd0 error 4 in scylla[c86000+2758000]
2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | kernel: Code: 89 54 24 08 48 89 0c 24 e8 fc 9b f3 ff 48 8b 54 24 08 48 8b 0c 24 49 89 c0 44 0f be 72 0f 45 84 f6 79 07 48 8b 02 44 8b 70 08 <41> 0f b6 44 24 0f 44 89 f5 4c 89 e7 84 c0 0f 89 9b 00 00 00 49 8b

Translated:

2020-06-11T02:30:15+00:00  longevity-lwt-3h-master-db-node-ac2871f9-4 !INFO    | scylla: Segmentation fault on shard 0.

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/fmt/format.h:2188
seastar::backtrace_buffer::append_backtrace() at /usr/include/fmt/format.h:2188
 (inlined by) print_with_backtrace at ./build/release/seastar/./seastar/src/core/reactor.cc:763
seastar::print_with_backtrace(char const*) at /usr/include/fmt/format.h:2188
sigsegv_action at /usr/include/fmt/format.h:2188
 (inlined by) operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:3460
 (inlined by) _FUN at ./build/release/seastar/./seastar/src/core/reactor.cc:3456
?? ??:0
managed_bytes::size() const at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) managed_bytes::operator std::basic_string_view<signed char, std::char_traits<signed char> >() const at ./utils/managed_bytes.hh:341
 (inlined by) compound_wrapper<partition_key, partition_key_view>::equal(schema const&, partition_key const&) const at ./keys.hh:304
 (inlined by) query::specific_ranges::range_for(schema const&, partition_key const&) const at ./query-request.hh:101
 (inlined by) query::partition_slice::row_ranges(schema const&, partition_key const&) const at ./query.cc:162
query::clustering_key_filter_ranges::get_ranges(schema const&, query::partition_slice const&, partition_key const&) at ./clustering_key_filter.hh:60
 (inlined by) scanning_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() const at ./memtable.cc:433
seastar::future<> seastar::futurize<seastar::future<> >::invoke<scanning_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&>(scanning_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&) at ././seastar/include/seastar/core/future.hh:1865
 (inlined by) auto seastar::futurize_invoke<scanning_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&>(scanning_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&) at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) seastar::future<> seastar::do_until<scanning_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}, scanning_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}>(scanning_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, scanning_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}) at ././seastar/include/seastar/core/future-util.hh:525
 (inlined by) scanning_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./memtable.cc:447
mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at mutation_reader.cc:?
 (inlined by) flat_mutation_reader::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./flat_mutation_reader.hh:319
 (inlined by) mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at ./mutation_reader.cc:361
mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at ././seastar/include/seastar/core/future.hh:1865
 (inlined by) futurize_invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) parallel_for_each<mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at ././seastar/include/seastar/core/future-util.hh:159
 (inlined by) parallel_for_each_impl<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at ././seastar/include/seastar/core/future-util.hh:204
 (inlined by) invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at ././seastar/include/seastar/core/future.hh:1865
 (inlined by) futurize_invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) parallel_for_each<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at ././seastar/include/seastar/core/future-util.hh:216
 (inlined by) mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./mutation_reader.cc:353
mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&>(combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&) at mutation_reader.cc:?
 (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./mutation_reader.cc:108
 (inlined by) operator() at ./mutation_reader.cc:578
 (inlined by) invoke<combined_mutation_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at ././seastar/include/seastar/core/future.hh:1865
combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./mutation_reader.cc:589
(anonymous namespace)::shard_reader::remote_reader::do_fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) do_fill_buffer at ./mutation_reader.cc:1258
auto seastar::internal::do_with_impl<flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(nonwrapping_range<dht::ring_position> const&, bool, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)#1}>(flat_mutation_reader&&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&&, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(nonwrapping_range<dht::ring_position> const&, bool, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)#1}&&) at mutation_reader.cc:?
 (inlined by) operator() at ./mutation_reader.cc:1355
 (inlined by) __invoke_impl<seastar::future<(anonymous namespace)::shard_reader::fill_buffer_result>, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&> at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) __invoke<(anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&> at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) __apply_impl<(anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, std::tuple<flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> > >&, 0, 1> at /usr/include/c++/10/tuple:1724
 (inlined by) apply<(anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&, std::tuple<flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> > >&> at /usr/include/c++/10/tuple:1736
 (inlined by) do_with_impl<flat_mutation_reader, seastar::circular_buffer<mutation_fragment>, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)> > at ././seastar/include/seastar/core/do_with.hh:106
(anonymous namespace)::shard_reader::remote_reader::fill_buffer(nonwrapping_range<dht::ring_position> const&, bool, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) futurize_invoke<seastar::future<(anonymous namespace)::shard_reader::fill_buffer_result> (*&)(flat_mutation_reader&&, seastar::circular_buffer<mutation_fragment>&&, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment>&)>&&), flat_mutation_reader, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)> > at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) do_with<flat_mutation_reader, seastar::circular_buffer<mutation_fragment>, (anonymous namespace)::shard_reader::remote_reader::fill_buffer(const partition_range&, bool, seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&, seastar::circular_buffer<mutation_fragment, std::allocator<mutation_fragment> >&)> > at ././seastar/include/seastar/core/do_with.hh:141
 (inlined by) fill_buffer at ./mutation_reader.cc:1360
(anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const at mutation_reader.cc:?
(anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) submit_to<(anonymous namespace)::shard_reader::do_fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at ././seastar/include/seastar/core/smp.hh:333
 (inlined by) submit_to<(anonymous namespace)::shard_reader::do_fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at ././seastar/include/seastar/core/smp.hh:360
 (inlined by) do_fill_buffer at ./mutation_reader.cc:1397
(anonymous namespace)::shard_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
multishard_combining_reader::handle_empty_reader_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() const at mutation_reader.cc:?
multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) futurize_invoke<multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) do_until<multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>, multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at ././seastar/include/seastar/core/future-util.hh:525
 (inlined by) multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./mutation_reader.cc:1629
flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /usr/include/fmt/format.h:2188
 (inlined by) flat_mutation_reader::peek(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at ./flat_mutation_reader.hh:457
 (inlined by) auto query::consume_page<(emit_only_live_rows)0, reconcilable_result_builder>(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >, query::partition_slice const&, reconcilable_result_builder&&, unsigned int, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, unsigned long) at ./querier.hh:87
auto seastar::internal::do_with_impl<flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >, do_query_mutations(seastar::sharded<database>&, seastar::lw_shared_ptr<schema const>, query::read_command const&, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, query::result_memory_accounter&&)::{lambda(seastar::shared_ptr<read_context>&)#1}::operator()(seastar::shared_ptr<read_context>&)::{lambda()#1}::operator()()::{lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)#2}>(flat_mutation_reader&&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&&, do_query_mutations(seastar::sharded<database>&, seastar::lw_shared_ptr<schema const>, query::read_command const&, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, query::result_memory_accounter&&)::{lambda(seastar::shared_ptr<read_context>&)#1}::operator()(seastar::shared_ptr<read_context>&)::{lambda()#1}::operator()()::{lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)#2}&&) at multishard_mutation_query.cc:?
 (inlined by) __invoke_impl<seastar::future<(anonymous namespace)::page_consume_result>, do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)>&, flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&> at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) __invoke<do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)>&, flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&> at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) __apply_impl<do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)>&, std::tuple<flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> > >&, 0, 1> at /usr/include/c++/10/tuple:1724
 (inlined by) apply<do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)>&, std::tuple<flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> > >&> at /usr/include/c++/10/tuple:1736
 (inlined by) do_with_impl<flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >, do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)> > at ././seastar/include/seastar/core/do_with.hh:106
do_query_mutations(seastar::sharded<database>&, seastar::lw_shared_ptr<schema const>, query::read_command const&, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, query::result_memory_accounter&&)::{lambda(seastar::shared_ptr<read_context>&)#1}::operator()(seastar::shared_ptr<read_context>&)::{lambda()#1}::operator()() at multishard_mutation_query.cc:?
 (inlined by) futurize_invoke<seastar::future<(anonymous namespace)::page_consume_result> (*&)(flat_mutation_reader&&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&&, do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)>&&), flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >, do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)> > at ././seastar/include/seastar/core/future.hh:1949
 (inlined by) do_with<flat_mutation_reader, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >, do_query_mutations(seastar::distributed<database>&, schema_ptr, const query::read_command&, const partition_range_vector&, tracing::trace_state_ptr, seastar::lowres_clock::time_point, query::result_memory_accounter&&)::<lambda(seastar::shared_ptr<read_context>&)> mutable::<lambda()> mutable::<lambda(flat_mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)0> >&)> > at ././seastar/include/seastar/core/do_with.hh:141
 (inlined by) operator() at ./multishard_mutation_query.cc:620
seastar::continuation<seastar::internal::promise_base_with_type<reconcilable_result>, seastar::future<reconcilable_result> seastar::future<>::then_impl_nrvo<do_query_mutations(seastar::sharded<database>&, seastar::lw_shared_ptr<schema const>, query::read_command const&, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, query::result_memory_accounter&&)::{lambda(seastar::shared_ptr<read_context>&)#1}::operator()(seastar::shared_ptr<read_context>&)::{lambda()#1}, seastar::future<reconcilable_result> >(do_query_mutations(seastar::sharded<database>&, seastar::lw_shared_ptr<schema const>, query::read_command const&, std::vector<nonwrapping_range<dht::ring_position>, std::allocator<nonwrapping_range<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, query::result_memory_accounter&&)::{lambda(seastar::shared_ptr<read_context>&)#1}::operator()(seastar::shared_ptr<read_context>&)::{lambda()#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<reconcilable_result>&, seastar::future_state<>&&)#1}>::run_and_dispose() at multishard_mutation_query.cc:?
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2724
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /usr/include/boost/program_options/variables_map.hpp:146
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at /usr/include/boost/program_options/variables_map.hpp:146
main at ./main.cc:485
?? ??:0
_start at ??:?

Logs:

+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                           Log links for testrun with test id ac2871f9-8e37-456f-a05d-98baaf02619d                                                                                            |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Date            | Log type    | Link                                                                                                                                                                                                                         |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 20190101_010101 | prometheus  | https://cloudius-jenkins-test.s3.amazonaws.com/ac2871f9-8e37-456f-a05d-98baaf02619d/prometheus_snapshot_20200611_025629.tar.gz                                                                                               |
| 20200611_024125 | grafana     | https://cloudius-jenkins-test.s3.amazonaws.com/ac2871f9-8e37-456f-a05d-98baaf02619d/20200611_024125/grafana-screenshot-overview-20200611_024501-longevity-lwt-3h-master-monitor-node-ac2871f9-1.png                          |
| 20200611_024125 | grafana     | https://cloudius-jenkins-test.s3.amazonaws.com/ac2871f9-8e37-456f-a05d-98baaf02619d/20200611_024125/grafana-screenshot-scylla-per-server-metrics-nemesis-20200611_024125-longevity-lwt-3h-master-monitor-node-ac2871f9-1.png |
| 20200611_024912 | grafana     | https://cloudius-jenkins-test.s3.amazonaws.com/ac2871f9-8e37-456f-a05d-98baaf02619d/20200611_024912/grafana-screenshot-overview-20200611_025217-longevity-lwt-3h-master-monitor-node-ac2871f9-1.png                          |
| 20200611_024912 | grafana     | https://cloudius-jenkins-test.s3.amazonaws.com/ac2871f9-8e37-456f-a05d-98baaf02619d/20200611_024912/grafana-screenshot-scylla-per-server-metrics-nemesis-20200611_024912-longevity-lwt-3h-master-monitor-node-ac2871f9-1.png |
| 20200611_025645 | db-cluster  | https://cloudius-jenkins-test.s3.amazonaws.com/ac2871f9-8e37-456f-a05d-98baaf02619d/20200611_025645/db-cluster-ac2871f9.zip                                                                                                  |
| 20200611_025645 | loader-set  | https://cloudius-jenkins-test.s3.amazonaws.com/ac2871f9-8e37-456f-a05d-98baaf02619d/20200611_025645/loader-set-ac2871f9.zip                                                                                                  |
| 20200611_025645 | monitor-set | https://cloudius-jenkins-test.s3.amazonaws.com/ac2871f9-8e37-456f-a05d-98baaf02619d/20200611_025645/monitor-set-ac2871f9.zip                                                                                                 |
| 20200611_025645 | sct-runner  | https://cloudius-jenkins-test.s3.amazonaws.com/ac2871f9-8e37-456f-a05d-98baaf02619d/20200611_025645/sct-runner-ac2871f9.zip                                                                                                  |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

ShlomiBalalis on 15 Jun 2020

I'm clueless... I can see from gdb that cache_flat_mutation_reader uses some LSA linearized memory and eventually fails when trying to extract a clustering key from managed_bytes, but the stacktrace is so long and optimized out that I'm not able to spot the problem.

Also, I'm unfortunately unable to run scylla-gdb.py commands, even with all the freshest tricks from https://github.com/scylladb/scylla/blob/master/docs/debugging.md applied - it still complains that it can't locate thread-local storage.

psarna on 15 Jun 2020

@tgrabiec, @denesb could this be caused by https://github.com/scylladb/scylla/issues/6637?

bhalevy on 15 Jun 2020

@bhalevy No, keys are small here.

tgrabiec on 15 Jun 2020

This issue seems to reoccur in the latest lwt-3h job, where two segmentation faults occurred in two nodes::

@ShlomiBalalis what's scylla's build-id? same as the one in the original report?
Please make sure to identify the build-id and scylla git version otherwise we can't even start to debug any core dump

bhalevy on 15 Jun 2020

@bhalevy it's definitely different from the original. I extracted the info from cluster logs (database.log):

Scylla version 666.development-0.20200610.65d3e3992 with build-id e42da3a7b88f607f972cc487f08ce35faaa60fe5 starting ...

, but the https://github.com/scylladb/scylla-pkg/#looking-for-a-build script returns an empty directory when I used it to look up and download the version... so maybe it has never been uploaded yet.

psarna on 15 Jun 2020

@bhalevy it's definitely different from the original. I extracted the info from cluster logs (database.log):
Scylla version 666.development-0.20200610.65d3e3992 with build-id e42da3a7b88f607f972cc487f08ce35faaa60fe5 starting ...
, but the https://github.com/scylladb/scylla-pkg/#looking-for-a-build script returns an empty directory when I used it to look up and download the version... so maybe it has never been uploaded yet.

http://downloads.scylladb.com/relocatable/unstable/master/2020-06-10T13:31:52Z/ just looks empty due to https://github.com/scylladb/scylla-pkg/issues/1365
http://downloads.scylladb.com/relocatable/unstable/master/2020-06-10T13:31:52Z// will serve you better.

bhalevy on 15 Jun 2020

In any case I downloaded http://downloads.scylladb.com.s3.amazonaws.com/relocatable/unstable/master/2020-06-07T17%3A36%3A22Z//scylla-package.tar.gz to debug the origin core dump (gs://upload.scylladb.com/core.scylla.996.d36fdba55cb44fc98409492539496158.3898.1591613633000000/core.scylla.996.d36fdba55cb44fc98409492539496158.3898.1591613633000000.gz)
and it seems to go well so far.

bhalevy on 15 Jun 2020

(gdb) frame
#0  managed_bytes::managed_bytes (this=<optimized out>, o=...) at ./utils/managed_bytes.hh:251
(gdb) p blob_src
$13 = (blob_storage *) 0xd000000000000000
(gdb) p size_src
$16 = 0
(gdb) p s
$17 = 64
(gdb) p offs_dst
$18 = 0

bhalevy on 15 Jun 2020

(gdb) p *this
value has been optimized out
(gdb) p next_dst
$22 = (blob_storage::ref_type *) 0x7f184cfe55d2
(gdb) p *next_dst
$23 = {
  ptr = 0x601000b50d80
}
(gdb) p blob_dst
$24 = (blob_storage *) 0x0

bhalevy on 15 Jun 2020

@denesb can you please look into this?

bhalevy on 16 Jun 2020

$13 = (blob_storage *) 0xd000000000000000

This doesn't look like a valid address.

In frame 11 (position_in_partition::for_range_start(const query::clustering_range& r)):

(gdb) p &r
$12 = (const query::clustering_range *) 0x413e59

This doesn't look like a valid address either.

(gdb) p r._range._start._M_payload._M_engaged 
$14 = 73

A corrupt value as expected.

denesb on 16 Jun 2020

In frame 13 cache::cache_flat_mutation_reader::move_to_next_range (cache_flat_mutation_reader.hh:546):

(gdb) p _ck_ranges_curr
$16 = {
  _M_current = 0x413e34 <_ZTVN7seastar12continuationINS_8internal22promise_base_with_typeIJNS_11foreign_ptrISt10unique_ptrIN13cql_transport8responseESt14default_deleteIS6_EEEEEEEZZNS_6futureIJSA_EE17then_wrapped_nrvoISD_NSD_12finally_bodyIZNS_3smp9submit
_toIZNS_7shardedINS5_10cql_serverEE9invoke_onIZNSJ_10connection16process_on_shardIPFNSC_IJSt7variantIJSA_jEEEEERN7service12client_stateERNSI_IN4cql315query_processorEEENS5_14request_readerEth24cql_serialization_formatRK14timeout_config14service_permitN7t
racing15trace_state_ptrEbEEESD_jtN27fragmented_temporary_buffer7istreamEST_S13_S15_T_EUlRSJ_E_JESD_EET1_jNS_21smp_submit_to_optionsEOS1A_DpOT0_EUlvE_EENS_8futurizeINSt9result_ofIFS1A_vEE4typeEE4typeEjS1E_S1F_EUlvE_Lb0EEEEENS1K_IS1A_E4typeEOT0_ENKUlvE_clE
vEUlRSB_ONS_12future_stateIJSA_EEEE_JSA_EEE+4>
}

The _ck_ranges_curr iterator is corrupt.

denesb on 16 Jun 2020

_ck_ranges seems to reference a valid memory area, which contains a corrupt vector:

(gdb) p _ck_ranges
$17 = {
  _storage = {
    <std::_Vector_base<nonwrapping_range<clustering_key_prefix>, std::allocator<nonwrapping_range<clustering_key_prefix> > >> = {
      _M_impl = {
        <std::allocator<nonwrapping_range<clustering_key_prefix> >> = {
          <__gnu_cxx::new_allocator<nonwrapping_range<clustering_key_prefix> >> = {<No data fields>}, <No data fields>},
        <std::_Vector_base<nonwrapping_range<clustering_key_prefix>, std::allocator<nonwrapping_range<clustering_key_prefix> > >::_Vector_impl_data> = {
          _M_start = 0x0,
          _M_finish = 0x0,
          _M_end_of_storage = 0x0
        }, <No data fields>}
    }, <No data fields>},
  _ref = {
    _M_data = 0x601009306640
  }
}

(gdb) scylla ptr 0x601009306640
thread 1, small (size <= 160), live (0x601009306640 +0)

(gdb) p *_ck_ranges._ref._M_data
$20 = {
  <std::_Vector_base<nonwrapping_range<clustering_key_prefix>, std::allocator<nonwrapping_range<clustering_key_prefix> > >> = {
    _M_impl = {
      <std::allocator<nonwrapping_range<clustering_key_prefix> >> = {
        <__gnu_cxx::new_allocator<nonwrapping_range<clustering_key_prefix> >> = {<No data fields>}, <No data fields>},
      <std::_Vector_base<nonwrapping_range<clustering_key_prefix>, std::allocator<nonwrapping_range<clustering_key_prefix> > >::_Vector_impl_data> = {
        _M_start = 0x413da0 <vtable for seastar::continuation<seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::unique_ptr<cql_transport::response, std::default_delete<cql_transport::response> > > >, seastar::future<std::variant<seastar
::foreign_ptr<std::unique_ptr<cql_transport::response, std::default_delete<cql_transport::response> > >, unsigned int> >::then_impl_nrvo<cql_transport::cql_server::connection::process<seastar::future<std::variant<seastar::foreign_ptr<std::unique_ptr<cql_
transport::response, std::default_delete<cql_transport::response> > >, unsigned int> > (*)(service::client_state&, seastar::sharded<cql3::query_processor>&, cql_transport::request_reader, unsigned short, unsigned char, cql_serialization_format, timeout_c
onfig const&, service_permit, tracing::trace_state_ptr, bool)>(unsigned short, cql_transport::request_reader, service::client_state&, service_permit, tracing::trace_state_ptr, seastar::future<std::variant<seastar::foreign_ptr<std::unique_ptr<cql_transpor
t::response, std::default_delete<cql_transport::response> > >, unsigned int> > (*)(service::client_state&, seastar::sharded<cql3::query_processor>&, cql_transport::request_reader, unsigned short, unsigned char, cql_serialization_format, timeout_config co
nst&, service_permit, tracing::trace_state_ptr, bool))::{lambda(std::variant<seastar::foreign_ptr<std::unique_ptr<cql_transport::response, std::default_delete<cql_transport::response> > >, unsigned int>)#1}, seastar::future<seastar::foreign_ptr<std::uniq
ue_ptr<cql_transport::response, std::default_delete<cql_transport::response> > > > >(cql_transport::cql_server::connection::process<seastar::future<std::variant<seastar::foreign_ptr<std::unique_ptr<cql_transport::response, std::default_delete<cql_transpo
rt::response> > >, unsigned int> > (*)(service::client_state&, seastar::sharded<cql3::query_processor>&, cql_transport::request_reader, unsigned short, unsigned char, cql_serialization_format, timeout_config const&, service_permit, tracing::trace_state_p
tr, bool)>(unsigned short, cql_transport::request_reader, service::client_state&, service_permit, tracing::trace_state_ptr, seastar::future<std::variant<seastar::foreign_ptr<std::unique_ptr<cql_transport::response, std::default_delete<cql_transport::resp
onse> > >, unsigned int> > (*)(service::client_state&, seastar::sharded<cql3::query_processor>&, cql_transport::request_reader, unsigned short, unsigned char, cql_serialization_format, timeout_config const&, service_permit, tracing::trace_state_ptr, bool
))::{lambda(std::variant<seastar::foreign_ptr<std::unique_ptr<cql_transport::response, std::default_delete<cql_transport::response> > >, unsigned int>)#1}&&)::{lambda()#1}::operator()() const::{lambda(seastar::internal::promise_base_with_type<seastar::fo
reign_ptr<std::unique_ptr<cql_transport::response, std::default_delete<cql_transport::response> > > >&, seastar::future_state<std::variant<seastar::foreign_ptr<std::unique_ptr<cql_transport::response, std::default_delete<cql_transport::response> > >, uns
igned int> >&&)#1}, std::variant<seastar::foreign_ptr<std::unique_ptr<cql_transport::response, std::default_delete<cql_transport::response> > >, unsigned int> >+16>,
        _M_finish = 0x13cb00000005,
        _M_end_of_storage = 0x1
      }, <No data fields>}
  }, <No data fields>}

denesb on 16 Jun 2020

The memory at 0x601009306640 seems to contain a continuation which belongs to legit-looking fiber.

denesb on 16 Jun 2020

It appears the _slice member of the read_context is a dangling reference.

denesb on 16 Jun 2020

The _range field appears to be corrupt as well, but other fields look sane.

denesb on 16 Jun 2020

@roydahan this looks like use-after-free, any chance we can try to reproduce this using debug build syclla?

bhalevy on 17 Jun 2020

@bhalevy i doubt we will be able to start the Scylla debug in SCT.
Last time @juliayakovlev tried it, it didn’t work.

roydahan on 17 Jun 2020

After some further investigation I managed to narrow the problem down, but haven't managed to pinpoint the faulty code yet.

# querier cache stats
(gdb) p $19->_querier_cache._stats
$20 = {
  inserts = 111,
  lookups = 215,
  misses = 129,
  drops = 43,
  time_based_evictions = 7,
  resource_based_evictions = 0,
  memory_based_evictions = 0,
  population = 18
}

# inactive read stats
(gdb) p $19->_read_concurrency_sem._inactive_read_stats
$26 = {
  permit_based_evictions = 0,
  population = 20
}

# cache::read_context
(gdb) p &$dereference_smart_ptr(_read_context)
$42 = (cache::read_context *) 0x601000da2e00
(gdb) p &$42->_range
$66 = (const dht::partition_range *) 0x601000a5fd00
(gdb) p &$42->_slice
$67 = (const query::partition_slice *) 0x601009306640

# read_context::reader_meta::remote_parts
(gdb) p &$9
$41 = (read_context::reader_meta::remote_parts *) 0x601000ef6a00
(gdb) p &$dereference_smart_ptr($9.range)
$10 = (const nonwrapping_range<dht::ring_position> *) 0x601000ab3e00
(gdb) p &$dereference_smart_ptr($9.slice)
$11 = (const query::partition_slice *) 0x601000abb500

# read_context
(gdb) p $4
$39 = (read_context *) 0x601000b2e290
(gdb) p $4._cmd.is_first_page
$37 = {
      _value = false
}
(gdb) p $4._cmd.query_uuid
$38 = "9e2f94de-51f1-426e-8034-0fc220807ce9"
(gdb) p &$4._cmd.slice
$32 = (query::partition_slice *) 0x601000dae028
(gdb) p &$collection_element($4->_ranges, 0)
$35 = (nonwrapping_range<dht::ring_position> *) 0x60100095f780

# shard_reader::remote_reader (not evicted yet in this page)
(gdb) p $14
$25 = ((anonymous namespace)::shard_reader::remote_reader * const) 0x60100090e1c0
(gdb) p $14->_pr
$15 = (const dht::partition_range *) 0x60100095f780
_range_override -> unengaged
(gdb) p $14->_ps
$17 = (const query::partition_slice &) 0x601000dae028
_slice_override -> unengaged

# shard_reader
(gdb) p this
$22 = ((anonymous namespace)::shard_reader * const) 0x601000ccbb60
(gdb) p this->_pr
$23 = (const dht::partition_range *) 0x60100095f780
(gdb) p this->_ps
$24 = (const query::partition_slice &) 0x601000dae028

This is the non-first page of a multi-page scan. It is not known whether the reader was successfully looked up on the start of the page or not. No reader or querier was evicted due to resource shortage yet.
In different parts of the reader stacks there are 3 different slice/range pairs in use:

In the top part of the reader stack (multishard reader down to read_context) seems to use slice and range passed down on this page.
reader_context::reader_meta::remote_parts has a different slice/range instance. This is expected as the read context creates a copy of the slice/range on each shard, before passing it to the reader it creates.
cache::read_context has yet another slice/range pointer. This is not expected, these should reference the slice/range from the remote parts (above). I don't know yet why this is the case.

denesb on 24 Jun 2020

The plot thickens. I confirmed that the pointers to the slice and range (that are corrupt) in the cache and memtable were indeed at one point allocated on the heap, very likely as part of reader_context::reader_meta::remote_parts. This means that these are not pointers to either the slice member of query::read_command or the dht::partition_ranges passed down on each page, and which don't survive across pages. We did allocate a copy on the heap to make the addresses stable... but then somehow discarded them.
However, when searching for the actual slice and range members of the current reader_context::reader_meta::remote_parts I found a live memtable and cache reader that is referencing them! So maybe we created two readers on the same shard, and the slice/range of the latter overridden those of the former? This smells like some unsynchronized-with read-ahead or similar.

denesb on 25 Jun 2020

This other live reader, is paused. I couldn't find a handle referencing it in the multishard reader or the read context. It looks like we leaked it.

denesb on 25 Jun 2020

Theory:

On page K:
1) request to create reader on shard s
2) save slice0 into reader_context::_readers[s]->rparts.slice (slice is an std::unique_ptr<const query::partition_slice>)
3) save range0 into reader_context::_readers[s]->rparts.range (range is an std::unique_ptr<const dht::partition_range>)
4) create reader r0 with references to saved copies of slice0 and range0.
5) call fill_buffer() on r0

6) request to create reader on shard s
7) save slice1 into reader_context::_readers[s]->rparts.slice, references to slice0 are now dangling
8) save range1 into reader_context::_readers[s]->rparts.range, references to range0 are now dangling
9) create reader r1 with references to saved copies of slice1 and range1.
10) call fill_buffer() on r1
11) r1.fill_buffer() finishes
12) pause r1, save the pause hande in remote_reader::_irh.

13) r0.fill_buffer() finishes
14) pause r0, save the pause handle in remote_reader::_irh. We now leaked the pause handle to r0.

15) save reader at the end of the page: we resume the reader paused via remote_reader::_irh. This is r0, with the dangling range and slice references.

On page K+1:
1) lookup reader r0 and start using it. Boom!

Started auditing the code for places where we could issue fill_buffer() without waiting for a previous one to finish.

denesb on 25 Jun 2020

@juliayakovlev @roydahan if I prepare a branch with additional assertions, will I be to start a build with parameters run using my repo/branch?

denesb on 25 Jun 2020

@juliayakovlev @roydahan if I prepare a branch with additional assertions, will I be to start a build with parameters run using my repo/branch?

Yes, it's possible to create staging job and configure to use your git branch

juliayakovlev on 25 Jun 2020

I ended up sending a patch because I think the assert I wanted to add is worth including in production code. Once the patch is merged, I will ask you to re-run the test with the hope that my assert will be tripped.

denesb on 25 Jun 2020

@denesb is it "mutation_reader: evictable_reader: add assert against pause handle leak"?

bhalevy on 28 Jun 2020

@bhalevy yes, it was committed already as https://github.com/scylladb/scylla/commit/e31f7316c0e5ceade9c165ca38eaf3857ef69569.

denesb on 29 Jun 2020

The issue reoccurred in a run of lwt-3h longevity of the master branch:
scylla version: 666.development-0.20200624.a9c7a1a86 with build-id c76e567b15f8881e56fdaa79b6f9e92a0bceaee3
(Does not seem to contain Botond's patch)

At 2020-06-25 00:55:33.000 node#2 experienced a segmentation fault:
backtrace:

2020-06-25T00:55:33+00:00  longevity-lwt-3h-master-db-node-fd51f372-2 !INFO    | scylla: Segmentation fault on shard 0.
scylla: Backtrace:
scylla: 0x0000000002eb52e2
scylla: 0x0000000002e59820
scylla: 0x0000000002e59ac5
scylla: 0x0000000002e59b60
scylla: 0x00007f4294cf3a8f
scylla: 0x0000000001f67550
scylla: 0x0000000000fbed9d
scylla: 0x0000000000fc008c
scylla: 0x0000000000fd7837
scylla: 0x0000000000fe3c49
scylla: 0x0000000000fe4b3b
scylla: 0x0000000000fe4ebd
scylla: 0x000000000115b54c
scylla: 0x000000000115c81a
scylla: 0x000000000115f695
scylla: 0x0000000001161c0d
scylla: 0x000000000116218a
scylla: 0x000000000112ad44
scylla: 0x000000000115dc13
scylla: 0x000000000115e1e7
scylla: 0x000000000115e86d
scylla: 0x00000000011353d6
scylla: 0x000000000114f887
scylla: 0x0000000001151a74
scylla: 0x0000000001153b4d
scylla: 0x000000000115400b
scylla: 0x0000000001154134
scylla: 0x00000000010b8930
scylla: 0x00000000010c12ed
scylla: 0x0000000002e56bc7
scylla: 0x0000000002e56f3e
scylla: 0x0000000002e8e05d
scylla: 0x0000000002deaeba
scylla: 0x0000000002deb56e
scylla: 0x0000000000d96db0
scylla: /opt/scylladb/libreloc/libc.so.6+0x0000000000027041
scylla: 0x0000000000cb3c0d
kernel: scylla[3896]: segfault at 10 ip 0000000001f67551 sp 00007ffd6a9f3fe0 error 4 in scylla[cb3000+2720000]
kernel: Code: 89 54 24 08 48 89 0c 24 e8 1c 04 f4 ff 48 8b 54 24 08 48 8b 0c 24 49 89 c0 44 0f be 72 0f 45 84 f6 79 07 48 8b 02 44 8b 70 08 <41> 0f b6 44 24 0f 44 89 f5 4c 89 e7 84 c0 0f 89 9b 00 00 00 49 8b

Translated:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/fmt/format.h:2188
seastar::backtrace_buffer::append_backtrace() at /usr/include/fmt/format.h:2188
 (inlined by) print_with_backtrace at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/src/core/reactor.cc:751
seastar::print_with_backtrace(char const*) at /usr/include/fmt/format.h:2188
sigsegv_action at /usr/include/fmt/format.h:2188
 (inlined by) operator() at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/src/core/reactor.cc:3448
 (inlined by) _FUN at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/src/core/reactor.cc:3444
?? ??:0
managed_bytes::size() const at /usr/include/c++/10/bits/stl_vector.h:1759
 (inlined by) managed_bytes::operator std::basic_string_view<signed char, std::char_traits<signed char> >() const at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/utils/managed_bytes.hh:341
 (inlined by) compound_wrapper<partition_key, partition_key_view>::equal(schema const&, partition_key const&) const at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/keys.hh:304
 (inlined by) query::specific_ranges::range_for(schema const&, partition_key const&) const at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/query-request.hh:101
 (inlined by) query::partition_slice::row_ranges(schema const&, partition_key const&) const at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/query.cc:162
cache_entry::do_read(row_cache&, cache::read_context&) at row_cache.cc:?
 (inlined by) cache_entry::do_read(row_cache&, cache::read_context&) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/row_cache.cc:1275
cache_entry::read(row_cache&, cache::read_context&) at row_cache.cc:?
scanning_and_populating_reader::read_from_entry(cache_entry&) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/row_cache.cc:592
 (inlined by) scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/row_cache.cc:619
std::result_of<scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const::{lambda()#1} ()>::type with_linearized_managed_bytes<scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const::{lambda()#1}>(scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const::{lambda()#1}&&) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/utils/managed_bytes.hh:428
 (inlined by) scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/row_cache.cc:644
 (inlined by) _ZN8logalloc18allocating_section24with_reclaiming_disabledIRZN30scanning_and_populating_reader20do_read_from_primaryENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_EEDcRNS_6regionEOT_ at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/./utils/logalloc.hh:746
 (inlined by) _ZZN8logalloc18allocating_sectionclIZN30scanning_and_populating_reader20do_read_from_primaryENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_EEDcRNS_6regionEOT_ENKUlvE_clEv at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/./utils/logalloc.hh:768
 (inlined by) _ZN8logalloc18allocating_section12with_reserveIZNS0_clIZN30scanning_and_populating_reader20do_read_from_primaryENSt6chrono10time_pointIN7seastar12lowres_clockENS4_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_EEDcRNS_6regionEOT_EUlvE_EEDcSH_ at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/./utils/logalloc.hh:718
 (inlined by) _ZN8logalloc18allocating_sectionclIZN30scanning_and_populating_reader20do_read_from_primaryENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_EEDcRNS_6regionEOT_ at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/./utils/logalloc.hh:769
 (inlined by) scanning_and_populating_reader::do_read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/row_cache.cc:645
 (inlined by) scanning_and_populating_reader::read_from_primary(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/row_cache.cc:649
scanning_and_populating_reader::read_next_partition(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/row_cache.cc:673
 (inlined by) scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() const at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/row_cache.cc:699
seastar::future<> seastar::futurize<seastar::future<> >::invoke<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) auto seastar::futurize_invoke<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) seastar::future<> seastar::do_until<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}, scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future-util.hh:525
 (inlined by) scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/row_cache.cc:707
mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at mutation_reader.cc:?
 (inlined by) flat_mutation_reader::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/flat_mutation_reader.hh:319
 (inlined by) mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/mutation_reader.cc:361
mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) futurize_invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) parallel_for_each<mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future-util.hh:159
 (inlined by) parallel_for_each_impl<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future-util.hh:204
 (inlined by) invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2026
 (inlined by) futurize_invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) parallel_for_each<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future-util.hh:216
 (inlined by) mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/mutation_reader.cc:353
mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&>(combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&) at mutation_reader.cc:?
 (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/mutation_reader.cc:108
 (inlined by) operator() at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/mutation_reader.cc:578
 (inlined by) invoke<combined_mutation_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2026
combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/mutation_reader.cc:589
evictable_reader::do_fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) evictable_reader::do_fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/mutation_reader.cc:1233
evictable_reader::fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
auto seastar::internal::do_with_impl<flat_mutation_reader, evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}>(flat_mutation_reader&&, evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}&&) at mutation_reader.cc:?
 (inlined by) __invoke_impl<seastar::future<>, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, flat_mutation_reader&> at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) __invoke<evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, flat_mutation_reader&> at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) __apply_impl<evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, std::tuple<flat_mutation_reader>&, 0> at /usr/include/c++/10/tuple:1724
 (inlined by) apply<evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, std::tuple<flat_mutation_reader>&> at /usr/include/c++/10/tuple:1736
 (inlined by) do_with_impl<flat_mutation_reader, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)> > at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/do_with.hh:106
evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) futurize_invoke<seastar::future<> (*&)(flat_mutation_reader&&, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&&), flat_mutation_reader, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)> > at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) do_with<flat_mutation_reader, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)> > at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/do_with.hh:141
 (inlined by) evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/mutation_reader.cc:1342
seastar::future<(anonymous namespace)::fill_buffer_result> seastar::futurize<seastar::future<(anonymous namespace)::fill_buffer_result> >::invoke<(anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&>((anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&) at mutation_reader.cc:?
 (inlined by) invoke<(anonymous namespace)::shard_reader::do_fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2026
(anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) submit_to<(anonymous namespace)::shard_reader::do_fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/smp.hh:360
 (inlined by) do_fill_buffer at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/mutation_reader.cc:1553
(anonymous namespace)::shard_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
multishard_combining_reader::handle_empty_reader_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() const at mutation_reader.cc:?
multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) futurize_invoke<multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2110
 (inlined by) do_until<multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>, multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future-util.hh:525
 (inlined by) multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/mutation_reader.cc:1778
seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() at /usr/include/fmt/format.h:2188
seastar::internal::do_until_state<seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}>::run_and_dispose() at /usr/include/fmt/format.h:2188
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::reactor::run() at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/seastar/src/core/reactor.cc:2712
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /usr/include/boost/program_options/variables_map.hpp:146
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at /usr/include/boost/program_options/variables_map.hpp:146
main at /tmp/jenkins-79211881/workspace/scylla-master/next/scylla/main.cc:488
?? ??:0
_start at ??:?

Coredump:

2020-06-25 00:55:33.000: (CoreDumpEvent Severity.ERROR): node=Node longevity-lwt-3h-master-db-node-fd51f372-2 [52.31.53.215 | 10.0.81.14] (seed: False)
corefile_url=
https://storage.cloud.google.com/upload.scylladb.com/core.scylla.996.a52588c0f20749e29cec36623040a38f.3896.1593046533000000/core.scylla.996.a52588c0f20749e29cec36623040a38f.3896.1593046533000000.gz
backtrace=           PID: 3896 (scylla)
           UID: 996 (scylla)
           GID: 1001 (scylla)
        Signal: 11 (SEGV)
     Timestamp: Thu 2020-06-25 00:55:33 UTC (1min 28s ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --cpuset 0-7 --lock-memory=1
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /scylla.slice/scylla-server.slice/scylla-server.service
          Unit: scylla-server.service
         Slice: scylla-server.slice
       Boot ID: a52588c0f20749e29cec36623040a38f
    Machine ID: df877a200226bc47d06f26dae0736ec9
      Hostname: longevity-lwt-3h-master-db-node-fd51f372-2
      Coredump: /var/lib/systemd/coredump/core.scylla.996.a52588c0f20749e29cec36623040a38f.3896.1593046533000000
       Message: Process 3896 (scylla) of user 996 dumped core.
...
download_instructions=
gsutil cp gs://upload.scylladb.com/core.scylla.996.a52588c0f20749e29cec36623040a38f.3896.1593046533000000/core.scylla.996.a52588c0f20749e29cec36623040a38f.3896.1593046533000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.996.a52588c0f20749e29cec36623040a38f.3896.1593046533000000.gz

logs:

+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                           Log links for testrun with test id fd51f372-46ae-4e8a-8973-f863138749df                                            |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------+
| Date            | Log type    | Link                                                                                                                         |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------+
| 20200625_045929 | db-cluster  | https://cloudius-jenkins-test.s3.amazonaws.com/fd51f372-46ae-4e8a-8973-f863138749df/20200625_045929/db-cluster-fd51f372.zip  |
| 20200625_045929 | loader-set  | https://cloudius-jenkins-test.s3.amazonaws.com/fd51f372-46ae-4e8a-8973-f863138749df/20200625_045929/loader-set-fd51f372.zip  |
| 20200625_045929 | monitor-set | https://cloudius-jenkins-test.s3.amazonaws.com/fd51f372-46ae-4e8a-8973-f863138749df/20200625_045929/monitor-set-fd51f372.zip |
| 20200625_045929 | sct-runner  | https://cloudius-jenkins-test.s3.amazonaws.com/fd51f372-46ae-4e8a-8973-f863138749df/20200625_045929/sct-runner-fd51f372.zip  |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------+

ShlomiBalalis on 29 Jun 2020

The issue reoccurred in a run of lwt-3h longevity of the master branch:
scylla version: 666.development-0.20200624.a9c7a1a86 with build-id c76e567b15f8881e56fdaa79b6f9e92a0bceaee3
(Does not seem to contain Botond's patch)

Yes, it should have crashed in a different place with my patch (provided my patch works as intended).

denesb on 30 Jun 2020

Waiting for QA to reproduce this again with the assert

slivne on 1 Jul 2020

there is a new ami - checking if the issue is hit with the additional assert code from botond, QA will run trying to reproduce

slivne on 9 Jul 2020

I just managed to reproduce this locally, of course as soon as I added more debug code I cannot reproduce anymore. All I did was run range scans against a cluster of three nodes to reproduce. I will try again.

denesb on 10 Jul 2020

Just for the record, the assert()I added wasn't triggered.

denesb on 10 Jul 2020

@denesb - since you were able to reproduce this, is there any point to get a core from QA - it doesn't seem so - how can we make progress here ?

slivne on 15 Jul 2020

The issue occurred again in lwt-3h:
scylla version: 666.development-0.20200710.13316234653 with build-id 0cd24e63604bdbce54b6e953cff65ad4e18f6c7b
cluster size: 4
Ami: ami-06149af2dee715fbb (Ireland)

The Segmentation fault happened on node#2 at 2020-07-11T00:49.

Backtrace:

2020-07-11T00:49:13+00:00  longevity-lwt-3h-master-db-node-95c6482e-2 !INFO    | scylla: Backtrace:
0x0000000002e2e242
0x0000000002dd32b0
0x0000000002dd3555
0x0000000002dd35f0
0x00007f397546aa8f
0x0000000000e0e353
0x0000000000fb95ef
0x0000000000fbcf5a
0x0000000000fc1dc3
0x0000000000fc3ae1
0x0000000000fc419b
0x0000000000fc3b4c
0x0000000001172dd7
0x0000000000fb3428
0x0000000000fb3995
0x000000000113afe1
0x000000000113c1da
0x000000000113f835
0x0000000001141925
0x000000000114225a
0x000000000110b3bc
0x000000000113de13
0x000000000113e3a5
0x000000000113ea0d
0x00000000011158c6
0x000000000111a25a
0x000000000112f2bb
0x0000000001131884
0x0000000001133fbd
0x000000000113447b
0x000000000113459c
0x000000000109ac90
0x00000000010a34cd
0x0000000002dd0707
0x0000000002dd0a7e
0x0000000002e074a5
0x0000000002e16b1a
0x0000000002d9b9ad
/opt/scylladb/libreloc/libpthread.so.0+0x0000000000009431
/opt/scylladb/libreloc/libc.so.6+0x00000000001019d2

Translated:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/include/fmt/format.h:2188
seastar::backtrace_buffer::append_backtrace() at /usr/include/fmt/format.h:2188
 (inlined by) print_with_backtrace at /jenkins/workspace/scylla-master/next/scylla/seastar/src/core/reactor.cc:751
seastar::print_with_backtrace(char const*) at /usr/include/fmt/format.h:2188
sigsegv_action at /usr/include/fmt/format.h:2188
 (inlined by) operator() at /jenkins/workspace/scylla-master/next/scylla/seastar/src/core/reactor.cc:3459
 (inlined by) _FUN at /jenkins/workspace/scylla-master/next/scylla/seastar/src/core/reactor.cc:3455
?? ??:0
managed_bytes::size() const at /usr/include/fmt/core.h:606
 (inlined by) managed_bytes::managed_bytes(managed_bytes const&) at /jenkins/workspace/scylla-master/next/scylla/./utils/managed_bytes.hh:233
compound_wrapper<clustering_key_prefix, clustering_key_prefix_view>::compound_wrapper(compound_wrapper<clustering_key_prefix, clustering_key_prefix_view> const&) at /jenkins/workspace/scylla-master/next/scylla/keys.hh:165
 (inlined by) prefix_compound_wrapper<clustering_key_prefix, clustering_key_prefix_view, clustering_key_prefix>::prefix_compound_wrapper(prefix_compound_wrapper<clustering_key_prefix, clustering_key_prefix_view, clustering_key_prefix> const&) at /jenkins/workspace/scylla-master/next/scylla/keys.hh:558
 (inlined by) clustering_key_prefix::clustering_key_prefix(clustering_key_prefix const&) at /jenkins/workspace/scylla-master/next/scylla/keys.hh:788
 (inlined by) std::_Optional_payload_base<clustering_key_prefix>::_Storage<clustering_key_prefix, false>::_Storage<clustering_key_prefix const&>(std::in_place_t, clustering_key_prefix const&) at /usr/include/c++/10/optional:226
 (inlined by) std::_Optional_payload_base<clustering_key_prefix>::_Optional_payload_base<clustering_key_prefix const&>(std::in_place_t, clustering_key_prefix const&) at /usr/include/c++/10/optional:118
 (inlined by) _ZNSt17_Optional_payloadI21clustering_key_prefixLb1ELb0ELb0EECI4St22_Optional_payload_baseIS0_EIJRKS0_EEESt10in_place_tDpOT_ at /usr/include/c++/10/optional:359
 (inlined by) _ZNSt17_Optional_payloadI21clustering_key_prefixLb0ELb0ELb0EECI4St22_Optional_payload_baseIS0_EIJRKS0_EEESt10in_place_tDpOT_ at /usr/include/c++/10/optional:393
 (inlined by) std::_Optional_base<clustering_key_prefix, false, false>::_Optional_base<clustering_key_prefix const&, false>(std::in_place_t, clustering_key_prefix const&) at /usr/include/c++/10/optional:485
 (inlined by) std::optional<clustering_key_prefix>::optional<clustering_key_prefix const&, true>(clustering_key_prefix const&) at /usr/include/c++/10/optional:702
 (inlined by) position_in_partition::position_in_partition(position_in_partition::range_tag_t, bound_view) at /jenkins/workspace/scylla-master/next/scylla/./position_in_partition.hh:249
 (inlined by) position_in_partition::for_range_start(nonwrapping_interval<clustering_key_prefix> const&) at /jenkins/workspace/scylla-master/next/scylla/./position_in_partition.hh:520
 (inlined by) cache::cache_flat_mutation_reader::move_to_range(__gnu_cxx::__normal_iterator<nonwrapping_interval<clustering_key_prefix> const*, std::vector<nonwrapping_interval<clustering_key_prefix>, std::allocator<nonwrapping_interval<clustering_key_prefix> > > >) at /jenkins/workspace/scylla-master/next/scylla/cache_flat_mutation_reader.hh:552
cache::cache_flat_mutation_reader::move_to_next_range() at /jenkins/workspace/scylla-master/next/scylla/cache_flat_mutation_reader.hh:546
 (inlined by) cache::cache_flat_mutation_reader::copy_from_cache_to_buffer() at /jenkins/workspace/scylla-master/next/scylla/cache_flat_mutation_reader.hh:529
 (inlined by) cache::cache_flat_mutation_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}::operator()() const at /jenkins/workspace/scylla-master/next/scylla/cache_flat_mutation_reader.hh:297
_ZZZN5cache11lsa_manager19run_in_read_sectionIZNS_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_ENKUlvE_clEvENKUlvE_clEv at /jenkins/workspace/scylla-master/next/scylla/row_cache.hh:590
 (inlined by) _Z29with_linearized_managed_bytesIZZN5cache11lsa_manager19run_in_read_sectionIZNS0_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS4_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_ENKUlvE_clEvEUlvE_ENSt9result_ofIFSE_vEE4typeEOSE_ at /jenkins/workspace/scylla-master/next/scylla/utils/managed_bytes.hh:428
 (inlined by) _ZZN5cache11lsa_manager19run_in_read_sectionIZNS_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_ENKUlvE_clEv at /jenkins/workspace/scylla-master/next/scylla/row_cache.hh:591
 (inlined by) _ZN8logalloc18allocating_section24with_reclaiming_disabledIRZN5cache11lsa_manager19run_in_read_sectionIZNS2_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS6_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_EUlvE_EEDcRNS_6regionEOSG_ at /jenkins/workspace/scylla-master/next/scylla/./utils/logalloc.hh:746
 (inlined by) _ZZN8logalloc18allocating_sectionclIZN5cache11lsa_manager19run_in_read_sectionIZNS2_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS6_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_EUlvE_EEDcRNS_6regionEOSG_ENKUlvE_clEv at /jenkins/workspace/scylla-master/next/scylla/./utils/logalloc.hh:768
 (inlined by) _ZN8logalloc18allocating_section12with_reserveIZNS0_clIZN5cache11lsa_manager19run_in_read_sectionIZNS3_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS7_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_EUlvE_EEDcRNS_6regionEOSH_EUlvE_EEDcSN_ at /jenkins/workspace/scylla-master/next/scylla/./utils/logalloc.hh:718
 (inlined by) _ZN8logalloc18allocating_sectionclIZN5cache11lsa_manager19run_in_read_sectionIZNS2_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS6_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_EUlvE_EEDcRNS_6regionEOSG_ at /jenkins/workspace/scylla-master/next/scylla/./utils/logalloc.hh:769
 (inlined by) _ZN5cache11lsa_manager19run_in_read_sectionIZNS_26cache_flat_mutation_reader14do_fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS3_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE1_EEDcRKT_ at /jenkins/workspace/scylla-master/next/scylla/row_cache.hh:592
 (inlined by) cache::cache_flat_mutation_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/cache_flat_mutation_reader.hh:306
cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}::operator()() const at /jenkins/workspace/scylla-master/next/scylla/cache_flat_mutation_reader.hh:237
 (inlined by) seastar::future<> seastar::futurize<seastar::future<> >::invoke<cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&>(cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&) at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2025
 (inlined by) auto seastar::futurize_invoke<cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&>(cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&) at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2109
 (inlined by) seastar::future<> seastar::do_until<cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}, cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}>(cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}, cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}) at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future-util.hh:529
 (inlined by) cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/cache_flat_mutation_reader.hh:238
cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const at /jenkins/workspace/scylla-master/next/scylla/cache_flat_mutation_reader.hh:227
cache::cache_flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/cache_flat_mutation_reader.hh:232
flat_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /usr/include/fmt/format.h:2188
 (inlined by) seastar::future<bool> flat_mutation_reader::impl::fill_buffer_from<flat_mutation_reader>(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/flat_mutation_reader.cc:197
scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() const at /jenkins/workspace/scylla-master/next/scylla/row_cache.cc:701
seastar::future<> seastar::futurize<seastar::future<> >::invoke<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&) at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2025
 (inlined by) auto seastar::futurize_invoke<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}&) at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2109
 (inlined by) seastar::future<> seastar::do_until<scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}, scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}>(scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}) at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future-util.hh:529
 (inlined by) scanning_and_populating_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/row_cache.cc:707
mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at mutation_reader.cc:?
 (inlined by) flat_mutation_reader::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/flat_mutation_reader.hh:319
 (inlined by) mutation_reader_merger::prepare_one(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, mutation_reader_merger::reader_and_last_fragment_kind, seastar::bool_class<mutation_reader_merger::reader_galloping_tag>) at /jenkins/workspace/scylla-master/next/scylla/mutation_reader.cc:361
mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2025
 (inlined by) futurize_invoke<mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2109
 (inlined by) parallel_for_each<mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future-util.hh:163
 (inlined by) parallel_for_each_impl<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future-util.hh:208
 (inlined by) invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2025
 (inlined by) futurize_invoke<seastar::future<> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2109
 (inlined by) parallel_for_each<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4>&, mutation_reader_merger::prepare_next(seastar::lowres_clock::time_point)::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future-util.hh:220
 (inlined by) mutation_reader_merger::prepare_next(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/mutation_reader.cc:353
mutation_reader_merger::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&>(combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}&) at mutation_reader.cc:?
 (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator()(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/mutation_reader.cc:108
 (inlined by) operator() at /jenkins/workspace/scylla-master/next/scylla/mutation_reader.cc:578
 (inlined by) invoke<combined_mutation_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2025
combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/mutation_reader.cc:589
evictable_reader::do_fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) evictable_reader::do_fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/mutation_reader.cc:1234
evictable_reader::fill_buffer(flat_mutation_reader&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
auto seastar::internal::do_with_impl<flat_mutation_reader, evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}>(flat_mutation_reader&&, evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda(flat_mutation_reader&)#1}&&) at mutation_reader.cc:?
 (inlined by) __invoke_impl<seastar::future<>, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, flat_mutation_reader&> at /usr/include/c++/10/bits/invoke.h:60
 (inlined by) __invoke<evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, flat_mutation_reader&> at /usr/include/c++/10/bits/invoke.h:96
 (inlined by) __apply_impl<evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, std::tuple<flat_mutation_reader>&, 0> at /usr/include/c++/10/tuple:1724
 (inlined by) apply<evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&, std::tuple<flat_mutation_reader>&> at /usr/include/c++/10/tuple:1736
 (inlined by) do_with_impl<flat_mutation_reader, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)> > at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/do_with.hh:106
evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) futurize_invoke<seastar::future<> (*&)(flat_mutation_reader&&, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)>&&), flat_mutation_reader, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)> > at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2109
 (inlined by) do_with<flat_mutation_reader, evictable_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda(flat_mutation_reader&)> > at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/do_with.hh:141
 (inlined by) evictable_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/mutation_reader.cc:1343
seastar::future<(anonymous namespace)::fill_buffer_result> seastar::futurize<seastar::future<(anonymous namespace)::fill_buffer_result> >::invoke<(anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&>((anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}&) at mutation_reader.cc:?
 (inlined by) invoke<(anonymous namespace)::shard_reader::do_fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2025
seastar::futurize<std::result_of<(anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3} ()>::type>::type seastar::smp::submit_to<(anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#3}>(unsigned int, seastar::smp_submit_to_options, std::result_of&&) [clone .constprop.0] at mutation_reader.cc:?
(anonymous namespace)::shard_reader::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) do_fill_buffer at /jenkins/workspace/scylla-master/next/scylla/mutation_reader.cc:1554
(anonymous namespace)::shard_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
multishard_combining_reader::handle_empty_reader_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() const at mutation_reader.cc:?
multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at mutation_reader.cc:?
 (inlined by) futurize_invoke<multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>&> at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future.hh:2109
 (inlined by) do_until<multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()>, multishard_combining_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at /jenkins/workspace/scylla-master/next/scylla/seastar/include/seastar/core/future-util.hh:529
 (inlined by) multishard_combining_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/next/scylla/mutation_reader.cc:1779
seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() at /usr/include/fmt/format.h:2188
seastar::internal::do_until_state<seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)0, query::clustering_position_tracker<reconcilable_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}>::run_and_dispose() at /usr/include/fmt/format.h:2188
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
seastar::reactor::run_some_tasks() at /usr/include/fmt/format.h:2188
 (inlined by) seastar::reactor::run() at /jenkins/workspace/scylla-master/next/scylla/seastar/src/core/reactor.cc:2723
seastar::smp::configure(boost::program_options::variables_map, seastar::reactor_config)::{lambda()#3}::operator()() const at /usr/include/fmt/format.h:2188
std::function<void ()>::operator()() const at /usr/include/c++/10/bits/basic_string.h:323
 (inlined by) seastar::posix_thread::start_routine(void*) at /jenkins/workspace/scylla-master/next/scylla/seastar/src/core/posix.cc:60
?? ??:0

Coredump:

PID: 3656 (scylla)
           UID: 996 (scylla)
           GID: 1001 (scylla)
        Signal: 11 (SEGV)
     Timestamp: Sat 2020-07-11 00:49:13 UTC (1min 54s ago)
  Command Line: /usr/bin/scylla --blocked-reactor-notify-ms 500 --abort-on-lsa-bad-alloc 1 --abort-on-seastar-bad-alloc --abort-on-internal-error 1 --abort-on-ebadf 1 --enable-sstable-key-validation 1 --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/
etc/scylla.d/io_properties.yaml --cpuset 0-7 --lock-memory=1
    Executable: /opt/scylladb/libexec/scylla
 Control Group: /
       Boot ID: 07108dc0d418414681e8d0e3e086e8f7
    Machine ID: df877a200226bc47d06f26dae0736ec9
      Hostname: longevity-lwt-3h-master-db-node-95c6482e-2
      Coredump: /var/lib/systemd/coredump/core.scylla.996.07108dc0d418414681e8d0e3e086e8f7.3656.1594428553000000
       Message: Process 3656 (scylla) of user 996 dumped core.
...
download_instructions=
gsutil cp gs://upload.scylladb.com/core.scylla.996.07108dc0d418414681e8d0e3e086e8f7.3656.1594428553000000/core.scylla.996.07108dc0d418414681e8d0e3e086e8f7.3656.1594428553000000.gz .
gunzip /var/lib/systemd/coredump/core.scylla.996.07108dc0d418414681e8d0e3e086e8f7.3656.1594428553000000.gz

Logs:

+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                           Log links for testrun with test id 95c6482e-c01c-46a3-aedd-7070a3ade01e                                            |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------+
| Date            | Log type    | Link                                                                                                                         |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------+
| 20200711_045542 | db-cluster  | https://cloudius-jenkins-test.s3.amazonaws.com/95c6482e-c01c-46a3-aedd-7070a3ade01e/20200711_045542/db-cluster-95c6482e.zip  |
| 20200711_045542 | loader-set  | https://cloudius-jenkins-test.s3.amazonaws.com/95c6482e-c01c-46a3-aedd-7070a3ade01e/20200711_045542/loader-set-95c6482e.zip  |
| 20200711_045542 | monitor-set | https://cloudius-jenkins-test.s3.amazonaws.com/95c6482e-c01c-46a3-aedd-7070a3ade01e/20200711_045542/monitor-set-95c6482e.zip |
| 20200711_045542 | sct-runner  | https://cloudius-jenkins-test.s3.amazonaws.com/95c6482e-c01c-46a3-aedd-7070a3ade01e/20200711_045542/sct-runner-95c6482e.zip  |
+-----------------+-------------+------------------------------------------------------------------------------------------------------------------------------+

ShlomiBalalis on 16 Jul 2020

@denesb - since you were able to reproduce this, is there any point to get a core from QA - it doesn't seem so - how can we make progress here ?

No, it will be much faster if I reproduce this locally, so no point in getting any more cores from QA.

denesb on 20 Jul 2020

Is there a reliable reproducer? If so we can start a bisect in parallel.

avikivity on 20 Jul 2020

I hit this accidentally while doing manual testing with ranges scans for [PATCH v4 00/21] Limit non-paged query memory consumption. As soon as I added some debug code, I couldn't reproduce any more. But this was late Friday afternoon, so I didn't spend a lot of time experimenting. When I reproduced, all I did was do range scans on a 3 node cluster on a roughly 4 GB dataset with scylla_bench.

denesb on 20 Jul 2020

I now have a reproducer that reproduces 100% of the time in a few seconds, however it is not trivial to automate it to a level which would facilitate easy bisecting. One has to start a three node cluster and start a range scan read workload, then observe the crash.

I started zeroing in on the bug with introspection code added to the range scan algorithm. I managed to trace the bug to save_reader() which seems to drop the range/slice. I hope I'm not chasing some ghost created by my introspection code.

denesb on 23 Jul 2020

👍1

Can you write down the reproducer commands? Perhaps someone else can start the bisect in parallel.

avikivity on 23 Jul 2020

I started a three node cluster, I have written data into it in multiple stages. I forgot the commands I used in the earlier stages, the last write command I have used was:

../go/scylla-bench/scylla-bench \
    -partition-count=6000 \
    -clustering-row-count=2000 \
    -clustering-row-size=1000 \
    -replication-factor=2 \
    -mode=write \
    -workload=sequential \
    -range-count=64000 \
    -concurrency=18 \
    -connection-count=18 \
    -nodes=127.0.0.1,127.0.0.2,127.0.0.3 \
    -timeout=60s

The range scan command which reproduces the crash:

../go/scylla-bench/scylla-bench \
    -partition-count=2000 \
    -clustering-row-count=2000 \
    -clustering-row-size=1000 \
    -replication-factor=2 \
    -mode=scan \
    -range-count=64000 \
    -concurrency=18 \
    -connection-count=18 \
    -nodes=127.0.0.1,127.0.0.2,127.0.0.3 \
    -timeout=60s

denesb on 23 Jul 2020

Scylla command lines:

./scylla --options-file=/home/bdenes/ScyllaDB/test-cluster/node1/scylla.yaml --developer-mode=1 --overprovisioned --unsafe-bypass-fsync=1 --relaxed-dma --listen-address=127.0.0.1 --rpc-address=127.0.0.1 --api-address=127.0.0.1 --prometheus-address=127.0.0.1 --seed-provider-parameters=seeds=127.0.0.1 --data-file-directories=/home/bdenes/ScyllaDB/test-cluster/node1/data --commitlog-directory=/home/bdenes/ScyllaDB/test-cluster/node1/commitlog --hints-directory=/home/bdenes/ScyllaDB/test-cluster/node1/hints --view-hints-directory=/home/bdenes/ScyllaDB/test-cluster/node1/view_hints
./scylla --options-file=/home/bdenes/ScyllaDB/test-cluster/node2/scylla.yaml --developer-mode=1 --overprovisioned --unsafe-bypass-fsync=1 --relaxed-dma --listen-address=127.0.0.2 --rpc-address=127.0.0.2 --api-address=127.0.0.2 --prometheus-address=127.0.0.2 --seed-provider-parameters=seeds=127.0.0.1 --data-file-directories=/home/bdenes/ScyllaDB/test-cluster/node2/data --commitlog-directory=/home/bdenes/ScyllaDB/test-cluster/node2/commitlog --hints-directory=/home/bdenes/ScyllaDB/test-cluster/node2/hints --view-hints-directory=/home/bdenes/ScyllaDB/test-cluster/node2/view_hints
./scylla --options-file=/home/bdenes/ScyllaDB/test-cluster/node3/scylla.yaml --developer-mode=1 --overprovisioned --unsafe-bypass-fsync=1 --relaxed-dma --listen-address=127.0.0.3 --rpc-address=127.0.0.3 --api-address=127.0.0.3 --prometheus-address=127.0.0.3 --seed-provider-parameters=seeds=127.0.0.1 --data-file-directories=/home/bdenes/ScyllaDB/test-cluster/node3/data --commitlog-directory=/home/bdenes/ScyllaDB/test-cluster/node3/commitlog --hints-directory=/home/bdenes/ScyllaDB/test-cluster/node3/hints --view-hints-directory=/home/bdenes/ScyllaDB/test-cluster/node3/view_hints

Heh, I just realized some of my args got dropped (--cpuset=1,2 -c2 -m4G --range-request-timeout-in-ms=60000 --logger-log-level multishard_mutation_query=trace), but I'm not touching anything at this point :laughing:.

denesb on 23 Jul 2020

scylla.yaml is copy paste from the scylla repo.

denesb on 23 Jul 2020

For current moment latest ami where issue was detected:
For ami-07962bc6eb944ef99 (us-east-1) from 1st of june issue reproduced but without coredump
666.development-0.20200521.c29ccdea7 with build-id 4e3de54c9e8a65011446f42451dd6266212d02e0

aleksbykov on 23 Jul 2020

To find a bug, I created a thread-local registry which maps slices and ranges to live readers using them. I wrapped the shard reader in a special reader which unregisters itself from these maps when destroyed. Then I wrapped the slice and range pointers with special ones that look into these maps to see if the readers referencing them are still alive when destroyed. After much tweaking it looks like I finally got an assert that seems to have enough information to find out what is happening.

The assert is triggered in read_context::save_reader() which apparently fails to unpause the reader (readers are handed to read_context::save_reader() in a paused state). This is expected -- it can happen that the reader was evicted in the meanwhile -- and in this case the reader is simply not saved and the range and slice is destroyed. But in this case the reader is still alive. After some reference following the reader turns out to be alive and well in reader_concurrency_semaphore::_inactive_readers (where paused readers are)... just on another semaphore. The reader is in the user semaphore, but read_context::save_reader() is looking for it in the system semaphore. This is likely a regression caused by https://github.com/scylladb/scylla/commit/0c6bbc84cd30df1c46a6cd81d87170b99d5a8144.

denesb on 23 Jul 2020

This aligns with what I saw in the core attached to this issue: a reader in use which doesn't has a dangling slice and range reference and a paused reader referencing the current range and slice. With enough reads going on the read handle will succeed in accidentally unpausing a reader from the wrong semaphore. Unpaused readers are simply keyed on a monotonically increased integer, it can easily happen that using that using the handle on another semaphore succeeds in retrieving a paused read, happening to be keyed with the same integer.

denesb on 23 Jul 2020

Opened https://github.com/scylladb/scylla/issues/6907 for the handle uniqueness issue.

denesb on 23 Jul 2020

Somehow we switch semaphore mid-read. Maybe caused by a switch in scheduling group?

denesb on 23 Jul 2020

The semaphores vector:

(gdb) p $12._semaphores
$21 = std::vector of length 32, capacity 32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6070074aedd8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
(gdb) p seastar::local_engine->_id
$22 = 7

This contains the user semaphore.

The semaphore on the permit stored in the reader_meta::remote_parts:

(gdb) p this->permit._semaphore                                                                                                                                                                                                                           
$20 = (reader_concurrency_semaphore *) 0x6070074af060

This is the system semaphore. Both are obtained the same way, by calling database::make_query_class_config(), just at different points in the code.

For a start, we should not obtain the semaphore more than one time, all code should just use the one from the semaphore vector. This alone should already fix this bug. But we should get an answer to the question of how the heck do we get different semaphores for the same read?! This might reveal another bug, either in the multishard reader code or somewhere else.

denesb on 23 Jul 2020

I found where the system semaphore comes from: the READ_DIGEST verb handler is executed in the system group due to a bug in the verb handling code in messaging service (also broken by https://github.com/scylladb/scylla/commit/0c6bbc84cd30df1c46a6cd81d87170b99d5a8144).

Now I just need to find out how we also get a user semaphore into the mix.

denesb on 23 Jul 2020

Opened https://github.com/scylladb/scylla/issues/6908 for the messaging service bug.

denesb on 23 Jul 2020

When the coordinator == replica and the messaging service is not involved, the scan will run in the statement group. It can happen that a page is read in the statement group, the reader is saved in the querier cache with the permit. The next page is read in the system group, the reader is looked up. The read_context will have the system semaphore, while the reader's permit will be for the user semaphore. We have two semaphores in play.

denesb on 23 Jul 2020

I think we have all the culprits now, I'll start working on the fixes.

denesb on 23 Jul 2020

👍1

the ami ami-029d8d23db17ae21e with scylla version: 666.development-0.20200521.c29ccdea7 with build-id 4e3de54c9e8a65011446f42451dd6266212d02e0 not reproduce issue.

aleksbykov on 23 Jul 2020

@aleksbykov i think you can stop the bisecting.
Your tests fits the findings from Botond.
The suspected commit was merged in May 29th, that’s why you’re able to reproduce it only on AMI starting on June 1st.

roydahan on 23 Jul 2020

Backported to 4.2.

avikivity on 27 Jul 2020

@penberg why is the bot re-adding the "Backport candidate" label?

avikivity on 9 Aug 2020

Was this page helpful?

0 / 5 - 0 ratings