Scylla: oversized allocation: 536870912 bytes - system.compaction_history huge size

Created on 28 Jan 2019  路  23Comments  路  Source: scylladb/scylla
  • [x] I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.

Installation details
Scylla version (or git commit hash): 3.0.0-0.20190116.2d181da65.el7.x86_64
Cluster size: 12
OS: CentOS Linux release 7.4.1708 (Core)

Hardware details (for performance issues) Delete if unneeded
Platform: Physical
Hardware: sockets=2 cores=72 hyperthreading=yes memory=512 (256GB allocated)
Disks: 3 * NVMe

The issues seems to start happeding I start or stop a nodetool cleanup

Jan 28 15:31:49 as-dfcas-3b12 scylla: [shard 2] seastar_memory - oversized allocation: 536870912 bytes, please report: at   0x7ccfa8#012  0x758343#012  0x75db46#012  0x75e094#012
 0x6b33c1#012  /opt/scylladb/lib64/libstdc++.so.6+0xcbf85#012  /opt/scylladb/lib64/libstdc++.so.6+0x124e08#012  /opt/scylladb/lib64/libstdc++.so.6+0x114541#012  0x4181a4e#012  0x4
1843a9#012  0x5a0cee#012  0x5a0eb1#012  0x66f547#012  0x673cba#012  0x756d4d#012  /lib64/libpthread.so.0+0x7e24#012  /lib64/libc.so.6+0xf834c
Jan 28 15:31:50 as-dfcas-3b12 scylla: [shard 0] rpc - client 10.46.163.31: fail to connect: Connection refused
Jan 28 15:31:50 as-dfcas-3b12 scylla: [shard 58] compaction - Compacted 2 sstables to [/var/lib/scylla/data/dragonfruit/ratesilo-20e3f110037f11e9812c000000000007/la-2775418-big-Da
ta.db:level=0, ]. 3890661303 bytes to 3023771355 (~77% of original) in 30763ms = 93.74MB/s. ~327168 total partitions merged to 267743.
Jan 28 15:31:53 as-dfcas-3b12 scylla: Reactor stalled for 16000 ms on shard 2.
Jan 28 15:31:53 as-dfcas-3b12 scylla: Backtrace:
Jan 28 15:31:53 as-dfcas-3b12 scylla: 0x00000000006a8212
Jan 28 15:31:53 as-dfcas-3b12 scylla: 0x00000000005bf93c
Jan 28 15:31:53 as-dfcas-3b12 scylla: 0x00000000005bfdf6
Jan 28 15:31:53 as-dfcas-3b12 scylla: /lib64/libpthread.so.0+0x000000000000f5df
Jan 28 15:31:53 as-dfcas-3b12 scylla: 0x000000000418a72f
Jan 28 15:31:53 as-dfcas-3b12 scylla: 0x0000000004182576
Jan 28 15:31:53 as-dfcas-3b12 scylla: 0x00000000041843a9
Jan 28 15:31:53 as-dfcas-3b12 scylla: 0x00000000005a0cee
Jan 28 15:31:53 as-dfcas-3b12 scylla: 0x00000000005a0eb1
Jan 28 15:31:53 as-dfcas-3b12 scylla: 0x000000000066f547
Jan 28 15:31:53 as-dfcas-3b12 scylla: 0x0000000000673cba
Jan 28 15:31:53 as-dfcas-3b12 scylla: 0x0000000000756d4d
Jan 28 15:31:53 as-dfcas-3b12 scylla: /lib64/libpthread.so.0+0x0000000000007e24
Jan 28 15:31:53 as-dfcas-3b12 scylla: /lib64/libc.so.6+0x00000000000f834c
bug
Source
picture linux-wizard

All 23 comments

I tried to decode on another node where I had debug package installed

[root@as-dfcas-3b06 ~]# addr2line -fCipe /usr/lib/debug/bin/scylla.debug 0x7ccfa8#012  0x758343#012  0x75db46#012  0x75e094#012 0x6b33c1#012  /opt/scylladb/lib64/libstdc++.so.6+0xcbf85#012  /opt/scylladb/lib64/libstdc++.so.6+0x124e08#012  /opt/scylladb/lib64/libstdc++.so.6+0x114541#012  0x4181a4e#012  0x41843a9#012  0x5a0cee#012  0x5a0eb1#012  0x66f547#012  0x673cba#012  0x756d4d#012  /lib64/libpthread.so.0+0x7e24#012  /lib64/libc.so.6+0xf834c

seastar::current_backtrace() at /usr/src/debug/scylla-3.0.0/seastar/util/backtrace.hh:56
 (inlined by) seastar::current_backtrace() at /usr/src/debug/scylla-3.0.0/seastar/util/backtrace.cc:84
seastar::memory::cpu_pages::warn_large_allocation(unsigned long) at /usr/src/debug/scylla-3.0.0/seastar/core/memory.cc:609
seastar::memory::cpu_pages::check_large_allocation(unsigned long) at /usr/src/debug/scylla-3.0.0/seastar/core/memory.cc:617
 (inlined by) seastar::memory::cpu_pages::allocate_large(unsigned int) at /usr/src/debug/scylla-3.0.0/seastar/core/memory.cc:623
 (inlined by) seastar::memory::allocate_large(unsigned long) at /usr/src/debug/scylla-3.0.0/seastar/core/memory.cc:1194
 (inlined by) seastar::memory::allocate(unsigned long) at /usr/src/debug/scylla-3.0.0/seastar/core/memory.cc:1244
operator new(unsigned long) at /usr/src/debug/scylla-3.0.0/seastar/core/memory.cc:1647
__gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) at /opt/scylladb/include/c++/7/ext/new_allocator.h:111
 (inlined by) std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) at /opt/scylladb/include/c++/7/bits/basic_string.tcc:1057
 (inlined by) std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long) at /opt/scylladb/include/c++/7/bits/basic_string.tcc:1078
 (inlined by) std::string::reserve(unsigned long) at /opt/scylladb/include/c++/7/bits/basic_string.tcc:960
?? ??:0
?? ??:0
?? ??:0
std::basic_ostream<char, std::char_traits<char> >& seastar::operator<< <char, unsigned int, 15u, true, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, seastar::basic_sstring<char, unsigned int, 15u, true> const&) at /usr/src/debug/scylla-3.0.0/seastar/core/sstring.hh:663
 (inlined by) seastar::basic_sstring<char, unsigned int, 15u, true> seastar::json::formatter::to_json<__gnu_cxx::__normal_iterator<seastar::httpd::compaction_manager_json::history const*, std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > > >(seastar::json::formatter::state, __gnu_cxx::__normal_iterator<seastar::httpd::compaction_manager_json::history const*, std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > >, __gnu_cxx::__normal_iterator<seastar::httpd::compaction_manager_json::history const*, std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > >) at /usr/src/debug/scylla-3.0.0/seastar/json/formatter.hh:69
 (inlined by) seastar::basic_sstring<char, unsigned int, 15u, true> seastar::json::formatter::to_json<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > const&) at /usr/src/debug/scylla-3.0.0/seastar/json/formatter.hh:184
 (inlined by) seastar::json::json_return_type::json_return_type<std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > >(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > const&) at /usr/src/debug/scylla-3.0.0/seastar/json/json_elements.hh:292
 (inlined by) std::_Head_base<0ul, seastar::json::json_return_type, false>::_Head_base<std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&>(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&) at /opt/scylladb/include/c++/7/tuple:133
 (inlined by) std::_Tuple_impl<0ul, seastar::json::json_return_type>::_Tuple_impl<std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&>(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&) at /opt/scylladb/include/c++/7/tuple:366
 (inlined by) void seastar::future_state<seastar::json::json_return_type>::set<std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&>(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&) at /opt/scylladb/include/c++/7/tuple:647
 (inlined by) seastar::future<seastar::json::json_return_type>::future<std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&>(seastar::ready_future_marker, std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&) at /usr/src/debug/scylla-3.0.0/seastar/core/future.hh:739
 (inlined by) seastar::future<seastar::json::json_return_type> seastar::make_ready_future<seastar::json::json_return_type, std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&>(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&) at /usr/src/debug/scylla-3.0.0/seastar/core/future.hh:1277
 (inlined by) operator() at /usr/src/debug/scylla-3.0.0/api/compaction_manager.cc:127
_ZN7seastar12continuationIZNS_6futureIJSt6vectorIN2db15system_keyspace24compaction_history_entryESaIS5_EEEE4thenIZZN3api22set_compaction_managerERNSA_12http_contextERNS_5httpd6routesEENKUlSt10unique_ptrINSD_7requestESt14default_deleteISH_EEE6_clESK_EUlS7_E_NS1_IJNS_4json16json_return_typeEEEEEET0_OT_EUlSS_E_JS7_EE15run_and_disposeEv at /usr/src/debug/scylla-3.0.0/seastar/core/apply.hh:35
 (inlined by) apply<api::set_compaction_manager(api::http_context&, seastar::httpd::routes&)::<lambda(std::unique_ptr<seastar::httpd::request>)>::<lambda(std::vector<db::system_keyspace::compaction_history_entry>)>, std::vector<db::system_keyspace::compaction_history_entry, std::allocator<db::system_keyspace::compaction_history_entry> > > at /usr/src/debug/scylla-3.0.0/seastar/core/apply.hh:43
 (inlined by) apply<api::set_compaction_manager(api::http_context&, seastar::httpd::routes&)::<lambda(std::unique_ptr<seastar::httpd::request>)>::<lambda(std::vector<db::system_keyspace::compaction_history_entry>)>, std::vector<db::system_keyspace::compaction_history_entry, std::allocator<db::system_keyspace::compaction_history_entry> > > at /usr/src/debug/scylla-3.0.0/seastar/core/future.hh:1389
 (inlined by) operator()<seastar::future_state<std::vector<db::system_keyspace::compaction_history_entry, std::allocator<db::system_keyspace::compaction_history_entry> > > > at /usr/src/debug/scylla-3.0.0/seastar/core/future.hh:963
 (inlined by) run_and_dispose at /usr/src/debug/scylla-3.0.0/seastar/core/future.hh:414
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/src/debug/scylla-3.0.0/seastar/core/reactor.cc:2695
seastar::reactor::run_some_tasks() at /usr/src/debug/scylla-3.0.0/seastar/core/reactor.cc:3118
seastar::reactor::run_some_tasks() at /usr/src/debug/scylla-3.0.0/seastar/util/log.hh:313
 (inlined by) seastar::reactor::run() at /usr/src/debug/scylla-3.0.0/seastar/core/reactor.cc:3265
seastar::smp::configure(boost::program_options::variables_map)::{lambda()#3}::operator()() const at /usr/src/debug/scylla-3.0.0/seastar/core/reactor.cc:4334
std::function<void ()>::operator()() const at /opt/scylladb/include/c++/7/bits/std_function.h:706
 (inlined by) seastar::posix_thread::start_routine(void*) at /usr/src/debug/scylla-3.0.0/seastar/core/posix.cc:52
?? ??:0
?? ??:0
linux-wizard picture linux-wizard on 28 Jan 2019

There were also many reactor stalled messages

Jan 28 17:03:29 as-dfcas-3b12 scylla: Reactor stalled for 8000 ms on shard 4.
Jan 28 17:03:29 as-dfcas-3b12 scylla: Backtrace:
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x00000000006a8212
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x00000000005bf93c
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x00000000005bfdf6
Jan 28 17:03:29 as-dfcas-3b12 scylla: /lib64/libpthread.so.0+0x000000000000f5df
Jan 28 17:03:29 as-dfcas-3b12 scylla: /opt/scylladb/lib64/libstdc++.so.6+0x0000000000124dc1
Jan 28 17:03:29 as-dfcas-3b12 scylla: /opt/scylladb/lib64/libstdc++.so.6+0x0000000000114933
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x0000000000b186d0
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x0000000000b1ed56
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x0000000004181a2a
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x00000000041843a9
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x00000000005a0cee
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x00000000005a0eb1
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x000000000066f547
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x0000000000673cba
Jan 28 17:03:29 as-dfcas-3b12 scylla: 0x0000000000756d4d
Jan 28 17:03:29 as-dfcas-3b12 scylla: /lib64/libpthread.so.0+0x0000000000007e24
Jan 28 17:03:29 as-dfcas-3b12 scylla: /lib64/libc.so.6+0x00000000000f834c

[root@as-dfcas-3b06 ~]# addr2line -fCipe  /usr/lib/debug/usr/bin/scylla.debug 0x00000000006a8212 0x00000000005bf93c 0x00000000005bfdf6 /lib64/libpthread.so.0+0x000000000000f5df /opt/scylladb/lib64/libstdc++.so.6+0x0000000000124dc1 /opt/scylladb/lib64/libstdc++.so.6+0x0000000000114933 0x0000000000b186d0 0x0000000000b1ed56 0x0000000004181a2a 0x00000000041843a9 0x00000000005a0cee 0x00000000005a0eb1 0x000000000066f547 0x0000000000673cba 0x0000000000756d4d /lib64/libpthread.so.0+0x0000000000007e24 /lib64/libc.so.6+0x00000000000f834c
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/src/debug/scylla-3.0.0/seastar/util/backtrace.hh:56
seastar::backtrace_buffer::append_backtrace() at /usr/src/debug/scylla-3.0.0/seastar/core/reactor.cc:410
 (inlined by) print_with_backtrace at /usr/src/debug/scylla-3.0.0/seastar/core/reactor.cc:431
seastar::reactor::block_notifier(int) at /usr/src/debug/scylla-3.0.0/seastar/core/reactor.cc:749
?? ??:0
?? ??:0
?? ??:0
std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /opt/scylladb/include/c++/7/bits/basic_string.h:6277
 (inlined by) seastar::json::json_builder::add(std::string const&, std::string const&) at /usr/src/debug/scylla-3.0.0/seastar/json/json_elements.cc:59
 (inlined by) seastar::json::json_builder::add(seastar::json::json_base_element*) at /usr/src/debug/scylla-3.0.0/seastar/json/json_elements.cc:70
 (inlined by) seastar::json::json_base::to_json() const at /usr/src/debug/scylla-3.0.0/seastar/json/json_elements.cc:158
seastar::json::formatter::to_json(seastar::json::jsonable const&) at /usr/src/debug/scylla-3.0.0/seastar/json/formatter.cc:103
seastar::basic_sstring<char, unsigned int, 15u, true> seastar::json::formatter::to_json<seastar::httpd::compaction_manager_json::history>(seastar::json::formatter::state, seastar::httpd::compaction_manager_json::history const&) at /usr/src/debug/scylla-3.0.0/seastar/json/formatter.hh:78
 (inlined by) seastar::basic_sstring<char, unsigned int, 15u, true> seastar::json::formatter::to_json<__gnu_cxx::__normal_iterator<seastar::httpd::compaction_manager_json::history const*, std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > > >(seastar::json::formatter::state, __gnu_cxx::__normal_iterator<seastar::httpd::compaction_manager_json::history const*, std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > >, __gnu_cxx::__normal_iterator<seastar::httpd::compaction_manager_json::history const*, std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > >) at /usr/src/debug/scylla-3.0.0/seastar/json/formatter.hh:69
 (inlined by) seastar::basic_sstring<char, unsigned int, 15u, true> seastar::json::formatter::to_json<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > const&) at /usr/src/debug/scylla-3.0.0/seastar/json/formatter.hh:184
 (inlined by) seastar::json::json_return_type::json_return_type<std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > >(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> > const&) at /usr/src/debug/scylla-3.0.0/seastar/json/json_elements.hh:292
 (inlined by) std::_Head_base<0ul, seastar::json::json_return_type, false>::_Head_base<std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&>(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&) at /opt/scylladb/include/c++/7/tuple:133
 (inlined by) std::_Tuple_impl<0ul, seastar::json::json_return_type>::_Tuple_impl<std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&>(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&) at /opt/scylladb/include/c++/7/tuple:366
 (inlined by) void seastar::future_state<seastar::json::json_return_type>::set<std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&>(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&) at /opt/scylladb/include/c++/7/tuple:647
 (inlined by) seastar::future<seastar::json::json_return_type>::future<std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&>(seastar::ready_future_marker, std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&) at /usr/src/debug/scylla-3.0.0/seastar/core/future.hh:739
 (inlined by) seastar::future<seastar::json::json_return_type> seastar::make_ready_future<seastar::json::json_return_type, std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&>(std::vector<seastar::httpd::compaction_manager_json::history, std::allocator<seastar::httpd::compaction_manager_json::history> >&) at /usr/src/debug/scylla-3.0.0/seastar/core/future.hh:1277
 (inlined by) operator() at /usr/src/debug/scylla-3.0.0/api/compaction_manager.cc:127
_ZN7seastar12continuationIZNS_6futureIJSt6vectorIN2db15system_keyspace24compaction_history_entryESaIS5_EEEE4thenIZZN3api22set_compaction_managerERNSA_12http_contextERNS_5httpd6routesEENKUlSt10unique_ptrINSD_7requestESt14default_deleteISH_EEE6_clESK_EUlS7_E_NS1_IJNS_4json16json_return_typeEEEEEET0_OT_EUlSS_E_JS7_EE15run_and_disposeEv at /usr/src/debug/scylla-3.0.0/seastar/core/apply.hh:35
 (inlined by) apply<api::set_compaction_manager(api::http_context&, seastar::httpd::routes&)::<lambda(std::unique_ptr<seastar::httpd::request>)>::<lambda(std::vector<db::system_keyspace::compaction_history_entry>)>, std::vector<db::system_keyspace::compaction_history_entry, std::allocator<db::system_keyspace::compaction_history_entry> > > at /usr/src/debug/scylla-3.0.0/seastar/core/apply.hh:43
 (inlined by) apply<api::set_compaction_manager(api::http_context&, seastar::httpd::routes&)::<lambda(std::unique_ptr<seastar::httpd::request>)>::<lambda(std::vector<db::system_keyspace::compaction_history_entry>)>, std::vector<db::system_keyspace::compaction_history_entry, std::allocator<db::system_keyspace::compaction_history_entry> > > at /usr/src/debug/scylla-3.0.0/seastar/core/future.hh:1389
 (inlined by) operator()<seastar::future_state<std::vector<db::system_keyspace::compaction_history_entry, std::allocator<db::system_keyspace::compaction_history_entry> > > > at /usr/src/debug/scylla-3.0.0/seastar/core/future.hh:963
 (inlined by) run_and_dispose at /usr/src/debug/scylla-3.0.0/seastar/core/future.hh:414
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /usr/src/debug/scylla-3.0.0/seastar/core/reactor.cc:2695
seastar::reactor::run_some_tasks() at /usr/src/debug/scylla-3.0.0/seastar/core/reactor.cc:3118
seastar::reactor::run_some_tasks() at /usr/src/debug/scylla-3.0.0/seastar/util/log.hh:313
 (inlined by) seastar::reactor::run() at /usr/src/debug/scylla-3.0.0/seastar/core/reactor.cc:3265
seastar::smp::configure(boost::program_options::variables_map)::{lambda()#3}::operator()() const at /usr/src/debug/scylla-3.0.0/seastar/core/reactor.cc:4334
std::function<void ()>::operator()() const at /opt/scylladb/include/c++/7/bits/std_function.h:706
 (inlined by) seastar::posix_thread::start_routine(void*) at /usr/src/debug/scylla-3.0.0/seastar/core/posix.cc:52
?? ??:0
?? ??:0
linux-wizard picture linux-wizard on 28 Jan 2019

@linux-wizard

the backtrace shows api request related to compaction history

are you running nodetool compactionhistory ?

can you please run on the node that had the issue the following:

select count(*) from system.compaction_history ;

would it be possible for you to share the system.compaction_history files - it would help us understand why this is happening.

slivne picture slivne on 28 Jan 2019

@slivne it's the REST API call for compaction history; probably we aren't streaming them as we should.

@amnonh please check that we use paging to read compaction history and that we stream the result.

avikivity picture avikivity on 28 Jan 2019

wow, half a gigabyte!

avikivity picture avikivity on 28 Jan 2019

I'm not running nodetool compactionhistory, but i have scylla-jmx running.

[cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> select count(*) from system.compaction_history ;
OperationTimedOut: errors={'127.0.0.1': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=127.0.0.1
cqlsh> select count(*) from system.compaction_history LIMIT 100;
OperationTimedOut: errors={'127.0.0.1': 'Client request timeout. See Session.execute[_async](timeout)'}, last_host=127.0.0.1
linux-wizard picture linux-wizard on 28 Jan 2019 
[root@as-dfcas-3b12 ~]# du -sh /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca
338M    /var/lib/scylla/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca
linux-wizard picture linux-wizard on 28 Jan 2019

I just checked and the code reads the entire table into memory.

avikivity picture avikivity on 28 Jan 2019
😕2

part of the issue is our schema for compaction_history

CREATE TABLE system.compaction_history (
    id uuid PRIMARY KEY,
    bytes_in bigint,
    bytes_out bigint,
    columnfamily_name text,
    compacted_at timestamp,
    keyspace_name text,
    rows_merged map<int, bigint>
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = 'week-long compaction history'
    AND compaction = {'class': 'SizeTieredCompactionStrategy'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 604800
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

there is no reason for gc_grace_seconds to be set for 10 days - this table is local to a node - we should change that to be 0.

It would probably be also beneficial to move this to TWCS.

slivne picture slivne on 28 Jan 2019 
[root@as-dfcas-3b12 ~]# cqlsh
Connected to DRFT-ASH-B at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> ALTER TABLE system.compaction_history WITH default_time_to_live = 259200 AND gc_grace_seconds = 0;
Unauthorized: Error from server: code=2100 [Unauthorized] message="system keyspace is not user-modifiable."

As i cannot change schema, i will just trash the content by removing the sstables :(

linux-wizard picture linux-wizard on 28 Jan 2019

@slivne Do we really need 1 week history? But it's true with shorter gc_grace_seconds this will help.
However I'm using LCS for 1 table with an avg of 40K write/s and max partition size for this table = 1MB. It means LCS will generate a lot of compactions, and so this table will grow big fast.
The situation may have been worst when i did a nodetool cleanup too

linux-wizard picture linux-wizard on 28 Jan 2019

@slivne @avikivity I see a patch from a year and a half ago
[PATCH scylla v5 0/3] use paging for compaction history

amnonh picture amnonh on 28 Jan 2019

@amnonh so please rebase (if needed) and resend/ping

slivne picture slivne on 28 Jan 2019

amnon's paging code is merged https://github.com/scylladb/scylla/commit/ebff739a845542993715f59fdf8ce892689a6efb

slivne picture slivne on 28 Jan 2019

I think we should limit the results, we can also add streaming to the results but I believe it's better to limit it anyhow.

Does someone actually need the entire 0.5GB history?

amnonh picture amnonh on 28 Jan 2019

@glommer does. Even if we limit it it can still be quite large, a compaction history record can be > 1kB, so a few thousand compactions -> too large.

avikivity picture avikivity on 28 Jan 2019

@linux-wizard yes, a cleanup will rewrite all sstables, and LCS is "generous" with is sstable count.

Notwithstanding this bug I recommend you migrate to STCS.

avikivity picture avikivity on 28 Jan 2019

From the backtrace, I see that it fails on the to_json in the API.

luckily, I already have a simple patch for that, that uses streaming to return the result.
I suggest that we will limit the number of results on query level as well.

amnonh picture amnonh on 28 Jan 2019

so @avikivity / @tzach how many records do you want to limit ...

slivne picture slivne on 28 Jan 2019

Let's convert to streaming first, since we don't have a good ordering for the table.

avikivity picture avikivity on 28 Jan 2019

We ended up reverting a patch that got rid of the compaction messages in the log in favor of always consulting the compaction history tables. (which reminds me I wrote patches to augment that table with info we are missing but never had the time to go back to it).

We reverted because there were issues but we still want to go that route.
So yes, it'd be nice to keep history for longer in this table.

glommer picture glommer on 28 Jan 2019

What about partitioning the table by time range? If we want to see compaction history, most of the time we are interested by the last N (could be last N entries or last N minutes).

So if we have a X minutes windows, we just show the closest time window to when we query (now-X min), and when going further in the history, we just go further.

So instead of doing a full scan, we do queries by providing a partition key as this table can be considered as a timeseries.

CREATE TABLE system.compaction_history (
    time_bucket bigint,
    id uuid,
    bytes_in bigint,
    bytes_out bigint,
    columnfamily_name text,
    compacted_at timestamp,
    keyspace_name text,
    rows_merged map<int, bigint>
) PRIMARY KEY (time_bucket, id)

or if we only looked by keyspace and to reduce a little bit bucket size

CREATE TABLE system.compaction_history (
    time_bucket bigint,
    keyspace_name text,
    id uuid,
    bytes_in bigint,
    bytes_out bigint,
    columnfamily_name text,
    compacted_at timestamp,
    rows_merged map<int, bigint>
) PRIMARY KEY (( time_bucket,keyspace_name), id)

or more advanced with a mod based on hash(keyspace_name.columnfamily_name)

CREATE TABLE system.compaction_history (
    time_bucket bigint,
    mod bigint,
    keyspace_name text,
    id uuid,
    bytes_in bigint,
    bytes_out bigint,
    columnfamily_name text,
    compacted_at timestamp,
    rows_merged map<int, bigint>
) PRIMARY KEY (( time_bucket, mod), keyspace_name, columnfamily_name, id)

This should allow to use pretty big time window bucket (30min or even 1h) even with LCS, and also be able to easily show compactions history per keyspace or per table.

WDYT? We may even add _shard_id_ In fact i think we should and this could replace the mod: we gain visibility of compaction history at the shard level, but also at the table level. And paginating through the history becomes pretty easy

CREATE TABLE system.compaction_history (
    time_bucket bigint,
    shard_id bigint,
    keyspace_name text,
    id uuid,
    bytes_in bigint,
    bytes_out bigint,
    columnfamily_name text,
    compacted_at timestamp,
    rows_merged map<int, bigint>
) PRIMARY KEY (( time_bucket, shard_id), keyspace_name, columnfamily_name, id)
  • Show last 30 minutes compactions for 29/01/2019 2019 03:30 for shard 1 SELECT * FROM system.compaction_history WHERE time_bucket=1548732600 AND shard_id = 1
  • Show last 30 minutes compactions for 29/01/2019 2019 03:30 for shard 1 and table mytable SELECT * FROM system.compaction_history WHERE time_bucket=1548732600 AND shard_id = 1 AND keyspace_name = 'myks' AND columnfamily_name = 'mytable'

@avikivity This should greatly improve the ordering on system.compaction_history, shouldn't it?

@slivne Switching to TCWS will become very easy in this case as we know the time window used and thus dropping tombstones will become very easy

linux-wizard picture linux-wizard on 29 Jan 2019

@amnonh is working on fixing the allocation

We will work on a better structure for compaction_history as well.

slivne picture slivne on 3 Feb 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

hellowaywewe picture hellowaywewe  路  3Comments
amoskong picture amoskong  路  6Comments
amoskong picture amoskong  路  5Comments
duarten picture duarten  路  5Comments
gnumoreno picture gnumoreno  路  5Comments