Clickhouse: Third segfault on INSERT into source table for several MATERIALIZED VIEWs.

[banburybill]

We are seeing segfaults on INSERT in ClickHouse installed on Ubuntu Xenial from stable deb packages from 19.11.2.7 onwards. From the backtraces I believe several are related, so am reporting these as a single issue. This was discovered while updating from 18.5.1 to 19.11.3.11. So far in our testing it appears 19.10.1.5 does not have the problem.

The segfaults are frequent, but cannot be reliably triggered by a particular insert. At this point we have not managed to isolate a simple method to reproduce, but hope the backtraces may give sufficient clues.

The inserts are to a simple table, but that table is in turn used as the source for a collection of MATERIALIZED VIEWs, using either SummingMergeTree or AggregatingMergeTree.

This may have the same underlying cause as #6244, but I'm not sure, so am filing as a separate issue.

ClickHouse 19.11.3.11:

2019.07.30 12:22:10.621308 [ 63 ] {} <Error> BaseDaemon: ########################################
2019.07.30 12:22:10.621380 [ 63 ] {} <Error> BaseDaemon: (version 19.11.3.11 (official build)) (from thread 49) Received signal Segmentation fault (11).
2019.07.30 12:22:10.621395 [ 63 ] {} <Error> BaseDaemon: Address: NULL pointer. Access: read. Address not mapped to object.
2019.07.30 12:22:10.656770 [ 63 ] {} <Error> BaseDaemon: 0. /usr/bin/clickhouse-server(StackTrace::StackTrace(ucontext_t const&)+0x22) [0x781c2a2]
1. /usr/bin/clickhouse-server() [0x3b139ce]
2. /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f28a403c390]
3. /usr/bin/clickhouse-server(std::_Function_handler<bool (std::shared_ptr<DB::IExternalLoadable const> const&), std::_Bind<bool (DB::IExternalLoadable::*(std::_Placeholder<1>))() const> >::_M_invoke(std::_Any_data const&, std::shared_ptr<DB::IExternalLoadable const> const&)+0x11) [0x6755481]
4. /usr/bin/clickhouse-server(DB::ExternalLoader::LoadingDispatcher::reloadOutdated()+0x23a) [0x675e71a]
5. /usr/bin/clickhouse-server(DB::ExternalLoader::PeriodicUpdater::doPeriodicUpdates()+0xc3) [0x675ec63]
6. /usr/bin/clickhouse-server(ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ExternalLoader::PeriodicUpdater::*)(), DB::ExternalLoader::PeriodicUpdater*>(void (DB::ExternalLoader::PeriodicUpdater::*&&)(), DB::ExternalLoader::PeriodicUpdater*&&)::{lambda()#1}::operator()() const+0x49) [0x6755c69]
7. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)+0x1af) [0x71c778f]
8. /usr/bin/clickhouse-server() [0xb2ac5bf]
9. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f28a40326ba]
10. /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f28a385b41d]

[filimonov]

Is it possible to share the schema or to install on problematic server special clickhouse debug build (with address sanitizer, will work much slower) and try to reproduce the problem in that debug version?

[banburybill]

The schema is part of a project the sponsors intends to open source in the future, but unfortunately we cannot currently share. I can, however, certainly install a debug build locally and reproduce the problem. Do I need to build this debug build myself, or is it available somewhere?

[filimonov]

Please use build from our CI (you can find links to builds next to each commit in github).
For example - current master:
https://clickhouse-builds.s3.yandex.net/0/eeeab85bbe01db807f97a7a673e561f6b308cdf5/report.html

Pick debs from clang-7 / relwithdebuginfo / address.
Look into stderr or run server in terminal while executing tests. ClickHouse will work much slower (it's normal).

[banburybill]

I tried the current master build as you linked.

The build originally worked, but failed when I started the incoming data feed. I then realised I needed llvm-symbolizer installed, so after finding the LLVM APT repo installed it and restarted. clickhouse-server fails immediately with the following.

Aug  1 15:19:47 wombat1 systemd[1]: Started ClickHouse Server (analytic DBMS for big data).
Aug  1 15:19:47 wombat1 clickhouse-server[3210]: Include not found: clickhouse_remote_servers
Aug  1 15:19:47 wombat1 clickhouse-server[3210]: Include not found: clickhouse_compression
Aug  1 15:19:47 wombat1 clickhouse-server[3210]: Cannot set max size of core file to 1073741824
Aug  1 15:19:47 wombat1 clickhouse-server[3210]: Logging notice to /var/log/clickhouse-server/clickhouse-server.log
Aug  1 15:19:47 wombat1 clickhouse-server[3210]: Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Aug  1 15:19:47 wombat1 clickhouse-server[3210]: Include not found: networks
Aug  1 15:19:47 wombat1 clickhouse-server[3210]: Include not found: networks
Aug  1 15:19:48 wombat1 clickhouse-server[3210]: =================================================================
Aug  1 15:19:48 wombat1 clickhouse-server[3210]: ==3210==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61b0000a0888 at pc 0x00001060ea6b bp 0x7f3c8150aa80 sp 0x7f3c8150aa78
Aug  1 15:19:48 wombat1 clickhouse-server[3210]: WRITE of size 8 at 0x61b0000a0888 thread T56 (ParalInputsProc)
Aug  1 15:19:50 wombat1 clickhouse-server[3210]: Include not found: clickhouse_remote_servers
Aug  1 15:19:50 wombat1 clickhouse-server[3210]: Include not found: clickhouse_compression
Aug  1 15:19:51 wombat1 clickhouse-server[3210]:     #0 0x1060ea6a in ZeroTraits::set(StringRef&) /build/obj-x86_64-linux-gnu/../libs/libcommon/include/common/StringRef.h:292:45
Aug  1 15:19:51 wombat1 clickhouse-server[3210]:     #1 0x1060ea6a in HashMapCell<StringRef, DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::Counter*, StringRefHash, HashTableNoState>::setZero() /build/obj-x86_64-linux-gnu/../dbms/src/Common/HashTable/HashMap.h:77
Aug  1 15:19:51 wombat1 clickhouse-server[3210]:     #2 0x1060ea6a in DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::destroyLastElement() /build/obj-x86_64-linux-gnu/../dbms/src/Common/SpaceSaving.h:350
Aug  1 15:19:51 wombat1 clickhouse-server[3210]:     #3 0x1060c0ec in DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::insert(StringRef const&, unsigned long, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/Common/SpaceSaving.h:176:13
Aug  1 15:19:51 wombat1 clickhouse-server[3210]:     #4 0x106106a9 in DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::merge(DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> > const&) /build/obj-x86_64-linux-gnu/../dbms/src/Common/SpaceSaving.h
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #5 0x12143a9c in void DB::Aggregator::executeImplCase<false, DB::AggregationMethodOneNumber<unsigned short, FixedHashMap<unsigned short, char*, AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul> > > >(DB::AggregationMethodOneNumber<unsigned short, FixedHashMap<unsigned short, char*, AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul> > >&, DB::AggregationMethodOneNumber<unsigned short, FixedHashMap<unsigned short, char*, AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul> > >::State&, DB::Arena*, unsigned long, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*> >&, DB::Aggregator::AggregateFunctionInstruction*, std::__1::vector<StringRef, std::__1::allocator<StringRef> >&, char*) const /build/obj-x86_64-linux-gnu/../dbms/src/Interpreters/Aggregator.cpp:653:13
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #6 0x120d0841 in void DB::Aggregator::executeImpl<DB::AggregationMethodOneNumber<unsigned short, FixedHashMap<unsigned short, char*, AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul> > > >(DB::AggregationMethodOneNumber<unsigned short, FixedHashMap<unsigned short, char*, AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul> > >&, DB::Arena*, unsigned long, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*> >&, DB::Aggregator::AggregateFunctionInstruction*, std::__1::vector<StringRef, std::__1::allocator<StringRef> >&, bool, char*) const /build/obj-x86_64-linux-gnu/../dbms/src/Interpreters/Aggregator.cpp:593:9
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #7 0x1209b77d in DB::Aggregator::executeOnBlock(DB::Block const&, DB::AggregatedDataVariants&, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*> >&, std::__1::vector<std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*> >, std::__1::allocator<std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*> > > >&, std::__1::vector<StringRef, std::__1::allocator<StringRef> >&, bool&) /build/obj-x86_64-linux-gnu/../dbms/src/Interpreters/Aggregator.cpp:832:13
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #8 0x123b97c3 in DB::ParallelAggregatingBlockInputStream::Handler::onBlock(DB::Block&, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/ParallelAggregatingBlockInputStream.cpp:112:23
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #9 0x123b97c3 in DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::publishPayload(DB::Block&, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/ParallelInputsProcessor.h:173
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #10 0x123b97c3 in DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop(unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/ParallelInputsProcessor.h:295
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #11 0x123b63ae in DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/ParallelInputsProcessor.h:208:13
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #12 0x123b8ccc in decltype(*(std::__1::forward<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>* const&>(fp0)).*fp(std::__1::forward<std::__1::shared_ptr<DB::ThreadGroupStatus> const&>(fp1), std::__1::forward<unsigned long const&>(fp1))) std::__1::__invoke_constexpr<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>* const&, std::__1::shared_ptr<DB::ThreadGroupStatus> const&, unsigned long const&, void>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>* const&, std::__1::shared_ptr<DB::ThreadGroupStatus> const&, unsigned long const&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:4358:1
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #13 0x123b8ccc in decltype(auto) std::__1::__apply_tuple_impl<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), std::__1::tuple<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long> const&, 0ul, 1ul, 2ul>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), std::__1::tuple<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long> const&, std::__1::__tuple_indices<0ul, 1ul, 2ul>) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1358
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #14 0x123b8ccc in decltype(auto) std::__1::apply<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), std::__1::tuple<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long> const&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), std::__1::tuple<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long> const&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1367
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #15 0x123b8ccc in ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::__1::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::'lambda'()::operator()() const /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.h:147
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #16 0x744f31e in std::__1::__function::__value_func<void ()>::operator()() const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1799:16
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #17 0x744f31e in std::__1::function<void ()>::operator()() const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2347
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #18 0x744f31e in ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:177
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #19 0x7458797 in void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:73:73
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #20 0x7458797 in decltype(std::__1::forward<void>(fp)()) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:4410
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #21 0x7458797 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>&, std::__1::__tuple_indices<>) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:341
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #22 0x7458797 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:351
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #23 0x7f3d8e9a56b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #24 0x7f3d8e3d241c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]: 0x61b0000a0888 is located 8 bytes to the right of 1536-byte region [0x61b0000a0280,0x61b0000a0880)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]: allocated by thread T56 (ParalInputsProc) here:
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #0 0x7369292 in realloc (/usr/bin/clickhouse+0x7369292)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #1 0x7856fbd in AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>::realloc(void*, unsigned long, unsigned long, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/Common/Allocator.h:142:30
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #2 0x1060f0b5 in AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul>::realloc(void*, unsigned long, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/Common/Allocator.h:311:26
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #3 0x1060f0b5 in HashTable<StringRef, HashMapCell<StringRef, DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::Counter*, StringRefHash, HashTableNoState>, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::resize(unsigned long, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/Common/HashTable/HashTable.h:360
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #4 0x1060ddf1 in HashTable<StringRef, HashMapCell<StringRef, DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::Counter*, StringRefHash, HashTableNoState>, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::emplaceNonZeroImpl(unsigned long, StringRef, HashTable<StringRef, HashMapCell<StringRef, DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::Counter*, StringRefHash, HashTableNoState>, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::iterator&, bool&, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/Common/HashTable/HashTable.h:679:17
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #5 0x1060ddf1 in HashTable<StringRef, HashMapCell<StringRef, DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::Counter*, StringRefHash, HashTableNoState>, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::emplaceNonZero(StringRef, HashTable<StringRef, HashMapCell<StringRef, DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::Counter*, StringRefHash, HashTableNoState>, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::iterator&, bool&, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/Common/HashTable/HashTable.h:700
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #6 0x1060ddf1 in HashTable<StringRef, HashMapCell<StringRef, DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::Counter*, StringRefHash, HashTableNoState>, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::emplace(StringRef, HashTable<StringRef, HashMapCell<StringRef, DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::Counter*, StringRefHash, HashTableNoState>, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::iterator&, bool&) /build/obj-x86_64-linux-gnu/../dbms/src/Common/HashTable/HashTable.h:755
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #7 0x1060ddf1 in HashMapTable<StringRef, HashMapCell<StringRef, DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::Counter*, StringRefHash, HashTableNoState>, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::operator[](StringRef) /build/obj-x86_64-linux-gnu/../dbms/src/Common/HashTable/HashMap.h:156
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #8 0x1060ddf1 in DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::push(DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::Counter*) /build/obj-x86_64-linux-gnu/../dbms/src/Common/SpaceSaving.h:315
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #9 0x106106a9 in DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::merge(DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> > const&) /build/obj-x86_64-linux-gnu/../dbms/src/Common/SpaceSaving.h
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #10 0x12143a9c in void DB::Aggregator::executeImplCase<false, DB::AggregationMethodOneNumber<unsigned short, FixedHashMap<unsigned short, char*, AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul> > > >(DB::AggregationMethodOneNumber<unsigned short, FixedHashMap<unsigned short, char*, AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul> > >&, DB::AggregationMethodOneNumber<unsigned short, FixedHashMap<unsigned short, char*, AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul> > >::State&, DB::Arena*, unsigned long, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*> >&, DB::Aggregator::AggregateFunctionInstruction*, std::__1::vector<StringRef, std::__1::allocator<StringRef> >&, char*) const /build/obj-x86_64-linux-gnu/../dbms/src/Interpreters/Aggregator.cpp:653:13
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #11 0x120d0841 in void DB::Aggregator::executeImpl<DB::AggregationMethodOneNumber<unsigned short, FixedHashMap<unsigned short, char*, AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul> > > >(DB::AggregationMethodOneNumber<unsigned short, FixedHashMap<unsigned short, char*, AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul> > >&, DB::Arena*, unsigned long, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*> >&, DB::Aggregator::AggregateFunctionInstruction*, std::__1::vector<StringRef, std::__1::allocator<StringRef> >&, bool, char*) const /build/obj-x86_64-linux-gnu/../dbms/src/Interpreters/Aggregator.cpp:593:9
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #12 0x1209b77d in DB::Aggregator::executeOnBlock(DB::Block const&, DB::AggregatedDataVariants&, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*> >&, std::__1::vector<std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*> >, std::__1::allocator<std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*> > > >&, std::__1::vector<StringRef, std::__1::allocator<StringRef> >&, bool&) /build/obj-x86_64-linux-gnu/../dbms/src/Interpreters/Aggregator.cpp:832:13
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #13 0x123b97c3 in DB::ParallelAggregatingBlockInputStream::Handler::onBlock(DB::Block&, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/ParallelAggregatingBlockInputStream.cpp:112:23
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #14 0x123b97c3 in DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::publishPayload(DB::Block&, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/ParallelInputsProcessor.h:173
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #15 0x123b97c3 in DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::loop(unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/ParallelInputsProcessor.h:295
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #16 0x123b63ae in DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::thread(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/ParallelInputsProcessor.h:208:13
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #17 0x123b8ccc in decltype(*(std::__1::forward<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>* const&>(fp0)).*fp(std::__1::forward<std::__1::shared_ptr<DB::ThreadGroupStatus> const&>(fp1), std::__1::forward<unsigned long const&>(fp1))) std::__1::__invoke_constexpr<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>* const&, std::__1::shared_ptr<DB::ThreadGroupStatus> const&, unsigned long const&, void>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>* const&, std::__1::shared_ptr<DB::ThreadGroupStatus> const&, unsigned long const&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:4358:1
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #18 0x123b8ccc in decltype(auto) std::__1::__apply_tuple_impl<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), std::__1::tuple<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long> const&, 0ul, 1ul, 2ul>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), std::__1::tuple<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long> const&, std::__1::__tuple_indices<0ul, 1ul, 2ul>) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1358
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #19 0x123b8ccc in decltype(auto) std::__1::apply<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), std::__1::tuple<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long> const&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::* const&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), std::__1::tuple<DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long> const&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1367
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #20 0x123b8ccc in ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::__1::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::'lambda'()::operator()() const /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.h:147
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #21 0x744f31e in std::__1::__function::__value_func<void ()>::operator()() const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1799:16
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #22 0x744f31e in std::__1::function<void ()>::operator()() const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2347
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #23 0x744f31e in ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:177
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #24 0x7458797 in void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:73:73
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #25 0x7458797 in decltype(std::__1::forward<void>(fp)()) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:4410
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #26 0x7458797 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>&, std::__1::__tuple_indices<>) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:341
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #27 0x7458797 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:351
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #28 0x7f3d8e9a56b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]: Thread T56 (ParalInputsProc) created by T55 (AsyncBlockInput) here:
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #0 0x735178d in __interceptor_pthread_create (/usr/bin/clickhouse+0x735178d)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #1 0x745697e in std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:327:10
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #2 0x745697e in std::__1::thread::thread<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'(), void>(void&&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:367
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #3 0x744ba4c in void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:73:35
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #4 0x744d753 in ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, int, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:100:5
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #5 0x123b7afd in ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::__1::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.h:140:38
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #6 0x123ae0d0 in void std::__1::allocator<ThreadFromGlobalPool>::construct<ThreadFromGlobalPool, void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(ThreadFromGlobalPool*, void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::__1::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:1825:31
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #7 0x123ae0d0 in void std::__1::allocator_traits<std::__1::allocator<ThreadFromGlobalPool> >::__construct<ThreadFromGlobalPool, void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(std::__1::integral_constant<bool, true>, std::__1::allocator<ThreadFromGlobalPool>&, ThreadFromGlobalPool*, void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::__1::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:1717
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #8 0x123ae0d0 in void std::__1::allocator_traits<std::__1::allocator<ThreadFromGlobalPool> >::construct<ThreadFromGlobalPool, void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(std::__1::allocator<ThreadFromGlobalPool>&, ThreadFromGlobalPool*, void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::__1::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:1560
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #9 0x123ae0d0 in ThreadFromGlobalPool& std::__1::vector<ThreadFromGlobalPool, std::__1::allocator<ThreadFromGlobalPool> >::emplace_back<void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::*&&)(std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>*&&, std::__1::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/vector:1687
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #10 0x123ae0d0 in DB::ParallelInputsProcessor<DB::ParallelAggregatingBlockInputStream::Handler>::process() /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/ParallelInputsProcessor.h:102
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #11 0x123a83b4 in DB::ParallelAggregatingBlockInputStream::execute() /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/ParallelAggregatingBlockInputStream.cpp:174:15
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #12 0x123a4c23 in DB::ParallelAggregatingBlockInputStream::readImpl() /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/ParallelAggregatingBlockInputStream.cpp:58:9
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #13 0x119ee664 in DB::IBlockInputStream::read() /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/IBlockInputStream.cpp:56:15
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #14 0x119ea0c4 in DB::AsynchronousBlockInputStream::calculate() /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/AsynchronousBlockInputStream.cpp:74:34
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #15 0x119eb086 in DB::AsynchronousBlockInputStream::next()::$_0::operator()() const /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/AsynchronousBlockInputStream.cpp:59:9
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #16 0x119eb086 in decltype(std::__1::forward<DB::AsynchronousBlockInputStream::next()::$_0&>(fp)()) std::__1::__invoke<DB::AsynchronousBlockInputStream::next()::$_0&>(DB::AsynchronousBlockInputStream::next()::$_0&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:4410
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #17 0x119eb086 in void std::__1::__invoke_void_return_wrapper<void>::__call<DB::AsynchronousBlockInputStream::next()::$_0&>(DB::AsynchronousBlockInputStream::next()::$_0&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #18 0x119eb086 in std::__1::__function::__alloc_func<DB::AsynchronousBlockInputStream::next()::$_0, std::__1::allocator<DB::AsynchronousBlockInputStream::next()::$_0>, void ()>::operator()() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1527
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #19 0x119eb086 in std::__1::__function::__func<DB::AsynchronousBlockInputStream::next()::$_0, std::__1::allocator<DB::AsynchronousBlockInputStream::next()::$_0>, void ()>::operator()() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1651
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #20 0x74554eb in std::__1::__function::__value_func<void ()>::operator()() const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1799:16
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #21 0x74554eb in std::__1::function<void ()>::operator()() const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2347
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #22 0x74554eb in ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:177
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #23 0x745cfc6 in void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:73:73
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #24 0x745cfc6 in decltype(std::__1::forward<void>(fp)()) std::__1::__invoke_constexpr<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'() const&>(void&&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:4416
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #25 0x745cfc6 in decltype(auto) std::__1::__apply_tuple_impl<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'() const&, std::__1::tuple<> const&>(void&&, std::__1::tuple<> const&, std::__1::__tuple_indices<>) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1358
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #26 0x745cfc6 in decltype(auto) std::__1::apply<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'() const&, std::__1::tuple<> const&>(void&&, std::__1::tuple<> const&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/tuple:1367
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #27 0x745cfc6 in ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&)::'lambda'()::operator()() const /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.h:147
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #28 0x745cfc6 in decltype(std::__1::forward<void>(fp)()) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&)::'lambda'()&>(void&&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:4410
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #29 0x745cfc6 in void std::__1::__invoke_void_return_wrapper<void>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&)::'lambda'()&>(ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&)::'lambda'()&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #30 0x745cfc6 in std::__1::__function::__alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&)::'lambda'(), std::__1::allocator<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&)::'lambda'()>, void ()>::operator()() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1527
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #31 0x745cfc6 in std::__1::__function::__func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&)::'lambda'(), std::__1::allocator<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&)::'lambda'()>, void ()>::operator()() /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1651
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #32 0x744f31e in std::__1::__function::__value_func<void ()>::operator()() const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1799:16
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #33 0x744f31e in std::__1::function<void ()>::operator()() const /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2347
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #34 0x744f31e in ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:177
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #35 0x7458797 in void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()::operator()() const /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:73:73
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #36 0x7458797 in decltype(std::__1::forward<void>(fp)()) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:4410
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #37 0x7458797 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>&, std::__1::__tuple_indices<>) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:341
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #38 0x7458797 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:351
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #39 0x7f3d8e9a56b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]: Thread T55 (AsyncBlockInput) created by T45 (TCPHandler) here:
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #0 0x735178d in __interceptor_pthread_create (/usr/bin/clickhouse+0x735178d)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #1 0x745697e in std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:327:10
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #2 0x745697e in std::__1::thread::thread<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'(), void>(void&&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:367
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #3 0x744ba4c in void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:73:35
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #4 0x744d753 in ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, int, unsigned long) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:100:5
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #5 0x745a4a5 in ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.h:140:38
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #6 0x7450fd5 in void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:73:35
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #7 0x745026b in ThreadPoolImpl<ThreadFromGlobalPool>::schedule(std::__1::function<void ()>, int) /build/obj-x86_64-linux-gnu/../dbms/src/Common/ThreadPool.cpp:88:5
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #8 0x119ea6f2 in DB::AsynchronousBlockInputStream::next() /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/AsynchronousBlockInputStream.cpp:39:10
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #9 0x7559095 in DB::AsynchronousBlockInputStream::readPrefix() /build/obj-x86_64-linux-gnu/../dbms/src/DataStreams/AsynchronousBlockInputStream.h:41:13
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #10 0x7559095 in DB::TCPHandler::processOrdinaryQuery() /build/obj-x86_64-linux-gnu/../dbms/programs/server/TCPHandler.cpp:423
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #11 0x7548636 in DB::TCPHandler::runImpl() /build/obj-x86_64-linux-gnu/../dbms/programs/server/TCPHandler.cpp:224:17
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #12 0x7566b3b in DB::TCPHandler::run() /build/obj-x86_64-linux-gnu/../dbms/programs/server/TCPHandler.cpp:1083:9
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #13 0x14061d1e in Poco::Net::TCPServerConnection::start() /build/obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #14 0x140629bf in Poco::Net::TCPServerDispatcher::run() /build/obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:114:20
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #15 0x152ca894 in Poco::PooledThread::run() /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:214:14
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #16 0x152c4f3e in Poco::ThreadImpl::runnableEntry(void*) /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_STD.cpp:139:27
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #17 0x152c8fcd in decltype(std::__1::forward<void* (*)(void*)>(fp)(std::__1::forward<Poco::ThreadImpl*>(fp0))) std::__1::__invoke<void* (*)(void*), Poco::ThreadImpl*>(void* (*&&)(void*), Poco::ThreadImpl*&&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:4410:1
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #18 0x152c8fcd in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*>&, std::__1::__tuple_indices<2ul>) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:341
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #19 0x152c8fcd in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void* (*)(void*), Poco::ThreadImpl*> >(void*) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:351
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #20 0x7f3d8e9a56b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]: Thread T45 (TCPHandler) created by T0 here:
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #0 0x735178d in __interceptor_pthread_create (/usr/bin/clickhouse+0x735178d)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #1 0x152c8039 in std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:327:10
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #2 0x152c8039 in std::__1::thread::thread<void* (&)(void*), Poco::ThreadImpl*, void>(void* (&)(void*), Poco::ThreadImpl*&&) /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/thread:367
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #3 0x152c4468 in Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable> >) /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_STD.cpp:58:53
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #4 0x152c6ff7 in Poco::Thread::start(Poco::Runnable&) /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:116:2
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #5 0x152cae4f in Poco::PooledThread::start(int) /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:88:10
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #6 0x152cae4f in Poco::ThreadPool::ThreadPool(int, int, int, int, Poco::ThreadPool::ThreadAffinityPolicy) /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:278
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #7 0x73b5150 in DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /build/obj-x86_64-linux-gnu/../dbms/programs/server/Server.cpp:561:26
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #8 0x1428a510 in Poco::Util::Application::run() /build/obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:335:8
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #9 0x739e30c in DB::Server::run() /build/obj-x86_64-linux-gnu/../dbms/programs/server/Server.cpp:142:25
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #10 0x73ec1eb in mainEntryClickHouseServer(int, char**) /build/obj-x86_64-linux-gnu/../dbms/programs/server/Server.cpp:908:20
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #11 0x739bca1 in main /build/obj-x86_64-linux-gnu/../dbms/programs/main.cpp:182:12
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:     #12 0x7f3d8e2eb82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]: SUMMARY: AddressSanitizer: heap-buffer-overflow /build/obj-x86_64-linux-gnu/../libs/libcommon/include/common/StringRef.h:292:45 in ZeroTraits::set(StringRef&)
Aug  1 15:19:52 wombat1 clickhouse-server[3210]: Shadow bytes around the buggy address:
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   0x0c368000c0c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   0x0c368000c0d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   0x0c368000c0e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   0x0c368000c0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   0x0c368000c100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Aug  1 15:19:52 wombat1 clickhouse-server[3210]: =>0x0c368000c110: fa[fa]fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   0x0c368000c120: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   0x0c368000c130: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   0x0c368000c140: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   0x0c368000c150: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   0x0c368000c160: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Aug  1 15:19:52 wombat1 clickhouse-server[3210]: Shadow byte legend (one shadow byte represents 8 application bytes):
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Addressable:           00
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Partially addressable: 01 02 03 04 05 06 07
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Heap left redzone:       fa
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Freed heap region:       fd
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Stack left redzone:      f1
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Stack mid redzone:       f2
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Stack right redzone:     f3
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Stack after return:      f5
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Stack use after scope:   f8
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Global redzone:          f9
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Global init order:       f6
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Poisoned by user:        f7
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Container overflow:      fc
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Array cookie:            ac
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Intra object redzone:    bb
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   ASan internal:           fe
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Left alloca redzone:     ca
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Right alloca redzone:    cb
Aug  1 15:19:52 wombat1 clickhouse-server[3210]:   Shadow gap:              cc
Aug  1 15:19:52 wombat1 clickhouse-server[3210]: ==3210==ABORTING
Aug  1 15:19:52 wombat1 systemd[1]: clickhouse-server.service: Main process exited, code=exited, status=1/FAILURE
Aug  1 15:19:52 wombat1 systemd[1]: clickhouse-server.service: Unit entered failed state.
Aug  1 15:19:52 wombat1 systemd[1]: clickhouse-server.service: Failed with result 'exit-code'.

[filimonov]

The schema is part of a project the sponsors intends to open source in the future, but unfortunately we cannot currently share.

That asan report is useful.

Schema sample would also be helpful - we don't need real columns names, something like should be enough:

CREATE TABLE t1 (c1 UInt8, c2 String, c3 ... ) Engine = ... 
CREATE MATERIALIZED VIEW ... AS SELECT c1,c2,c3 FROM t1 ...

-- remove/replace any sensitive data 

[CurtizJ]

I found that segfault is appeared in SpaceSaving class, which is used only in topK and topKWeighted aggregate functions. And I could reproduce it with query select topKWeighted(10)(toString(number), number) from numbers(10000000). So, I guess that you have column with topKWeighted function in AggregatingMergeTree and you faced with that problem.

[banburybill]

I suspect all the faults I am seeing have the same cause. I managed to reproduce #6245 with a small schema this morning - I just added a report to #6245. I don't have topKWeighted, but I do have topK() in AggregatingMergeTree.

[CurtizJ]

I think that 3 issues you reported have different causes.
Segfault from this issue at line

4. /usr/bin/clickhouse-server(DB::ExternalLoader::LoadingDispatcher::reloadOutdated()+0x23a) [0x675e71a]

has been already fixed in #6082.

Second one from #6244 at line

3. /usr/bin/clickhouse-server(DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> >::merge(DB::SpaceSaving<StringRef, StringRefHash, HashTableGrower<4ul>, AllocatorWithStackMemory<AllocatorWithHint<true, AllocatorHints::DefaultHint, 67108864ul>, 256ul> > const&)+0xe9) [0x64c75d9]

was recently fixed in #6404.

And the last from #6245 is stayed unfixed.