Clickhouse: Failed to start up with WSL environment

Created on 13 Aug 2019 · 8Comments · Source: ClickHouse/ClickHouse

Describe the bug
Clickhouse-server failed to start up with defaults

How to reproduce
clickhouse-server --config-file=/etc/clickhouse-server/config.xml

Which ClickHouse server version to use
19.13.1.11 (official build)
Which interface to use, if matters
none
Non-default settings, if any
none
CREATE TABLE statements for all tables involved
none
Sample data for all these tables, use clickhouse-obfuscator if necessary
none
Queries to run that lead to unexpected result
none

Expected behavior
Server start up is successful

Error message and/or stacktrace
2019.08.13 23:29:18.499299 [ 1 ] {} BackgroundSchedulePool: Create BackgroundSchedulePool with 16 threads
2019.08.13 23:29:18.502924 [ 8 ] {} BaseDaemon: ########################################
2019.08.13 23:29:18.503391 [ 8 ] {} BaseDaemon: (version 19.13.1.11 (official build)) (from thread 1) Received signal Aborted (6).
2019.08.13 23:29:18.504350 [ 8 ] {} BaseDaemon:
2019.08.13 23:29:18.505231 [ 8 ] {} BaseDaemon: 0. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f7a5325ee97]

Aborted (core dumped)

Additional context
Windows 10 with Windows Subsystem for Linux and Ubuntu 18.04 installed

bug

Source

hamart

👍1

Most helpful comment

note for CH devs: 19.11.2.7 works under WSL.

den-crane on 13 Aug 2019

👍2

All 8 comments

note for CH devs: 19.11.2.7 works under WSL.

den-crane on 13 Aug 2019

👍2

Could you please run clickhouse-server with gdb:

gdb --args clickhouse-server --config-file=/etc/clickhouse-server/config.xml

In gdb, type r. After stopped by signal, type t apply all bt.

alexey-milovidov on 21 Aug 2019

I also get this problem in WSL, only difference that Debian 9

(gdb) r
Starting program: /usr/bin/clickhouse-server --config-file=/etc/clickhouse-server/config.xml
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffffd310700 (LWP 14040)]
[New Thread 0x7ffffcb00700 (LWP 14041)]
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
Logging trace to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Logging trace to console
[New Thread 0x7ffffb0f0700 (LWP 14042)]
2019.08.25 08:46:34.556121 [ 1 ] {} <Information> : Starting ClickHouse 19.13.3.26 with revision 54425
2019.08.25 08:46:34.556600 [ 1 ] {} <Information> Application: starting up

Thread 1 "clickhouse-serv" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

(gdb) t apply all bt

Thread 4 (Thread 0x7ffffb0f0700 (LWP 14042)):
#0  0x00007ffffefe022d in read () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000000003c5b47a in DB::ReadBufferFromFileDescriptor::nextImpl() ()
#2  0x0000000003da448e in SignalListener::run() ()
#3  0x000000000806828c in Poco::ThreadImpl::runnableEntry(void*) ()
#4  0x000000000ba3e1a0 in ?? ()
#5  0x00007ffffefd74a4 in start_thread (arg=0x7ffffb0f0700) at pthread_create.c:456
#6  0x00007ffffe7f8d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 3 (Thread 0x7ffffcb00700 (LWP 14041)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000b9ca21c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#2  0x0000000008003c1b in Poco::Event::wait() ()
#3  0x000000000806a4a8 in Poco::PooledThread::run() ()
#4  0x000000000806828c in Poco::ThreadImpl::runnableEntry(void*) ()
#5  0x000000000ba3e1a0 in ?? ()
#6  0x00007ffffefd74a4 in start_thread (arg=0x7ffffcb00700) at pthread_create.c:456
#7  0x00007ffffe7f8d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7ffffd310700 (LWP 14040)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000b9ca21c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#2  0x0000000008003c1b in Poco::Event::wait() ()
#3  0x000000000806a4a8 in Poco::PooledThread::run() ()
#4  0x000000000806828c in Poco::ThreadImpl::runnableEntry(void*) ()
#5  0x000000000ba3e1a0 in ?? ()
#6  0x00007ffffefd74a4 in start_thread (arg=0x7ffffd310700) at pthread_create.c:456
#7  0x00007ffffe7f8d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7fffff7c4f40 (LWP 14036)):
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffffe74442a in __GI_abort () at abort.c:89
#2  0x00000000037d22d0 in ?? ()
#3  0x000000000b9b9a96 in __cxxabiv1::__terminate(void (*)()) ()
#4  0x000000000b9b9ae1 in std::terminate() ()
#5  0x000000000b9b9c69 in __cxa_rethrow ()
#6  0x0000000003c4a413 in DB::getCurrentExceptionMessage[abi:cxx11](bool, bool) ()
#7  0x0000000003da09cf in ?? ()
#8  0x000000000b9b9a96 in __cxxabiv1::__terminate(void (*)()) ()
#9  0x000000000b9b9ae1 in std::terminate() ()
#10 0x000000000b9b9c15 in __cxa_throw ()
#11 0x00000000037b9768 in ?? ()
#12 0x0000000007a04098 in Poco::Util::Application::run() ()
#13 0x0000000003c7b5ee in DB::Server::run() ()
#14 0x0000000007a1a835 in Poco::Util::ServerApplication::run(int, char**) ()
#15 0x0000000003c79ac7 in mainEntryClickHouseServer(int, char**) ()
#16 0x0000000003b926e3 in main ()

nrv9999 on 25 Aug 2019

@nrv9999 Could you please also install clickhouse-common-static-dbg package so we can see functions with internal linkage and line numbers?

#11 0x00000000037b9768 in ?? ()

alexey-milovidov on 26 Aug 2019

$ addr2line -afiCe clickhouse-19.13.1.11 0x00000000037b9768
0x00000000037b9768
DB::Server::main(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) [clone .cold]
/usr/include/c++/9/bits/basic_string.h:231 (discriminator 31)
??
/usr/include/c++/9/bits/basic_string.h:658 (discriminator 31)
DB::Server::main(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)
/build/obj-x86_64-linux-gnu/../dbms/programs/server/Server.cpp:371 (discriminator 31)

alexey-milovidov on 26 Aug 2019

With clickhouse-common-static-dbg

(gdb) r
Starting program: /usr/bin/clickhouse-server --config-file=/etc/clickhouse-server/config.xml
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffffd310700 (LWP 56)]
[New Thread 0x7ffffcb00700 (LWP 57)]
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
Logging trace to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Logging trace to console
[New Thread 0x7ffffb0f0700 (LWP 58)]
2019.08.26 07:03:12.218461 [ 1 ] {} <Information> : Starting ClickHouse 19.13.3.26 with revision 54425
2019.08.26 07:03:12.222316 [ 1 ] {} <Information> Application: starting up

Thread 1 "clickhouse-serv" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

(gdb) t apply all bt

Thread 4 (Thread 0x7ffffb0f0700 (LWP 58)):
#0  0x00007ffffefe022d in read () at ../sysdeps/unix/syscall-template.S:84
#1  0x0000000003c5b47a in read (__nbytes=<optimized out>, __buf=<optimized out>, __fd=<optimized out>)
    at /usr/include/x86_64-linux-gnu/bits/unistd.h:44
#2  DB::ReadBufferFromFileDescriptor::nextImpl() () at ../dbms/src/IO/ReadBufferFromFileDescriptor.cpp:56
#3  0x0000000003da448e in DB::ReadBuffer::next (this=0x7ffffb0eabf0) at ../dbms/src/IO/ReadBuffer.h:53
#4  DB::ReadBuffer::eof (this=0x7ffffb0eabf0) at ../dbms/src/IO/ReadBuffer.h:81
#5  SignalListener::run (this=<optimized out>) at ../libs/libdaemon/src/BaseDaemon.cpp:151
#6  0x000000000806828c in Poco::ThreadImpl::runnableEntry(void*) ()
    at ../contrib/poco/Foundation/include/Poco/SharedPtr.h:380
#7  0x000000000ba3e1a0 in execute_native_thread_routine ()
#8  0x00007ffffefd74a4 in start_thread (arg=0x7ffffb0f0700) at pthread_create.c:456
#9  0x00007ffffe7f8d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 3 (Thread 0x7ffffcb00700 (LWP 57)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000b9ca21c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#2  0x0000000008003c1b in std::condition_variable::wait<Poco::Event::wait()::<lambda()> > (__p=..., __lock=...,
    this=0x7ffffe376b18) at /usr/include/c++/9/condition_variable:98
#3  Poco::Event::wait() () at ../contrib/poco/Foundation/src/Event.cpp:79
#4  0x000000000806a4a8 in Poco::PooledThread::run() () at ../contrib/poco/Foundation/src/ThreadPool.cpp:206
#5  0x000000000806828c in Poco::ThreadImpl::runnableEntry(void*) ()
    at ../contrib/poco/Foundation/include/Poco/SharedPtr.h:380
#6  0x000000000ba3e1a0 in execute_native_thread_routine ()
#7  0x00007ffffefd74a4 in start_thread (arg=0x7ffffcb00700) at pthread_create.c:456
#8  0x00007ffffe7f8d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 2 (Thread 0x7ffffd310700 (LWP 56)):
---Type <return> to continue, or q <return> to quit---
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000000000b9ca21c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#2  0x0000000008003c1b in std::condition_variable::wait<Poco::Event::wait()::<lambda()> > (__p=..., __lock=...,
    this=0x7ffffe376898) at /usr/include/c++/9/condition_variable:98
#3  Poco::Event::wait() () at ../contrib/poco/Foundation/src/Event.cpp:79
#4  0x000000000806a4a8 in Poco::PooledThread::run() () at ../contrib/poco/Foundation/src/ThreadPool.cpp:206
#5  0x000000000806828c in Poco::ThreadImpl::runnableEntry(void*) ()
    at ../contrib/poco/Foundation/include/Poco/SharedPtr.h:380
#6  0x000000000ba3e1a0 in execute_native_thread_routine ()
#7  0x00007ffffefd74a4 in start_thread (arg=0x7ffffd310700) at pthread_create.c:456
#8  0x00007ffffe7f8d0f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

Thread 1 (Thread 0x7fffff7c4f40 (LWP 52)):
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffffe74442a in __GI_abort () at abort.c:89
#2  0x00000000037d22d0 in terminate_handler () at ../libs/libdaemon/src/BaseDaemon.cpp:232
#3  0x000000000b9b9a96 in __cxxabiv1::__terminate(void (*)()) ()
#4  0x000000000b9b9ae1 in std::terminate() ()
#5  0x000000000b9b9c69 in __cxa_rethrow ()
#6  0x0000000003c4a413 in DB::getCurrentExceptionMessage[abi:cxx11](bool, bool) ()
    at ../dbms/src/Common/Exception.cpp:77
#7  0x0000000003da09cf in terminate_handler() () at ../libs/libdaemon/src/BaseDaemon.cpp:241
#8  0x000000000b9b9a96 in __cxxabiv1::__terminate(void (*)()) ()
#9  0x000000000b9b9ae1 in std::terminate() ()
#10 0x000000000b9b9c15 in __cxa_throw ()
#11 0x00000000037b9768 in DB::Server::main (this=<optimized out>) at ../dbms/programs/server/Server.cpp:250
#12 0x0000000007a04098 in Poco::Util::Application::run() () at ../contrib/poco/Util/src/Application.cpp:335
#13 0x0000000003c7b5ee in DB::Server::run() () at ../dbms/programs/server/Server.cpp:142
#14 0x0000000007a1a835 in Poco::Util::ServerApplication::run(int, char**) ()
---Type <return> to continue, or q <return> to quit---
    at ../contrib/poco/Util/src/ServerApplication.cpp:606
#15 0x0000000003c79ac7 in mainEntryClickHouseServer(int, char**) () at ../dbms/programs/server/Server.cpp:908
#16 0x0000000003b926e3 in main () at /usr/include/c++/9/bits/stl_vector.h:915
#17 0x00007ffffe7302e1 in __libc_start_main (main=0x3b924d0 <main>, argc=2, argv=0x7ffffffee688, init=<optimized out>,
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffffffee678) at ../csu/libc-start.c:291
#18 0x0000000003c446ba in _start () at ../dbms/programs/main.cpp:182

nrv9999 on 26 Aug 2019

It is https://github.com/yandex/ClickHouse/blob/v19.13.1.11-stable/dbms/programs/server/Server.cpp#L250

The is caused by mismatched owner of data directory.
But the root cause is probably that exception handling does not work at all.
https://github.com/yandex/ClickHouse/pull/4828

alexey-milovidov on 26 Aug 2019

Hi!
Same problem on WSL Ubuntu 18 on Win10
All settings configured by default as is after setup.

gdb --args clickhouse-server --config-file=/etc/clickhouse-server/config.xml
GNU gdb (Ubuntu 8.1-0ubuntu3.1) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from clickhouse-server...(no debugging symbols found)...done.
(gdb) r
Starting program: /usr/bin/clickhouse-server --config-file=/etc/clickhouse-server/config.xml
/bin/bash: /usr/bin/clickhouse-server: Operation not permitted
/bin/bash: /usr/bin/clickhouse-server: Success
During startup program exited with code 126.
(gdb) t apply all bt
(gdb)