ClickHouse client building issues

[daskol]

I am trying to build ClickHouse client from sources of version 1.1.54362-stable in the following manner.

  mkdir -p build/release-clang
  cd build/release-clang
  cmake \
    -D CMAKE_BUILD_TYPE:STRING=Release \
    -D CMAKE_C_COMPILER=clang \
    -D CMAKE_CXX_COMPILER=clang++ \
    -D UNBUNDLED:BOOL=False \
    -D USE_STATIC_LIBRARIES:BOOL=False \
    -D USE_INTERNAL_DOUBLE_CONVERSION_LIBRARY:BOOL=False \
    -D USE_INTERNAL_CAPNP_LIBRARY:BOOL=False \
    -D USE_INTERNAL_POCO_LIBRARY:BOOL=True \
    -D USE_INTERNAL_RE2_LIBRARY:BOOL=False \
    -D POCO_STATIC:BOOL=True \
    -D ENABLE_TESTS:BOOL=False \
    -D ENABLE_CAPNP:BOOL=False \
    -D ENABLE_DATA:BOOL=False \
    -D ENABLE_DATA_MYSQL:BOOL=False \
    -D ENABLE_DATA_ODBC:BOOL=False \
    -D ENABLE_DATA_POSTGRESQL:BOOL=False \
    -D ENABLE_DATA_SQLITE:BOOL=False \
    -D ENABLE_DEVEL:BOOL=False \
    -D ENABLE_EMBEDDED_COMPILER:BOOL=False \
    -D ENABLE_TCMALLOC:BOOl=False \
    -D ENABLE_UNWIND:BOOL=False \
    -D ENABLE_MONGODB:BOOL=False \
    -D ENABLE_MYSQL:BOOL=False \
    -D ENABLE_PDF:BOOL=False \
    -D ENABLE_POCODOC:BOOL=False \
    -D ENABLE_REDIS:BOOL=False \
    -D ENABLE_RDKAFKA:BOOL=False \
    -D ENABLE_CLICKHOUSE_ALL:BOOL=False \
    -D ENABLE_CLICKHOUSE_CLIENT:BOOL=True \
    ../..
  make -j4 clickhouse-client

During bulding I get the following compilation error.

...../dbms/src/Server/main.cpp:98:59: error: no template named 'vector' in namespace 'std'
bool isClickhouseApp(const std::string & app_suffix, std::vector<char *> & argv)

As I understand, option ENABLE_CLICKHOUSE_ALL:BOOL=False turns off includes that includes std::vectors.

When I am trying to build with GCC not Clang I have got another compilation error.

....../dbms/src/Columns/ColumnString.h: In member function ‘virtual DB::BlockInputStreams DB::StorageSystemAsynchronousMetrics::read(const Names&, const DB::SelectQueryInfo&, const DB::Context&, DB::QueryProcessingStage::Enum&, size_t, unsigned int)’:
....../dbms/src/Columns/ColumnString.h:96:45: error: ‘*((void*)&<anonymous> +8)’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
         const size_t size_to_append = s.size() + 1;
                                       ~~~~~~^~
cc1plus: all warnings being treated as errors

If I include <vector> explicitly then I build successfully but clickhouse-client throws exception double free or corruption.

[proller]

fix second error: cmake -DNO_WERROR=1

[proller]

can you show full cmake output ?

[daskol]

Click here.

[daskol]

@proller Do you plan make build matrix for clickhouse-client in Travis CI.

[proller]

Our builds in travis near time limits.
When we add any new build - wait for test result increased by hours

[daskol]

When we add any new build - wait for test result increased by hours.

I mean for release candidates or stable releases.

clickhouse-client throws exception double free or corruption

I rebuild Clickhouse client with RelWithDebInfo but it turns out that debug symbols are useless since it looks like exception was thrown on app init.

#0  0x00007f92caf66860 in raise () from /usr/lib/libc.so.6
#1  0x00007f92caf67ec9 in abort () from /usr/lib/libc.so.6
#2  0x00007f92cafa9437 in __libc_message () from /usr/lib/libc.so.6
#3  0x00007f92cafaf78b in malloc_printerr () from /usr/lib/libc.so.6
#4  0x00007f92cafb14fc in _int_free () from /usr/lib/libc.so.6
#5  0x00007f92caf697b2 in __cxa_finalize () from /usr/lib/libc.so.6
#6  0x00007f92c712c384 in ?? () from /usr/lib/libPocoNet.so.60
#7  0x00007ffda8e0cdd0 in ?? ()
#8  0x00007f92ced55af3 in _dl_fini () from /lib64/ld-linux-x86-64.so.2

However, I used GDB to find that connecting to server throws exception here. This occurres on clickhouse-client startup if there is no Clickhouse locally. If there is locally running ClickHouse then client throws exception on exit.

[daskol]

I have built dependency graph for target clickhouse-client with --graphviz CMake option. I wonder why client depends on heavy compiled full-featured parsers clickhouse_functions, clickhouse_table_functions, clickhouse_aggregate_functions.

The reason is presumably that clickhouse-client should parse query for pretty printing, printing selected columns in corresponding format and inserting in specified format. It seams that full-featured parser is not strictly required here. AST is too powerfull to determine names of output columns and input/output format of query so simple light-weighted query parser would work fine. What is your opinion?

[filimonov]

currently (and historically) clickhouse-client is rather fat client tightly integrated with server code. And there is only one query parser used both on client and server.

It has some advantages (DRY principle), and some drawbacks (client is fat). As I know changing that is not the part of (nearest) plans of Yandex team: https://clickhouse.yandex/docs/en/roadmap/

There are two third party 'thin clients' working via HTTP-protocol.
https://github.com/filimonov/chc
https://github.com/hatarist/clickhouse-cli

Also usql has clickhouse support (via native protocol, using go library by @kshvakov)

For GUI clients - DBeaver supports Clickhouse mostly 'out of the box', via HTTP-protocol and official jdbc driver

May be there are even more, which i don't know.

So you can contribute to Clickhouse client itself (a lot of improvements for client are possible, like autocompletion, windows support, pagers, proper handling of Ctrl+C when big stream of data is printed etc.), or use/contribute to one of that 3rd party clients, or create your own :)

[alexey-milovidov]

ClickHouse client is dependent on aggregate functions to allow read binary data with serialized aggregation states from the server (columns with type AggregateFunction(...)). This is useful, for example, to create data dumps of tables that have columns of such kind. But we can introduce a flag to disable this possibility (client will refuse to read columns of type AggregateFunction(...)). Note that serialized aggregation states are freeform binary data - they don't have explicit length prefix and cannot be skipped in a data stream without using the implementation of aggregate function to read them.

ClickHouse client is dependent on ordinary functions, because it sends data for insertion to the server in columar binary format. It needs to transform data from the INSERT query to binary format. And sometimes it requires to interpret arbitary expressions on the client side. Example: INSERT INTO t VALUES (1 + 2). Again, we can introduce a flag to turn off this possibility. (BTW, interpreting arbitary expressions in INSERT is slow and have some gotchas).

No idea, why client is dependent on table functions. I will remove this dependency right now.

Also we should extract Dictionaries to separate library from dbms. For example, MySQL (mysqlxx), ODBC (Poco::Data), MongoDB (Poco::MongoDB) are needed only for Dictionaries and TableFunctions.
btrie is needed only for Dictionaries.

[daskol]

I have finally built client for ClickHouse v1.1.54375-testing. I run clickhouse-client and quit then client throws exception with following stacktrace. I have found the same exception in older versions too.

Core was generated by `clickhouse-client'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fa87d128860 in raise () from /usr/lib/libc.so.6
(gdb) bt
#0    0x00007fa87d128860 in raise + 0x110 () at /usr/lib/libc.so.6
#1    0x00007fa87d129ec9 in abort + 0x1c9 () at /usr/lib/libc.so.6
#2    0x00007fa880a499b6 in No symbol matches 0x00007fa880a499b6. ()
                         at /usr/lib/libclickhouse.so.1
#3    0x00007fa87e017dd0 in <signal handler called> () at /usr/lib/libpthread.so.0
#4    0x0000000000000014 in No symbol matches 0x0000000000000014. () at None
#5    0x00007fa8806cd300 in Poco::Logger::shutdown() + 0x30 ()
                         at /usr/lib/libclickhouse.so.1
#6    0x00007fa8806d059d in Poco::AutoLoggerShutdown::~AutoLoggerShutdown() + 0x1d ()
                         at /usr/lib/libclickhouse.so.1
#7    0x00007fa87d12b7b2 in __cxa_finalize + 0x92 () at /usr/lib/libc.so.6
#8    0x00007fa879252fb4 in No symbol matches 0x00007fa879252fb4. () at /usr/lib/libPocoFoundation.so.60
#9    0x00007ffe8c6b1cc0 in No symbol matches 0x00007ffe8c6b1cc0. () at None
#10   0x00007fa8813f2af3 in _dl_fini + 0x1c3 () at /lib64/ld-linux-x86-64.so.2

[proller]

now tested on latest master:

mkdir -p build_client
cd build_client
cmake .. -DENABLE_CLICKHOUSE_ALL=0 -DENABLE_CLICKHOUSE_CLIENT=1 -DCMAKE_CXX_COMPILER=`which g++-7` -DCMAKE_C_COMPILER=`which gcc-7`
make -j clickhouse-client
dbms/src/Server/clickhouse-client --port 19000
СlickHouse client version 1.1.54377.
Connecting to localhost:19000.
Connected to ClickHouse server version 1.1.54377.

zzz :) select 1+1

SELECT 1 + 1

┌─plus(1, 1)─┐
│          2 │
└────────────┘

1 rows in set. Elapsed: 0.005 sec. 

zzz :) 

[alexey-milovidov]

PS. -D USE_STATIC_LIBRARIES:BOOL=False
This option is discouraged. Static linking is always preferred for ClickHouse.