Clickhouse: mysql engine crash when session_track_state_change=1

[chijiajian]

Describe the bug or unexpected behaviour
Clickhouse-server service crash and stop If Query Mysql engine table

How to reproduce

• Clickhouse-server 19.14.3.3

Table setup

CREATE TABLE test.mysql_engine (id Int16, userid Int16, roleid Int16, cuid Int16, create_time DateTime, isalive Int16) ENGINE = MySQL('10.x.x.x:port', 'database', ' table', 'username', 'pwd')

Queries to run that lead to unexpected result
select * from test.mysql_engine limit 1;

Error message and/or stacktrace

Exception on client:
Code: 32. DB::Exception: Attempt to read after eof: while receiving packet from 10.130.14.120:9000

2019.10.29 10:57:36.398960 [ 156 ] {} <Fatal> BaseDaemon: ########################################
2019.10.29 10:57:36.399007 [ 156 ] {} <Fatal> BaseDaemon: (version 19.14.3.3 (official build)) (from thread 71) Received signal Aborted (6).
2019.10.29 10:57:36.399041 [ 156 ] {} <Fatal> BaseDaemon: 
2019.10.29 10:57:36.399063 [ 156 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f3143a9d1f7 0x7f3143a9e8e8 0x7f3143adcf47 0x7f3143b77d87 0x7f3143b75f40 0x561dc559df40 0x561dc55a8060 0x561dc559c1fc 0x561dc5597d16 0x561dc53d874b 0x561dc53dc393 0x561dc53dc93e 0x561dc53dcbaf 0x561dc34f833d 0x561dc32cbc5f 0x561dc32d229e 0x561dc32b51f5 0x561dc32d3fc3 0x561dc32d5e88 0x561dc33e2a30 0x561dc33e3bec 0x561dc00918e0 0x561dc009227b 0x561dc3f00870 0x561dc3f00f8d 0x561dc554f821 0x561dc554d5cc 0x561dc5cc2e60 0x7f314423ee25
2019.10.29 10:57:36.399137 [ 156 ] {} <Fatal> BaseDaemon: 3. 0x7f3143a9d1f7 raise /usr/lib64/libc-2.17.so
2019.10.29 10:57:36.399146 [ 156 ] {} <Fatal> BaseDaemon: 4. 0x7f3143a9e8e8 abort /usr/lib64/libc-2.17.so
2019.10.29 10:57:36.399165 [ 156 ] {} <Fatal> BaseDaemon: 5. 0x7f3143adcf47 __libc_message /usr/lib64/libc-2.17.so
2019.10.29 10:57:36.399189 [ 156 ] {} <Fatal> BaseDaemon: 6. 0x7f3143b77d87 __fortify_fail /usr/lib64/libc-2.17.so
2019.10.29 10:57:36.399201 [ 156 ] {} <Fatal> BaseDaemon: 7. 0x7f3143b75f40 __readonly_area /usr/lib64/libc-2.17.so
2019.10.29 10:57:36.399214 [ 156 ] {} <Fatal> BaseDaemon: 8. 0x561dc559df40 ma_read_ok_packet ?
2019.10.29 10:57:36.399220 [ 156 ] {} <Fatal> BaseDaemon: 9. 0x561dc55a8060 run_plugin_auth ?
2019.10.29 10:57:36.399226 [ 156 ] {} <Fatal> BaseDaemon: 10. 0x561dc559c1fc mthd_my_real_connect ?
2019.10.29 10:57:36.399232 [ 156 ] {} <Fatal> BaseDaemon: 11. 0x561dc5597d16 mysql_real_connect ?
2019.10.29 10:57:36.399270 [ 156 ] {} <Fatal> BaseDaemon: 12. 0x561dc53d874b mysqlxx::Connection::connect(char const*, char const*, char const*, char const*, unsigned int, char const*, char const*, char const*, char const*, unsigned int, unsigned int, bool) ?
2019.10.29 10:57:36.399283 [ 156 ] {} <Fatal> BaseDaemon: 13. 0x561dc53dc393 mysqlxx::Pool::allocConnection(bool) ?
2019.10.29 10:57:36.399289 [ 156 ] {} <Fatal> BaseDaemon: 14. 0x561dc53dc93e mysqlxx::Pool::initialize() ?
2019.10.29 10:57:36.399295 [ 156 ] {} <Fatal> BaseDaemon: 15. 0x561dc53dcbaf mysqlxx::Pool::Get() ?
2019.10.29 10:57:36.399338 [ 156 ] {} <Fatal> BaseDaemon: 16. 0x561dc34f833d DB::StorageMySQL::read(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::SelectQueryInfo const&, DB::Context const&, DB::QueryProcessingStage::Enum, unsigned long, unsigned int) ?
2019.10.29 10:57:36.399372 [ 156 ] {} <Fatal> BaseDaemon: 17. 0x561dc32cbc5f void DB::InterpreterSelectQuery::executeFetchColumns<DB::InterpreterSelectQuery::Pipeline>(DB::QueryProcessingStage::Enum, DB::InterpreterSelectQuery::Pipeline&, std::shared_ptr<DB::SortingInfo> const&, std::shared_ptr<DB::PrewhereInfo> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) ?
2019.10.29 10:57:36.399386 [ 156 ] {} <Fatal> BaseDaemon: 18. 0x561dc32d229e void DB::InterpreterSelectQuery::executeImpl<DB::InterpreterSelectQuery::Pipeline>(DB::InterpreterSelectQuery::Pipeline&, std::shared_ptr<DB::IBlockInputStream> const&) ?
2019.10.29 10:57:36.399400 [ 156 ] {} <Fatal> BaseDaemon: 19. 0x561dc32b51f5 DB::InterpreterSelectQuery::executeWithMultipleStreams() ?
2019.10.29 10:57:36.399407 [ 156 ] {} <Fatal> BaseDaemon: 20. 0x561dc32d3fc3 DB::InterpreterSelectWithUnionQuery::executeWithMultipleStreams() ?
2019.10.29 10:57:36.399413 [ 156 ] {} <Fatal> BaseDaemon: 21. 0x561dc32d5e88 DB::InterpreterSelectWithUnionQuery::execute() ?
2019.10.29 10:57:36.399418 [ 156 ] {} <Fatal> BaseDaemon: 22. 0x561dc33e2a30 ? ?
2019.10.29 10:57:36.399426 [ 156 ] {} <Fatal> BaseDaemon: 23. 0x561dc33e3bec DB::executeQuery(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool) ?

[filimonov]

Do you use official build?

[chijiajian]

Do you use official build?

yes

[alexey-milovidov]

Do you use official build?

@filimonov You can see it from the log message: version 19.14.3.3 (official build)

[filimonov]

@chijiajian what is the exact version (and distribution) of mysql you use? Any non-default setting on mysql side?

Was it one-time issue, or it reproduces every time?

[chijiajian]

@filimonov The Version of mysql is 5.7.2. I find out the problem.
If using set global session_track_state_change = 1; setting on mysql, It throw this exception every time.

And If using set global session_track_state_change = 0; mysql engine is working well.

thx.

[alexey-milovidov]

We are using mariadbclient library (as opposing to mysqlclient library).
It might be a bug in this library that make it incompatible with the session_track_state_change feature.