Clickhouse: DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding

[ermakus]

Failure on load dictionary from PostgreSQL over unixODBC connector

ClickHouse client version 19.3.4.
ClickHouse server version 19.3.4

File /etc/odbcinst.ini

[DEFAULT]
Driver = staffcop
[staffcop]
Description         = PostgreSQL connection to staffcop
Driver              = PostgreSQL Unicode
Database            = staffcop
Servername          = 127.0.0.1
UserName            = staffcop
Password            = XXX
Port                = 5432
Protocol            = 9.3
ReadOnly            = No
RowVersioning       = No
ShowSystemTables    = No
ConnSettings        =

File /etc/odbcinst.ini

[PostgreSQL ANSI]
Description = PostgreSQL ODBC driver (ANSI version)
Driver      = /usr/lib/x86_64-linux-gnu/odbc/psqlodbca.so
Setup       = /usr/lib/x86_64-linux-gnu/odbc/libodbcpsqlS.so
Debug       = 0
CommLog     = 1
UsageCount  = 1

[PostgreSQL Unicode]
Description = PostgreSQL ODBC driver (Unicode version)
Driver      = /usr/lib/x86_64-linux-gnu/odbc/psqlodbcw.so
Setup       = /usr/lib/x86_64-linux-gnu/odbc/libodbcpsqlS.so
Debug       = 0
CommLog     = 1
UsageCount  = 1

error log

2019.02.20 12:42:49.499761 [ 26 ] {43f63aab-34fc-11e9-ae85-f403435768d8} <Error> executeQuery: Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding (from [::1]:57808) (in query: SELECT dictGetString('analytics_filter', 'name_ru', arrayJoin(agent_event_filters)) AS "analytics_filter.name", min(agent_event_local_time) AS local_time_min, max(agent_event_local_time) AS local_time_max, sum(agent_event_activity_time) AS activity_time_sum, sum(agent_event_idle_time) AS idle_time_sum, sum(agent_event_call_duration) AS call_duration_sum, count() AS record_count  FROM agent_event  WHERE agent_event_local_time <= 1550699999 AND agent_event_local_time >= 1550613600 GROUP BY dictGetString('analytics_filter', 'name_ru', arrayJoin(agent_event_filters)) ORDER BY record_count DESC  LIMIT 0, 50 FORMAT TabSeparatedWithNamesAndTypes), Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x6f13346]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3399d82]
2. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::startBridgeSync() const+0x3f8) [0x3504fb8]
3. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::getIdentifierQuotingStyle()+0x6d) [0x35039dd]
4. /usr/bin/clickhouse-server(DB::XDBCDictionarySource::XDBCDictionarySource(DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Block const&, DB::Context const&, std::shared_ptr<DB::IXDBCBridgeHelper>)+0x498) [0x5d35828]
5. /usr/bin/clickhouse-server() [0x5d36909]
6. /usr/bin/clickhouse-server() [0x5d36ad0]
7. /usr/bin/clickhouse-server(DB::DictionarySourceFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::DictionaryStructure const&, DB::Context&) const+0x642) [0x6768202]
8. /usr/bin/clickhouse-server(DB::DictionaryFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context&) const+0x292) [0x6766e32]
9. /usr/bin/clickhouse-server(DB::ExternalDictionaries::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x4a) [0x62ba96a]
10. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1248) [0x62be078]
11. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFiles(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xab) [0x62c081b]
12. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadAndUpdate(bool)+0x5e) [0x62c0f4e]
13. /usr/bin/clickhouse-server(DB::ExternalLoader::init(bool)+0x6a) [0x62c2a3a]
14. /usr/bin/clickhouse-server(DB::ExternalDictionaries::ExternalDictionaries(std::unique_ptr<DB::IExternalLoaderConfigRepository, std::default_delete<DB::IExternalLoaderConfigRepository> >, DB::Context&, bool)+0x1d0) [0x62bb0d0]
15. /usr/bin/clickhouse-server(DB::Context::getExternalDictionariesImpl(bool) const+0x107) [0x628e9a7]
16. /usr/bin/clickhouse-server(DB::SyntaxAnalyzer::analyze(std::shared_ptr<DB::IAST>&, DB::NamesAndTypesList const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<DB::IStorage>) const+0x12f3) [0x640abb3]
17. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::shared_ptr<DB::IBlockInputStream> const&, std::shared_ptr<DB::IStorage> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x448) [0x62f60c8]
18. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x56) [0x62f7136]
19. /usr/bin/clickhouse-server(DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x7e7) [0x6303427]
20. /usr/bin/clickhouse-server(DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, DB::Context&, DB::QueryProcessingStage::Enum)+0x368) [0x62de6a8]
21. /usr/bin/clickhouse-server() [0x643bc24]
22. /usr/bin/clickhouse-server(DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)+0xfa) [0x643dc3a]
23. /usr/bin/clickhouse-server(DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&)+0x3344) [0x33b79f4]
24. /usr/bin/clickhouse-server(DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&)+0x3ca) [0x33ba23a]
25. /usr/bin/clickhouse-server(Poco::Net::HTTPServerConnection::run()+0x332) [0x7051412]
26. /usr/bin/clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x7049cbf]
27. /usr/bin/clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x16a) [0x704a09a]
28. /usr/bin/clickhouse-server(Poco::PooledThread::run()+0x77) [0x71265a7]
29. /usr/bin/clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x38) [0x7122468]
30. /usr/bin/clickhouse-server() [0xacbfecf]
31. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f53e68986ba]

2019.02.20 12:42:49.499875 [ 26 ] {43f63aab-34fc-11e9-ae85-f403435768d8} <Error> HTTPHandler: Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x6f13346]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3399d82]
2. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::startBridgeSync() const+0x3f8) [0x3504fb8]
3. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::getIdentifierQuotingStyle()+0x6d) [0x35039dd]
4. /usr/bin/clickhouse-server(DB::XDBCDictionarySource::XDBCDictionarySource(DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Block const&, DB::Context const&, std::shared_ptr<DB::IXDBCBridgeHelper>)+0x498) [0x5d35828]
5. /usr/bin/clickhouse-server() [0x5d36909]
6. /usr/bin/clickhouse-server() [0x5d36ad0]
7. /usr/bin/clickhouse-server(DB::DictionarySourceFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::DictionaryStructure const&, DB::Context&) const+0x642) [0x6768202]
8. /usr/bin/clickhouse-server(DB::DictionaryFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context&) const+0x292) [0x6766e32]
9. /usr/bin/clickhouse-server(DB::ExternalDictionaries::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x4a) [0x62ba96a]
10. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1248) [0x62be078]
11. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFiles(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xab) [0x62c081b]
12. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadAndUpdate(bool)+0x5e) [0x62c0f4e]
13. /usr/bin/clickhouse-server(DB::ExternalLoader::init(bool)+0x6a) [0x62c2a3a]
14. /usr/bin/clickhouse-server(DB::ExternalDictionaries::ExternalDictionaries(std::unique_ptr<DB::IExternalLoaderConfigRepository, std::default_delete<DB::IExternalLoaderConfigRepository> >, DB::Context&, bool)+0x1d0) [0x62bb0d0]
15. /usr/bin/clickhouse-server(DB::Context::getExternalDictionariesImpl(bool) const+0x107) [0x628e9a7]
16. /usr/bin/clickhouse-server(DB::SyntaxAnalyzer::analyze(std::shared_ptr<DB::IAST>&, DB::NamesAndTypesList const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<DB::IStorage>) const+0x12f3) [0x640abb3]
17. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::shared_ptr<DB::IBlockInputStream> const&, std::shared_ptr<DB::IStorage> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x448) [0x62f60c8]
18. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x56) [0x62f7136]
19. /usr/bin/clickhouse-server(DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x7e7) [0x6303427]
20. /usr/bin/clickhouse-server(DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, DB::Context&, DB::QueryProcessingStage::Enum)+0x368) [0x62de6a8]
21. /usr/bin/clickhouse-server() [0x643bc24]
22. /usr/bin/clickhouse-server(DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>, std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>)+0xfa) [0x643dc3a]
23. /usr/bin/clickhouse-server(DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&)+0x3344) [0x33b79f4]
24. /usr/bin/clickhouse-server(DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&)+0x3ca) [0x33ba23a]
25. /usr/bin/clickhouse-server(Poco::Net::HTTPServerConnection::run()+0x332) [0x7051412]
26. /usr/bin/clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x7049cbf]
27. /usr/bin/clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x16a) [0x704a09a]
28. /usr/bin/clickhouse-server(Poco::PooledThread::run()+0x77) [0x71265a7]
29. /usr/bin/clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x38) [0x7122468]
30. /usr/bin/clickhouse-server() [0xacbfecf]
31. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f53e68986ba]

Additional context

Same configuration works for most of users, but some server installs have persistent error clickhouse-odbc-bridge is not responding

clickhouse-odbc-bridge process available

ps aux | grep clickhouse-odbc-bridge
clickho+ 3121 0.0 0.0 150780 2864 ? Sl 12:42 0:00 /usr/bin/clickhouse-odbc-bridge odbc-bridge --http-port 9018 --listen-host localhost --http-timeout 1800000000

odbc connection works:

root@SC:/home/root777# isql -v "staffcop"
+---------------------------------------+
| Connected!
..

Any hint how to debug issue?

[alesapin]

Same configuration works for most of users, but some server installs have persistent error clickhouse-odbc-bridge is not responding

This means that some hosts are OK and some have clickhouse-odbc-bridge is not responding?

You can set logging files for odbc-bridge with following lines in config.xml <logger> section:

        <odbc_bridge_log>/var/log/clickhouse-server/clickhouse-odbc-bridge.log</odbc_bridge_log>
        <odbc_bridge_errlog>/var/log/clickhouse-server/clickhouse-odbc-bridge.err.log</odbc_bridge_errlog>
        <odbc_bridge_level>trace</odbc_bridge_level>

[ermakus]

Yes, most of hosts works correctly.
We use same OS (Ubuntu 16.04LTS), same postgresql & clickhouse versions, but VMs/servers is very different (deployed on end users hardware as part of enterprise product).

Also there is propagated error in clickhouse.server.err.log
looks like clickhouse tried call startBridgeSync() somewhere in retry loop / exception handler and got huge trace (~500Mb)

Its happened not always, sometime there is only 1-10 loops and then exception reported.

2019.02.05 13:44:20.508827 [ 25 ] {} <Error> HTTPHandler: Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception: 
...
many times more

Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.wh
at() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, e.what() = DB::Exception, Stack trace:

0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x9fde646]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x3317a42]
2. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::startBridgeSync() const+0x3f8) [0x3476658]
3. /usr/bin/clickhouse-server(DB::XDBCBridgeHelper<DB::ODBCBridgeMixin>::getIdentifierQuotingStyle()+0x6d) [0x347507d]
4. /usr/bin/clickhouse-server(DB::XDBCDictionarySource::XDBCDictionarySource(DB::DictionaryStructure const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Block const&, DB::Context const&, std::shared_ptr<DB::IXDBCBridgeHelper>)+0x498) [0x9facd88]
5. /usr/bin/clickhouse-server() [0x9fade69]
6. /usr/bin/clickhouse-server() [0x9fae030]
7. /usr/bin/clickhouse-server(DB::DictionarySourceFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::DictionaryStructure const&, DB::Context&) const+0x642) [0x973d352]
8. /usr/bin/clickhouse-server(DB::DictionaryFactory::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context&) const+0x28e) [0x973bcee]
9. /usr/bin/clickhouse-server(DB::ExternalDictionaries::create(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Poco::Util::AbstractConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x4a) [0x6f2fa5a]
10. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x16f8) [0x6f33678]
11. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadFromConfigFiles(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xab) [0x6f35a2b]
12. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadAndUpdate(bool)+0x5e) [0x6f3615e]
13. /usr/bin/clickhouse-server(DB::ExternalLoader::init(bool)+0x65) [0x6f37be5]
14. /usr/bin/clickhouse-server(DB::ExternalDictionaries::ExternalDictionaries(std::unique_ptr<DB::IExternalLoaderConfigRepository, std::default_delete<DB::IExternalLoaderConfigRepository> >, DB::Context&, bool)+0x1d0) [0x6f301c0]
15. /usr/bin/clickhouse-server(DB::Context::getExternalDictionariesImpl(bool) const+0xf0) [0x6f052e0]
16. /usr/bin/clickhouse-server(std::shared_ptr<DB::IFunctionBuilder> DB::FunctionFactory::createDefaultFunction<DB::FunctionDictGetString>(DB::Context const&)+0x27) [0x3836df7]
17. /usr/bin/clickhouse-server(std::_Function_handler<std::shared_ptr<DB::IFunctionBuilder> (DB::Context const&), std::shared_ptr<DB::IFunctionBuilder> (*)(DB::Context const&)>::_M_invoke(std::_Any_data const&, DB::Context const&)+0x23) [0x356a1a3]
18. /usr/bin/clickhouse-server(DB::FunctionFactory::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::Context const&) const+0x1d1) [0x7356381]
19. /usr/bin/clickhouse-server(DB::ActionsVisitor::visit(std::shared_ptr<DB::IAST> const&)+0xf01) [0x75d5e21]
20. /usr/bin/clickhouse-server(DB::ActionsVisitor::visit(std::shared_ptr<DB::IAST> const&)+0x1312) [0x75d6232]
21. /usr/bin/clickhouse-server(DB::ActionsVisitor::visit(std::shared_ptr<DB::IAST> const&)+0x1312) [0x75d6232]
22. /usr/bin/clickhouse-server(DB::ActionsVisitor::visit(std::shared_ptr<DB::IAST> const&)+0x1312) [0x75d6232]
23. /usr/bin/clickhouse-server(DB::ActionsVisitor::visit(std::shared_ptr<DB::IAST> const&)+0x1747) [0x75d6667]
24. /usr/bin/clickhouse-server(DB::ExpressionAnalyzer::getRootActions(std::shared_ptr<DB::IAST> const&, bool, std::shared_ptr<DB::ExpressionActions>&, bool)+0xd8) [0x7508d28]
25. /usr/bin/clickhouse-server(DB::ExpressionAnalyzer::getConstActions()+0xc9) [0x75097a9]
26. /usr/bin/clickhouse-server(DB::KeyCondition::getBlockWithConstants(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::SyntaxAnalyzerResult const> const&, DB::Context const&)+0x24f) [0x71e667f]
27. /usr/bin/clickhouse-server(DB::MergeTreeWhereOptimizer::MergeTreeWhereOptimizer(DB::SelectQueryInfo&, DB::Context const&, DB::MergeTreeData const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, Poco::Logger*)+0x62a) [0x728385a]
28. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::executeImpl(DB::InterpreterSelectQuery::Pipeline&, std::shared_ptr<DB::IBlockInputStream> const&, bool)+0x16db) [0x6f6b2fb]
29. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::shared_ptr<DB::IBlockInputStream> const&, std::shared_ptr<DB::IStorage> const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x732) [0x6f6bd82]
30. /usr/bin/clickhouse-server(DB::InterpreterSelectQuery::InterpreterSelectQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x56) [0x6f6caf6]
31. /usr/bin/clickhouse-server(DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::shared_ptr<DB::IAST> const&, DB::Context const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, DB::QueryProcessingStage::Enum, unsigned long, bool)+0x7e7) [0x6f78837]

2019.02.11 10:02:46.048353 [ 2 ] {} <Error> BaseDaemon: (from thread 90) Terminate called without an active exception
2019.02.11 10:02:46.071932 [ 2 ] {} <Error> BaseDaemon: ########################################
2019.02.11 10:02:46.105713 [ 2 ] {} <Error> BaseDaemon: (from thread 90) Received signal Aborted (6).
2019.02.11 10:02:46.207453 [ 2 ] {} <Error> BaseDaemon: 0. /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7f25cc394428]
2019.02.11 10:02:46.207472 [ 2 ] {} <Error> BaseDaemon: 1. /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f25cc39602a]
2019.02.11 10:02:46.207478 [ 2 ] {} <Error> BaseDaemon: 2. /usr/bin/clickhouse-server() [0x6dd6b8d]
2019.02.11 10:02:46.207670 [ 2 ] {} <Error> BaseDaemon: 3. /usr/bin/clickhouse-server(__cxxabiv1::__terminate(void (*)())+0x6) [0xaa01e26]
2019.02.11 10:02:46.207698 [ 2 ] {} <Error> BaseDaemon: 4. /usr/bin/clickhouse-server() [0xaa01e71]
2019.02.11 10:02:46.207705 [ 2 ] {} <Error> BaseDaemon: 5. /usr/bin/clickhouse-server(__cxa_rethrow+0x49) [0xaa017d9]
2019.02.11 10:02:46.207715 [ 2 ] {} <Error> BaseDaemon: 6. /usr/bin/clickhouse-server(DB::getCurrentExceptionMessage[abi:cxx11](bool, bool)+0x19f) [0x9fbf5df]
2019.02.11 10:02:46.218675 [ 2 ] {} <Error> BaseDaemon: 7. /usr/bin/clickhouse-server(DB::tryLogCurrentException(Poco::Logger*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x219) [0x9fbff19]
2019.02.11 10:02:46.218708 [ 2 ] {} <Error> BaseDaemon: 8. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadAndUpdate(bool)+0xa57) [0x6f36b57]
2019.02.11 10:02:46.218716 [ 2 ] {} <Error> BaseDaemon: 9. /usr/bin/clickhouse-server(DB::ExternalLoader::reloadPeriodically()+0x2a) [0x6f37b5a]
2019.02.11 10:02:46.218722 [ 2 ] {} <Error> BaseDaemon: 10. /usr/bin/clickhouse-server() [0xaa3e73f]
2019.02.11 10:02:46.218733 [ 2 ] {} <Error> BaseDaemon: 11. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f25cce456ba]

EDIT: Probably second exception "Received signal Aborted (6)" not related to first one

[ermakus]

cat /var/log/clickhouse-server/clickhouse-odbc-bridge.log

2019.02.21 10:26:10.172293 [ 1 ] {} <Debug> ApplicationStartup: Initializing subsystem: Logging Subsystem
2019.02.21 10:26:10.207094 [ 1 ] {} <Information> ApplicationStartup: Starting up
2019.02.21 10:26:10.311844 [ 1 ] {} <Information> ApplicationStartup: Listening http://[::1]:9018
2019.02.21 10:29:23.572722 [ 2 ] {} <Information> Application: Received termination signal (Terminated)
2019.02.21 10:29:23.579108 [ 1 ] {} <Debug> ApplicationStartup: Received termination signal.
2019.02.21 10:29:23.579465 [ 1 ] {} <Debug> ApplicationStartup: Waiting for current connections to close.
2019.02.21 10:29:23.801719 [ 1 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem
2019.02.21 10:29:23.807449 [ 2 ] {} <Information> BaseDaemon: Stop SignalListener thread
2019.02.21 10:30:18.267115 [ 1 ] {} <Debug> ApplicationStartup: Initializing subsystem: Logging Subsystem
2019.02.21 10:30:18.297625 [ 1 ] {} <Information> ApplicationStartup: Starting up
2019.02.21 10:30:18.420133 [ 1 ] {} <Information> ApplicationStartup: Listening http://[::1]:9018

error log is empty

Looks like problem at clickhouse side. Or, maybe, ipv6 bound server?

[alesapin]

clickhouse-odbc-bridge always dies with server. Seems like server get abortion signal in ExternalLoader (class which load dictionaries including odbc) while loading dictionaries, after that it restarts and again loading dictionaries and get abortion. Can you collect some statistics about uptime of troublesome server (SELECT uptime())?

[ermakus]

Looks like abort is unrelated to this issue, server do not crash on this error and correctly return it to client

ODBC bridge is also available

root@SC:/home/root777# curl http://localhost:9018/ping
Ok.

Looks like that
checkBridgeIsRunning() can't ping bridge somehow, meanwhile curl can.

[ermakus]

Finally, solved by downgrading to 19.1.6

Unfortunately, can't figure out exact version with regression yet

[alesapin]

Looks like abort is unrelated to this issue, server do not crash on this error and correctly return it to client.

This is impossible, server always shutdown after abort signal.

[ermakus]

I think abort caused by another error, not related to clickhouse-odbc-bridge is not responding because it heppend 6 days later (see timestumps)
So it coincidence i think. Probably...

2019.02.05 13:44:20.508827 [ 25 ] {} <Error> HTTPHandler: Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding
...
2019.02.11 10:02:46.048353 [ 2 ] {} <Error> BaseDaemon: (from thread 90) Terminate called without an active exception

ODBCBridgeHelper: clickhouse-odbc-bridge is not responding do not crash server, just return error message to HTTP client

Example of such error from client side:

500 Error at /analytics/aggregate/agent_event/

Traceback (most recent call last):
  File "/usr/share/staffcop/lib/django/core/handlers/base.py", line 115, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "./util/decorators.py", line 13, in _wrapped_view
  File "./analytics/views.py", line 430, in aggregate
  File "./cubes/backends/clickhouse/flat.py", line 414, in aggregate
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2075, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1416, in _handle_dbapi_exception
    util.reraise(*exc_info)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/usr/share/staffcop/env/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
    cursor.execute(statement, parameters)
  File "./clickhouse/connector.py", line 224, in execute
  File "./clickhouse/connector.py", line 355, in _process_response
  File "/usr/share/staffcop/lib/infi/clickhouse_orm/database.py", line 163, in select
    r = self._send(query, settings, True)
  File "./clickhouse/connector.py", line 117, in _send
Exception: Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding: 
Code: 410, e.displayText() = DB::Exception: ODBCBridgeHelper: clickhouse-odbc-bridge is not responding

Number of lines varies from one to thousands.

[AntonVozdvizhenskiy]

On 19.4.2 revision 54416, with same simptoms, helped command system reload dictionaries.

[k0st1an]

On version 19.7.3.9 (official build) with same symptoms

[k0st1an]

The error occurs when you restart the ClickHouse server or restart the server itself.

[alexey-milovidov]

@alesapin Do I remember correctly that we have observed similar issue recently?

[alesapin]

Yes, probably it's a problem with a too-small timeout. It's fixed in https://github.com/ClickHouse/ClickHouse/pull/10990. Also we had a lot of fixes in the dictionaries loader.

[alesapin]

I think this issue can be closed.