Proxysql: Segfault when upgrading to 2.1.0

[patrickdk77]

Running docker image proxysql/proxysql:2.1.0, proxysql/proxysql:2.0.15 does not have this issue.

2021-01-15 03:45:25 MySQL_Session.cpp:3847:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection during query on (2,10.1.0.141,3306,1395441) , FD (Conn:59 , MyDS:59) : 2027, received malformed packet
2021-01-15 03:45:25 MySQL_Session.cpp:3847:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection during query on (2,10.1.0.148,3306,1427181) , FD (Conn:58 , MyDS:58) : 2000, Unknown MySQL error
2021-01-15 03:45:25 MySQL_Session.cpp:3847:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection during query on (2,10.1.0.141,3306,1395443) , FD (Conn:59 , MyDS:59) : 2027, received malformed packet
2021-01-15 03:45:25 MySQL_Session.cpp:3847:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection during query on (2,10.1.0.148,3306,1427183) , FD (Conn:58 , MyDS:58) : 2000, Unknown MySQL error
2021-01-15 03:45:25 MySQL_Session.cpp:3847:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection during query on (2,10.1.0.141,3306,1395445) , FD (Conn:59 , MyDS:59) : 2000, Unknown MySQL error
2021-01-15 03:45:25 MySQL_Session.cpp:3847:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection during query on (2,10.1.0.147,3306,1422402) , FD (Conn:61 , MyDS:61) : 2000, Unknown MySQL error
2021-01-15 03:45:26 MySQL_Session.cpp:3847:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection during query on (2,10.1.0.141,3306,1395448) , FD (Conn:60 , MyDS:60) : 2027, received malformed packet
Error: signal 11:
proxysql(_Z13crash_handleri+0x2a)[0x55f2978c7eea]
/lib/x86_64-linux-gnu/libc.so.6(+0x33060)[0x7fb62efda060]
proxysql(unpack_field+0x4e)[0x55f297d901fa]
proxysql(mthd_my_read_metadata_ex+0x179)[0x55f297d9103a]
proxysql(mthd_my_read_metadata+0x31)[0x55f297d91145]
proxysql(mthd_stmt_get_param_metadata+0x3b)[0x55f297da19d2]
proxysql(mysql_stmt_prepare+0x354)[0x55f297da1dde]
proxysql(+0x76c2c3)[0x55f297dbc2c3]
proxysql(my_context_spawn+0x61)[0x55f297dbee64]
proxysql(mysql_stmt_prepare_start+0xa3)[0x55f297dbc380]
proxysql(_ZN16MySQL_Connection18stmt_prepare_startEv+0x50)[0x55f297a77470]
proxysql(_ZN16MySQL_Connection7handlerEs+0xab0)[0x55f297a79eb0]
proxysql(_ZN16MySQL_Connection11async_queryEsPcmPP13st_mysql_stmtP23stmt_execute_metadata_t+0xd1)[0x55f297a7aa21]
proxysql(_ZN13MySQL_Session7handlerEv+0x1a5)[0x55f297938d75]
proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x399)[0x55f297915879]
proxysql(_ZN12MySQL_Thread3runEv+0xad0)[0x55f29791d940]
proxysql(_Z24mysql_worker_thread_funcPv+0x6c)[0x55f2978c09dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7fb6302104a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fb62f08fd0f]
---- proxysql(_Z13crash_handleri+0x2a) [0x55f2978c7eea] : crash_handler(int)
---- proxysql(_ZN16MySQL_Connection18stmt_prepare_startEv+0x50) [0x55f297a77470] : MySQL_Connection::stmt_prepare_start()
---- proxysql(_ZN16MySQL_Connection7handlerEs+0xab0) [0x55f297a79eb0] : MySQL_Connection::handler(short)
---- proxysql(_ZN16MySQL_Connection11async_queryEsPcmPP13st_mysql_stmtP23stmt_execute_metadata_t+0xd1) [0x55f297a7aa21] : MySQL_Connection::async_query(short, char, unsigned long, st_mysql_stmt, stmt_execute_metadata_t)
---- proxysql(_ZN13MySQL_Session7handlerEv+0x1a5) [0x55f297938d75] : MySQL_Session::handler()
---- proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x399) [0x55f297915879] : MySQL_Thread::process_all_sessions()
---- proxysql(_ZN12MySQL_Thread3runEv+0xad0) [0x55f29791d940] : MySQL_Thread::run()
---- proxysql(_Z24mysql_worker_thread_funcPv+0x6c) [0x55f2978c09dc] : mysql_worker_thread_func(void*)

[renecannao]

2021-01-15 03:45:26 MySQL_Session.cpp:3847:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection during query on (2,10.1.0.141,3306,1395448) , FD (Conn:60 , MyDS:60) : 2027, received malformed packet

This is likely really a malformed packet: the library for backend communication is detecting this.
To investigate/confirm this we need the core dump AND the tcpdump of the traffic (unencrypted).

Recently we came across to a crash in 1.4 , but also possible in 2.0 and 2.1 , and it seems to be related to this bug in MySQL:
https://bugs.mysql.com/bug.php?id=83346
If you are using MySQL Query Cache, please disable it.

[m4ce]

We just upgraded proxysql in dev from 2.0.15 to 2.1.0, got a segfault :(

2021-01-26 12:15:10 [INFO] Latest ProxySQL version available: 2.1.0-544-g17a4b4a7
Error: signal 11:
proxysql(_Z13crash_handleri+0x2a)[0x556dce823c8a]
/lib/x86_64-linux-gnu/libc.so.6(+0x37840)[0x7f2e1bd06840]
/lib/x86_64-linux-gnu/libc.so.6(+0x15cb4c)[0x7f2e1be2bb4c]
proxysql(mthd_my_read_metadata_ex+0x11f)[0x556dcecc483a]
proxysql(mthd_my_read_metadata+0x31)[0x556dcecc499f]
proxysql(mthd_stmt_get_param_metadata+0x3b)[0x556dcecd560f]
proxysql(mysql_stmt_prepare+0x354)[0x556dcecd5a1b]
proxysql(+0x74de7b)[0x556dcecefe7b]
proxysql(my_context_spawn+0x61)[0x556dcecf2a1c]
[0x7f2e1957c870]
 ---- proxysql(_Z13crash_handleri+0x2a) [0x556dce823c8a] : crash_handler(int)

[DmitriyMurzin]

Yes, some problem (upgrade 2.0.15 to 2.1.0):

2021-01-26 13:37:57 [INFO] Creating new server in HG 10 : 127.0.0.1:3306 , gtid_port=0, weight=1, status=0
2021-01-26 13:37:57 [INFO] New mysql_group_replication_hostgroups table
2021-01-26 13:37:57 [INFO] New mysql_galera_hostgroups table
2021-01-26 13:37:57 [INFO] New mysql_aws_aurora_hostgroups table
2021-01-26 13:37:57 [INFO] Checksum for table mysql_servers is 16662569410876617521
2021-01-26 13:37:57 [INFO] Checksum for table mysql_replication_hostgroups is 4886696280093287786
2021-01-26 13:37:57 [INFO] MySQL_HostGroups_Manager::commit() locked for 2ms
Standard Query Processor rev. 2.0.6.0805 -- Query_Processor.cpp -- Mon Jan 4 21:22:57 2021
2021-01-26 13:37:57 [INFO] Starting new mysql event log file /var/logs/query_log.00000474
In memory Standard Query Cache (SQC) rev. 1.2.0905 -- Query_Cache.cpp -- Mon Jan 4 21:22:57 2021
Standard MySQL Monitor (StdMyMon) rev. 2.0.1226 -- MySQL_Monitor.cpp -- Fri Jan 8 21:37:02 2021
2021-01-26 13:37:57 [INFO] Latest ProxySQL version available: 2.1.0-544-g17a4b4a7

2021-01-26 13:37:57 MySQL_Session.cpp:3847:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection during query on (10,127.0.0.1,3306,3699286) , FD (Conn:64 , MyDS:64) : 2027, received malformed packet
2021-01-26 13:37:57 MySQL_Session.cpp:3855:handler_minus1_ClientLibraryError(): [WARNING] Retrying query.
2021-01-26 13:37:57 MySQL_Session.cpp:3847:handler_minus1_ClientLibraryError(): [ERROR] Detected a broken connection during query on (10,127.0.0.1,3306,3699287) , FD (Conn:64 , MyDS:64) : 2027, received malformed packet
Error: signal 11:
proxysql(_Z13crash_handleri+0x2a)[0x55a0c8d3beea]
/lib/x86_64-linux-gnu/libc.so.6(+0x33060)[0x7fc960796060]
proxysql(unpack_field+0x4e)[0x55a0c92041fa]
proxysql(mthd_my_read_metadata_ex+0x179)[0x55a0c920503a]
proxysql(mthd_my_read_metadata+0x31)[0x55a0c9205145]
proxysql(mthd_stmt_get_param_metadata+0x3b)[0x55a0c92159d2]
proxysql(mysql_stmt_prepare+0x354)[0x55a0c9215dde]
proxysql(+0x76c2c3)[0x55a0c92302c3]
proxysql(my_context_spawn+0x61)[0x55a0c9232e64]
proxysql(mysql_stmt_prepare_start+0xa3)[0x55a0c9230380]
proxysql(_ZN16MySQL_Connection18stmt_prepare_startEv+0x50)[0x55a0c8eeb470]
proxysql(_ZN16MySQL_Connection7handlerEs+0xab0)[0x55a0c8eedeb0]
proxysql(_ZN16MySQL_Connection11async_queryEsPcmPP13st_mysql_stmtP23stmt_execute_metadata_t+0xd1)[0x55a0c8eeea21]
proxysql(_ZN13MySQL_Session7handlerEv+0x1a5)[0x55a0c8dacd75]
proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x399)[0x55a0c8d89879]
proxysql(_ZN12MySQL_Thread3runEv+0xad0)[0x55a0c8d91940]
proxysql(_Z24mysql_worker_thread_funcPv+0x6c)[0x55a0c8d349dc]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7fc9619cc4a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fc96084bd0f]
---- proxysql(_Z13crash_handleri+0x2a) [0x55a0c8d3beea] : crash_handler(int)
---- proxysql(_ZN16MySQL_Connection18stmt_prepare_startEv+0x50) [0x55a0c8eeb470] : MySQL_Connection::stmt_prepare_start()
---- proxysql(_ZN16MySQL_Connection7handlerEs+0xab0) [0x55a0c8eedeb0] : MySQL_Connection::handler(short)
---- proxysql(_ZN16MySQL_Connection11async_queryEsPcmPP13st_mysql_stmtP23stmt_execute_metadata_t+0xd1) [0x55a0c8eeea21] : MySQL_Connection::async_query(short, char, unsigned long, st_mysql_stmt, stmt_execute_metadata_t)
---- proxysql(_ZN13MySQL_Session7handlerEv+0x1a5) [0x55a0c8dacd75] : MySQL_Session::handler()
---- proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x399) [0x55a0c8d89879] : MySQL_Thread::process_all_sessions()
---- proxysql(_ZN12MySQL_Thread3runEv+0xad0) [0x55a0c8d91940] : MySQL_Thread::run()
---- proxysql(_Z24mysql_worker_thread_funcPv+0x6c) [0x55a0c8d349dc] : mysql_worker_thread_func(void*)

[JavierJF]

Hi,

I'm afraid the issue template is very clear for reporting crashes, for further investigation of this issue, a coredump and the extra information required by the issue template is required, the backtrace is not enough for conducting a investigation, specially in the case of potential malformed packets.

Thank you.

[patrickdk77]

Recently we came across to a crash in 1.4 , but also possible in 2.0 and 2.1 , and it seems to be related to this bug in MySQL:
https://bugs.mysql.com/bug.php?id=83346
If you are using MySQL Query Cache, please disable it.

This did resolve my issue, disable query cache on the servers.
Sorry I didn't attach the coredump, as the system wasn't setup for it, and I didnt have time at that moment, was going create them today, but since this comment fixed the issue I don't think there is a need to look into it more for my problem.

[m4ce]

Thanks @patrickdk77 - though, I wonder what changed? We only really upgraded proxysql and nothing else changed within the infrastructure. In fact, downgrading fixes the issue. I can only assume this is a problem introduced within the new version of proxysql?

[renecannao]

@patrickdk77 : I confirm there is no need for the core dump if disabling the query cache on the servers solved the issue, as this confirms my theory.

@m4ce : what changed is that ProxySQL 2.1 supports CLIENT_DEPRECATE_EOF , while ProxySQL 2.0 doesn't.
Therefore if you end in a mix of connections to the backend using CLIENT_DEPRECATE_EOF and not, you trigger MySQL bug 18846 .

For example, it is possible that some of your client connections use CLIENT_DEPRECATE_EOF while others don't , but ProxySQL 2.0 doesn't even know what CLIENT_DEPRECATE_EOF is so all connections had it disabled.
Or it is possible that you have a mix of ProxySQL 2.0 and ProxySQL 2.1 running against the same backends

[renecannao]

That said, we need a way to mitigate this issue.
Probably we will introduce 2 new global variables to:
a) enable/disable CLIENT_DEPRECATE_EOF at frontend level, so to control if clients will be able to use this feature
b) enable/disable CLIENT_DEPRECATE_EOF at backend level, so to control if connections to backends will use this feature

B specifically should be the main way to mitigate this issue.

[Hitesh-Agrawal]

core.42.gz
@renecannao Attached is the core dump

[renecannao]

Hi @Hitesh-Agrawal .
Which binary are you using exactly, and which OS distro/version ?
Thanks

[Hitesh-Agrawal]

@renecannao - We are using your official docker image proxysql/proxysql:2.1.0

[renecannao]

Thanks!
We will review the core dumps

[axot]

Is there updates or workaround for this? We want to use this version because it supports Prometheus export.

[JavierJF]

Hi,

this issue is a candidate for being solved by #3434. If you are affected please update to v2.2.0. Please let us know if the issue is solved after upgrading.

Thank you.

[JavierJF]

Hi,

due to inactivity and because of being a candidate for being solved by #3434, this issue is being close.

Thank you.