Proxysql: CRUSH and restart proxysql Error: signal 11 and main.cpp:905:ProxySQL_daemonize_phase3()

Created on 10 Apr 2018 · 2Comments · Source: sysown/proxysql

Hi! I hope you can help me)
I have two instance in amazon ec2 with instance type c3.large (2CPU, 3.75RAM, ssd) and Amzon Linux ( 2017.09.1).
On first server installed proxysql.x86_64 1.4.6-1, on second proxysql.x86_64 1.4.7-1. Proxysqls config on both servers is equal:

datadir="/var/lib/proxysql" 
admin_variables= 
{
    admin_credentials="admin:admin"
    mysql_ifaces="127.0.0.1:6032"
}
mysql_variables=
{
    threads=2
    max_connections=2048
    default_query_delay=0
    default_query_timeout=36000000
    have_compress=true
    poll_timeout=2000
    interfaces="0.0.0.0:6033"
    default_schema="information_schema"
    stacksize=1048576
    server_version="5.5.30"
    connect_timeout_server=3000
    monitor_username="monitor"
    monitor_password="monitor"
    monitor_history=600000
    monitor_connect_interval=60000
    monitor_ping_interval=10000
    monitor_read_only_interval=1500
    monitor_read_only_timeout=500
    ping_interval_server_msec=120000
    ping_timeout_server=500
    commands_stats=true
    sessions_sort=true
    connect_retries_on_failure=10
}

| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
| 10           | db1.88888.rds.amazonaws.com | 3306 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 20           | db1.88888.rds.amazonaws.com | 3306 | ONLINE | 1      | 0           | 1000            | 0                   | 0       | 0              |         |
| 20           | rr1.88888.rds.amazonaws.com | 3306 | ONLINE | 1000   | 0           | 1000            | 1                   | 0       | 0              |         |` 
We dont have any problems with connection to mysql servers (ping and monitor)

But in logs on both servers:
`Standard Query Processor rev. 0.2.0902 -- Query_Processor.cpp -- Thu Feb  1 02:08:14 2018
In memory Standard Query Cache (SQC) rev. 1.2.0905 -- Query_Cache.cpp -- Thu Feb  1 02:08:14 2018
Standard MySQL Monitor (StdMyMon) rev. 1.2.0723 -- MySQL_Monitor.cpp -- Thu Feb  1 02:08:14 2018
Error: signal 11:
proxysql(_Z13crash_handleri+0x1a)[0x44bb0a]
/lib64/libc.so.6(+0x35270)[0x7f261dc71270]
proxysql(_ZN14MySQL_Protocol18get_binds_from_pktEPvjP22MySQL_STMT_Global_infoPP23stmt_execute_metadata_t+0x1d8)[0x48dcd8]
proxysql(_ZN13MySQL_Session7handlerEv+0xafa)[0x48742a]
proxysql(_ZN12MySQL_Thread20process_all_sessionsEv+0x1fc)[0x47021c]
proxysql(_ZN12MySQL_Thread3runEv+0xd76)[0x478ef6]
proxysql(_Z24mysql_worker_thread_funcPv+0x6c)[0x449b3c]
/lib64/libpthread.so.0(+0x7de5)[0x7f261f593de5]
/lib64/libc.so.6(clone+0x6d)[0x7f261dd3430d]
2018-04-10 09:56:08 main.cpp:905:ProxySQL_daemonize_phase3(): [ERROR] ProxySQL crashed. Restarting!
2018-04-10 09:56:08 [INFO] Angel process started ProxySQL process 6069
Standard ProxySQL Cluster rev. 0.1.0702 -- ProxySQL_Cluster.cpp -- Thu Feb  1 02:08:14 2018
Standard ProxySQL Statistics rev. 1.4.1027 -- ProxySQL_Statistics.cpp -- Thu Feb  1 02:08:14 2018
Standard ProxySQL HTTP Server Handler rev. 1.4.1031 -- ProxySQL_HTTP_Server.cpp -- Thu Feb  1 02:08:14 2018
Standard ProxySQL Admin rev. 0.2.0902 -- ProxySQL_Admin.cpp -- Thu Feb  1 02:08:14 2018
Standard MySQL Threads Handler rev. 0.2.0902 -- MySQL_Thread.cpp -- Thu Feb  1 02:08:14 2018
Standard MySQL Authentication rev. 0.2.0902 -- MySQL_Authentication.cpp -- Thu Feb  1 02:08:14 2018
2018-04-10 09:56:08 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
2018-04-10 09:56:08 [INFO] Dumping mysql_servers
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
| hostgroup_id | hostname | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer |
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
+--------------+----------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-------------+
2018-04-10 09:56:08 [INFO] Dumping mysql_servers_incoming
+--------------+--------------------------------------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname                                         | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+--------------------------------------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10           | db1.88888.rds.amazonaws.com | 3306 | 1      | 0      | 0           | 1000            | 0                   | 0       | 0      |         |
| 20           | db1.88888.rds.amazonaws.com | 3306 | 1      | 0      | 0           | 1000            | 0                   | 0       | 0              |         |
| 20           | rr1.88888.rds.amazonaws.com | 3306 | 1000   | 0      | 0           | 1000            | 1                   | 0       | 0              |         |
+--------------+--------------------------------------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2018-04-10 09:56:08 [INFO] New mysql_replication_hostgroups table
writer_hostgroup: 10 , reader_hostgroup: 20, 
2018-04-10 09:56:08 [INFO] New mysql_group_replication_hostgroups table
2018-04-10 09:56:08 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 10 , address: db1.88888.rds.amazonaws.com , port: 3306 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 20 , address: db1.88888.rds.amazonaws.com , port: 3306 , weight: 1 , status: ONLINE , max_connections: 1000 , max_replication_lag: 0 , use_ssl: 0 , max_latency_ms: 0 , comment: 
HID: 20 , address: rr1.88888.rds.amazonaws.com , port: 3306 , weight: 1000 , status: ONLINE , max_connections: 1000 , max_replication_lag: 1 , use_ssl: 0 , max_latency_ms: 0 , comment: 
2018-04-10 09:56:08 [INFO] Dumping mysql_servers
+--------------+--------------------------------------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| hostgroup_id | hostname                                         | port | weight | status | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment | mem_pointer     |
+--------------+--------------------------------------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
| 10           | db1.88888.rds.amazonaws.com | 3306 | 1      | 0      | 0           | 1000            | 0                   | 0       | 0              |         | 139801649094912 |
| 20           | rr1.88888.rds.amazonaws.com | 3306 | 1000   | 0      | 0           | 1000            | 1                   | 0       | 0              |         | 139801649095040 |
| 20           | db1.88888.rds.amazonaws.com | 3306 | 1      | 0      | 0           | 1000            | 0                   | 0       | 0              |         | 139801649094784 |
+--------------+--------------------------------------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+-----------------+
Standard Query Processor rev. 0.2.0902 -- Query_Processor.cpp -- Thu Feb  1 02:08:14 2018
In memory Standard Query Cache (SQC) rev. 1.2.0905 -- Query_Cache.cpp -- Thu Feb  1 02:08:14 2018
Standard MySQL Monitor (StdMyMon) rev. 1.2.0723 -- MySQL_Monitor.cpp -- Thu Feb  1 02:08:14 2018

In core dump.
``[ec2-user@]$ sudo gdb /var/lib/proxysql/core.2494 GNU gdb (GDB) Amazon Linux (7.6.1-64.33.amzn1) Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-amazon-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... [New LWP 2498] [New LWP 2494] [New LWP 2496] [New LWP 2495] [New LWP 2499] [New LWP 2507] [New LWP 2497] [New LWP 2500] [New LWP 2501] [New LWP 2502] [New LWP 2503] [New LWP 2504] [New LWP 2505] [New LWP 2506] [New LWP 2508] [New LWP 2509] Missing separate debuginfo for the main executable file Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/0c/764f83a53c3d4911e8a5ef3fe6149c01a84478 Core was generated byproxysql -c /etc/proxysql.cnf -D /var/lib/proxysql'.
Program terminated with signal 11, Segmentation fault.