Proxysql: Program received SIGABRT signal aborted.

[taylor840326]

@renecannao proxysql received SIGABRT signal when proxysql can't connect any mysql backends.
I hope proxysql can handle exceptions when all mysql backends can't connect.

1.My test environment is :

1.proxysql = 1.3.0h
2.os = CentOS 7 64bit
3.benchmark toosl = tpcc-mysql
4.mysql = 5.6.x

172.18.6.195 is mysql node IP address.

172.18.6.194 is proxysql node IP address.

172.18.6.193 is benchmark tools node IP address.

2.test case:

2.1.download and compile tpcc-mysql

$ git clone https://github.com/Percona-Lab/tpcc-mysql.git
$ cd tpcc-mysql/src ; make

2.2.download and install proxysql 1.3.0h

# yum install https://github.com/sysown/proxysql/releases/download/v1.3.0h/proxysql-1.3.0h-1-centos7.x86_64.rpm

2.3.download and start mysql 5.6

# docker pull mysql:5.6
# docker run --network host --name m56 -m 4096m -e MYSQL_ROOT_PASSWORD=111111 -d mysql:5.6
# docker exec -it m56 /usr/bin/mysql -p111111
mysql> create database tpcc;
mysql> grant all privileges on *.* to 'dev'@'%' identified by 'dev';
mysql> create user 'monitor'@'%' identified by 'monitor';
mysql> flush privileges;

2.4.setup mysql variables in proxysql

mysql> insert into mysql_users(username,password,default_hostgroup) values('dev','dev',0);
mysql> insert into mysql_servers(hostgroup_id,hostname,port,status) vlaues(0,'172.18.6.195',3306,'ONLINE'),(1,'172.18.6.195',3306,'ONLINE');
mysql> insert into mysql_query_rules(active,apply,username,match_digest,destination_hostgroup) values(1,1,'dev','^SELECT',1),(1,1,'dev','^INSERT',0),(1,1,'dev','^UPDATE',0),(1,1,'dev','^DELETE',0);
mysql> load mysql users to runtime; save mysql users to disk;
mysql> load mysql servers to runtime;save mysql servers to disk;
mysql> load mysql query rules to runtime;save mysql query rules to disk;

2.5.create table schema on mysql node

$ cd tpcc-mysql
$ cat create_table.sql |mysql -h 172.18.6.194 -udev -pdev -P6033 tpcc
$ cat add_fkey_idx.sql | mysql -h 172.18.6.194 -udev -pdev -P6033 tpcc

2.6.restart proxysql with gdb command. (on proxysql node)

# gdb /root/rpmbuild/SOURCES/proxysql/src/proxysql
(gdb) set args -c /etc/proxysql.cnf -D /var/lib/proxysql/ -f
(gdb) run

2.7.generate data

$ vim tpcc-mysql/load.sh
change HOST=172.18.6.194 ,-u dev , -p dev ,-P 6033
$ ./load.sh tpcc 1000

2.8. use iptables to close mysql port (on mysql node)

# iptables -A INPUT -d 172.18.6.195 -p tcp --dport 3306 -j DROP

2.9. proxysql get a SIGABRT signal.

sample:

2016-12-08 05:47:00 mysql_connection.cpp:595:handler(): [ERROR] Connect timeout on 172.18.6.195:3306 : exceeded by 3773us
2016-12-08 05:47:00 MySQL_Session.cpp:2156:handler(): [ERROR] Detected an offline server during query: 172.18.6.195, 3306
2016-12-08 05:47:00 MySQL_Session.cpp:2156:handler(): [ERROR] Detected an offline server during query: 172.18.6.195, 3306
2016-12-08 05:47:00 MySQL_Session.cpp:2156:handler(): [ERROR] Detected an offline server during query: 172.18.6.195, 3306
2016-12-08 05:47:00 MySQL_Session.cpp:2164:handler(): [WARNING] Retrying query.
proxysql: MySQL_Session.cpp:2181: int MySQL_Session::handler(): Assertion `0' failed.
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffefbff700 (LWP 8521)]

### 2.10. debug with gdb (on proxysql node)

(gdb) b MySQL_Session.cpp:2171
(gdb) run

2.11. open mysql port (on mysql node)

# iptables -D INPUT -d dn03 -p tcp --dport 3306 -j DROP

2.12. benchmark again (on benchmark node)

$ ./load.sh tpcc 1000

2.13. close mysql port (on mysql node)

# iptables -A INPUT -d 172.18.6.195 -p tcp --dport 3306 -j DROP

2.14. use gdb print informations

(gdb) print status
$1 = PROCESSING_STMT_EXECUTE
(gdb) print myds->DSS
$2 = STATE_NOT_INITIALIZED

[taylor840326]

insert any codes to handle PROCESSING_STMT_EXECUTE ?

[renecannao]

@taylor840326 : thank you for the report.
As you have probably noticed, ProxySQL tracks the status of the sessions and intentionally dies (assert()) if the session ends in a state not expected.
I think the not handled case here is a server becoming unavailable not during connect phase (this should be handled correctly), but during the execution of a prepared statements (PROCESSING_STMT_EXECUTE) :

2016-12-08 05:47:00 MySQL_Session.cpp:2156:handler(): [ERROR] Detected an offline server during query: 172.18.6.195, 3306

I will consider this a high priority bug.
Thank you again for the report.

[renecannao]

This was fixed long time ago