Proxysql: Healthy MySQL goes OFFLINE_HARD by no reason
Hi!
Some strange thing happens sporadically by unknown yet reason.
There are 3 mysql servers in a pool. master + 2 slaves. Write queries are going to master. Read queries to slaves.
All mysql's are running behind 3 proxysql servers that are running on different servers.
Applications are connecting to proxysql via 127.0.0.1.
All setup can work without any issues let's say for a week.
After some unknown time, one of three proxysql's decides that master is dead and server is being marked as OFFLINE_HARD until I restart proxysql.
After restart, everything is good for another unknown period of time.
This happens from time to time on any of proxysql.
According to log file, there happens some timeout on checking read_only state and that's all is visible.
I also checked these queries and got empty set for both. Maybe because log retention is not long or maybe because of proxysql empties the log. Don't know yet.
select * from mysql_server_ping_log where hostname="172.31.55.104" and ping_error not null;
Empty set (0.00 sec)
select * from mysql_server_connect_log where hostname="172.31.55.104" and connect_error not null;
Empty set (0.00 sec)
2017-06-01 17:16:55 MySQL_Monitor.cpp:586:monitor_read_only_thread(): [ERROR] Timeout on read_only check for 172.31.55.104:3306 after 506ms. If the server is overload, increase mysql-monitor_read_only_timeout. Assuming read_only=1
2017-06-01 17:16:55 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 1 , address: 172.31.68.0 , port: 3306 , weight: 1 , status: ONLINE , max_connections: 10000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 1 , address: 172.31.55.104 , port: 3306 , weight: 1 , status: ONLINE , max_connections: 10000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 1 , address: 172.31.92.141 , port: 3306 , weight: 1 , status: ONLINE , max_connections: 10000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 0 , address: 172.31.55.104 , port: 3306 , weight: 1 , status: ONLINE , max_connections: 10000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment:
2017-06-01 17:16:55 [INFO] Dumping current MySQL Servers structures for hostgroup ALL
HID: 1 , address: 172.31.68.0 , port: 3306 , weight: 1 , status: ONLINE , max_connections: 10000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 1 , address: 172.31.55.104 , port: 3306 , weight: 1 , status: ONLINE , max_connections: 10000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 1 , address: 172.31.92.141 , port: 3306 , weight: 1 , status: ONLINE , max_connections: 10000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment:
HID: 0 , address: 172.31.55.104 , port: 3306 , weight: 1 , status: ONLINE , max_connections: 10000 , max_replication_lag: 60 , use_ssl: 0 , max_latency_ms: 0 , comment:
2017-06-01 17:16:55 MySQL_HostGroups_Manager.cpp:530:commit(): [WARNING] Removed server at address 140088452248064, hostgroup 0, address 172.31.55.104 port 3306. Setting status OFFLINE HARD and immediately dropping all free connections. Used connections will be dropped when trying to use them
2017-06-01 17:16:55 [INFO] New mysql_replication_hostgroups table
writer_hostgroup: 0 , reader_hostgroup: 1,
2017-06-01 17:16:55 [INFO] New mysql_group_replication_hostgroups table
2017-06-01 17:16:55 MySQL_Session.cpp:2512:handler(): [ERROR] Detected an offline server during query: 172.31.55.104, 3306
2017-06-01 17:16:56 MySQL_Session.cpp:2512:handler(): [ERROR] Detected an offline server during query: 172.31.55.104, 3306
...
Could please anyone help what exactly makes proxysql think that server is dead? Maybe additional logging is required here in order to find out which one checks fails.
And also it's unknown how to make server alive back without restarting proxysql.
Thank you in advance.
andreygolev
All 20 comments
@obion : proxysql doesn't believe that the server is dead, but is not able to verify that is a master (read_only=0), as this message reports:
2017-06-01 17:16:55 MySQL_Monitor.cpp:586:monitor_read_only_thread(): [ERROR] Timeout on read_only check for 172.31.55.104:3306 after 506ms. If the server is overload, increase mysql-monitor_read_only_timeout. Assuming read_only=1
You should:
- increase
mysql-monitor_read_only_timeout - check the read only check time in
mysql_server_read_only_log
renecannao
on 2 Jun 2017
@renecannao Thanks for an answer. Got it.
Is it right, that proxysql marks server offline_hard on a single failed check and only manual intervention can fix it?
andreygolev
on 5 Jun 2017
When this occurs, does your proxysql process go to 100% CPU utilization?
On Jun 5, 2017 3:24 PM, "obion" notifications@github.com wrote:
@renecannao https://github.com/renecannao Thanks for an answer. Got it.
Is it right, that proxysql marks server offline_hard on a single failed
check and only manual intervention can fix it?—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/sysown/proxysql/issues/1039#issuecomment-306278287,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AVCtPKkAVWT0UFcm8EX8Ky64kQhuBisIks5sBFXggaJpZM4Nt5oi
.
paulcarlucci
on 5 Jun 2017
nope, nothing is visible on CPU metrics during this incident.
andreygolev
on 6 Jun 2017
@obion : can you please share the error log? I want to dig into it a bit further.
Server 172.31.55.104 was set OFFLINE_HARD in hostgroup 0, but was still online in hostgroup 1 , therefore proxysql should have automatically bring it back to hostgroup when the next read_only check was going to return 0.
Also, what is the value of mysql-monitor_read_only_interval ?
renecannao
on 6 Jun 2017
@renecannao
I hope you need this one log.
proxysql.txt
mysql-monitor_read_only_interval is 1500
andreygolev
on 6 Jun 2017
@obion : the error log is full of dumps of the mysql_servers table.
I will investigate what could cause it.
renecannao
on 7 Jun 2017
I thought that's just a status update :)
Thank you!
andreygolev
on 7 Jun 2017
+1 on this issue, we're seeing a server go into OFFLINE_HARD and not come back to ONLINE status even after ping checks are healthy. Interestingly, only the hostgroup 0 connection stays in OFFLINE_HARD - the hostgroup 1 entry shows ONLINE.
Here is some supporting data:
https://gist.github.com/ryanschwartz/4ad7d27f047d492a7a602de9a757e98b#file-gistfile1-txt-L272
https://gist.github.com/ryanschwartz/4ad7d27f047d492a7a602de9a757e98b#file-gistfile1-txt-L284
And these are the configuration values in our environment:
mysql_variables=
{
threads=4
max_connections=2048 # default
default_query_delay=0 # default
default_query_timeout=86400000 # default (ms)
have_compress=true # default
poll_timeout=2000 # default
interfaces="0.0.0.0:6033;/tmp/proxysql.sock"
default_schema="information_schema"
stacksize=1048576 # default = 0
server_version="5.5.30" # default
connect_timeout_server=3000 # default = 1000 (ms)
monitor_username="monitor"
monitor_password="LOL-PASSWORD-HERE"
monitor_history=600000 # default
monitor_connect_interval=60000 # default = 120000 (ms)
monitor_ping_interval=5000 # default = 60000 (ms)
monitor_read_only_interval=1500 # default = 1000 (ms)
monitor_read_only_timeout=500 # default = 800 (ms)
ping_interval_server_msec=120000 # default = 60000 (ms)
ping_timeout_server=500 # default = 200
commands_stats=true # default
sessions_sort=true # default
connect_retries_on_failure=5 # default = 10
}
We are using ProxySQL version 1.3.2-1-gd71a745, codename Truls.
ryanschwartz
on 19 Jun 2017
I didn't reproduce it yet, but looking the code I think I can see the bug (references to 1.3.7, but any version is affected).
When the server come back as read_only=0 , it enters in this part of the code.
Because status is OFFLINE_HARD, it will take action:
- dump mysql servers from runtime to database : this writes a lot in error log
- run and UPDATE that doesn't change the status: this is the bug
- load mysql servers from database to runtime : this also writes a lot in error log
A workaround is to DELETE the row in mysql_servers and run LOAD MYSQL SERVERS TO RUNTIME fast enough before Monitor module kicks in: this will avoid the need to restart.
This is a serious bug, and will be fixed ASAP.
Thanks.
renecannao
on 20 Jun 2017
A workaround is to DELETE the row in mysql_servers and run LOAD MYSQL SERVERS TO RUNTIME fast enough before Monitor module kicks in: this will avoid the need to restart.
MySQL [(none)]> select * from mysql_servers;delete from mysql_servers where hostgroup_id=0;LOAD MYSQL SERVERS TO RUNTIME;select * from mysql_servers;
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 0 | mul-db-d01-use1b | 3306 | OFFLINE_HARD | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
3 rows in set (0.00 sec)
Query OK, 1 row affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2 rows in set (0.00 sec)
MySQL [(none)]> select * from mysql_servers;
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 0 | mul-db-d01-use1b | 3306 | OFFLINE_HARD | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
3 rows in set (0.00 sec)
MySQL [(none)]> SELECT * FROM monitor.mysql_server_read_only_log ORDER BY time_start_us DESC LIMIT 10;
+------------------+------+------------------+-----------------+-----------+-------+
| hostname | port | time_start_us | success_time_us | read_only | error |
+------------------+------+------------------+-----------------+-----------+-------+
| mul-db-d01-use1b | 3306 | 1497990623646568 | 929 | 0 | NULL |
| mul-db-d01-use1b | 3306 | 1497990622146604 | 831 | 0 | NULL |
| mul-db-d01-use1b | 3306 | 1497990620646517 | 805 | 0 | NULL |
| mul-db-d01-use1b | 3306 | 1497990619146401 | 802 | 0 | NULL |
| mul-db-d01-use1b | 3306 | 1497990617646330 | 830 | 0 | NULL |
| mul-db-d01-use1b | 3306 | 1497990616174053 | 1131 | 0 | NULL |
| mul-db-d01-use1b | 3306 | 1497990614646253 | 1188 | 0 | NULL |
| mul-db-d01-use1b | 3306 | 1497990613146128 | 987 | 0 | NULL |
| mul-db-d01-use1b | 3306 | 1497990611645907 | 785 | 0 | NULL |
| mul-db-d01-use1b | 3306 | 1497990610145960 | 874 | 0 | NULL |
+------------------+------+------------------+-----------------+-----------+-------+
10 rows in set (0.00 sec)
Server returns in OFFLINE_HARD after ~1-2 seconds.
Restarting proxysql resolves the situation, and for the meantime we'll be increasing monitor_read_only_timeout to 1250ms.
ryanschwartz
on 20 Jun 2017
Issue solved.
Thank you for the report
renecannao
on 25 Jun 2017
@renecannao, what release do you anticipate this bugfix will be included in?
ryanschwartz
on 26 Jun 2017
Both 1.3.8 and 1.4.1 .
@ryanschwartz , since you compile it yourself on Alpine, maybe you don't need to wait for an official release? There are just 3 commits from 1.3.7 to 1.3.8 : https://github.com/sysown/proxysql/compare/v1.3.7...v1.3.8-dev
renecannao
on 26 Jun 2017
@renecannao thank you for the update. We've moved all our containers to Ubuntu, alpine is so old school. ;-)
ryanschwartz
on 26 Jun 2017
That's a great news!! :smile:
renecannao
on 26 Jun 2017
@renecannao - Issue persists on 1.3.8. ☹️
MySQL [(none)]> SELECT strftime('%s','now')\G select @@version;
*************************** 1. row ***************************
strftime('%s','now'): 1498685838
1 row in set (0.00 sec)
+------------------+
| @@version |
+------------------+
| 1.3.8-1-g1a10b36 |
+------------------+
1 row in set (0.00 sec)
MySQL [(none)]> SELECT * FROM stats.stats_mysql_connection_pool;
+-----------+------------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| hostgroup | srv_host | srv_port | status | ConnUsed | ConnFree | ConnOK | ConnERR | Queries | Bytes_data_sent | Bytes_data_recv | Latency_us |
+-----------+------------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 265 |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 0 | mul-db-d01-use1b | 3306 | OFFLINE_HARD | 1 | 0 | 3 | 0 | 1064 | 135782 | 142849 | 265 |
+-----------+------------------+----------+--------------+----------+----------+--------+---------+---------+-----------------+-----------------+------------+
3 rows in set (0.00 sec)
MySQL [(none)]> select * from mysql_servers;
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 0 | mul-db-d01-use1b | 3306 | OFFLINE_HARD | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
3 rows in set (0.00 sec)
MySQL [(none)]> SELECT strftime('%s','now')\G
*************************** 1. row ***************************
strftime('%s','now'): 1498685838
1 row in set (0.00 sec)
MySQL [(none)]> select * from mysql_servers;delete from mysql_servers where hostgroup_id=0;LOAD MYSQL SERVERS TO RUNTIME; SELECT strftime('%s','now')\G select * from mysql_servers;
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 0 | mul-db-d01-use1b | 3306 | OFFLINE_HARD | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
3 rows in set (0.00 sec)
Query OK, 1 row affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
*************************** 1. row ***************************
strftime('%s','now'): 1498685838
1 row in set (0.00 sec)
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2 rows in set (0.00 sec)
MySQL [(none)]> SELECT strftime('%s','now')\G select * from mysql_servers;
*************************** 1. row ***************************
strftime('%s','now'): 1498685838
1 row in set (0.00 sec)
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2 rows in set (0.00 sec)
MySQL [(none)]> SELECT strftime('%s','now')\G select * from mysql_servers;
*************************** 1. row ***************************
strftime('%s','now'): 1498685838
1 row in set (0.00 sec)
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2 rows in set (0.00 sec)
MySQL [(none)]> SELECT strftime('%s','now')\G select * from mysql_servers;
*************************** 1. row ***************************
strftime('%s','now'): 1498685839
1 row in set (0.00 sec)
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2 rows in set (0.00 sec)
MySQL [(none)]> SELECT strftime('%s','now')\G select * from mysql_servers;
*************************** 1. row ***************************
strftime('%s','now'): 1498685839
1 row in set (0.00 sec)
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
2 rows in set (0.00 sec)
MySQL [(none)]> SELECT strftime('%s','now')\G select * from mysql_servers;
*************************** 1. row ***************************
strftime('%s','now'): 1498685839
1 row in set (0.00 sec)
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 0 | mul-db-d01-use1b | 3306 | OFFLINE_HARD | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
3 rows in set (0.00 sec)
MySQL [(none)]> SELECT strftime('%s','now')\G select * from mysql_servers;
*************************** 1. row ***************************
strftime('%s','now'): 1498685840
1 row in set (0.00 sec)
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | mul-db-d01-use1b | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | mul-db-d01-use1c | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 0 | mul-db-d01-use1b | 3306 | OFFLINE_HARD | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+------------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
3 rows in set (0.00 sec)
Happy to provide any log data that would be helpful. Ping and read_only log data immediately after the above: https://gist.github.com/ryanschwartz/d92082bce5b724908df4c5f153bf44a0
ryanschwartz
on 28 Jun 2017
That's disappointing.
Can you please share proxysql.log ?
renecannao
on 29 Jun 2017
Container log: container.log.gz
ryanschwartz
on 29 Jun 2017
Any update on this issue @renecannao?
ryanschwartz
on 6 Jul 2017
Related issues
lazzyfu
·
19Comments
lengerad
·
23Comments
ghost
·
20Comments
asharpaev
·
15Comments
renecannao
·
20Comments
Most helpful comment
I didn't reproduce it yet, but looking the code I think I can see the bug (references to 1.3.7, but any version is affected).
When the server come back as
read_only=0, it enters in this part of the code.Because status is OFFLINE_HARD, it will take action:
A workaround is to
DELETEthe row inmysql_serversand runLOAD MYSQL SERVERS TO RUNTIMEfast enough before Monitor module kicks in: this will avoid the need to restart.This is a serious bug, and will be fixed ASAP.
Thanks.