Emqx: How to gracefully evict clients on nodes

[mariusstaicu]

Environment

• OS: GKE node OS
• Erlang/OTP: 21.2
• EMQ: v3.25

I am running a multiple broker cluster in in GKE with a google LB in front. Currently, when I need to bring a node down for maintenance, all clients connected to the broker being pulled down are disconnecting and connecting instantly to another broker. This causes high spike of connect and disconnect events which creates a great amount of stress on the rest of the system.

Is there a way in which I could evict in a controlled way hundreds of clients at a time ? via API i saw there's only the possibility to evict one client at a time by clientid.

Also is there a way in which I could stop a broker from accepting new connecitons ? Something similar to cordoning of a kubernetes node ? That would be helpful in order to force new connections on other brokers.

Basically I'm just trying to find a way of controlling disconnects and connects over a larger period of time. Any suggestions welcome.

[tigercl]

@mariusstaicu We don't support this feature yet, but we'll consider implementing it in the future.

[mariusstaicu]

Meanwhile, I made a script to disconnect clients containing "clientid" in client id in batches, assuming that the broker API is exposed on local port 18083.

while read client; do echo Disconnecting $client...; curl -m10 -s -I $1 -u admin:public -X DELETE http://localhost:18083/api/v3/connections/$client | grep HTTP/1.1 | awk {'print $2'}; done < <(curl -u admin:public 'http://localhost:18083/api/v3/connections?_page=1&_limit=300' | jq -r '.data[].client_id | select(. | contains("clientid"))')

[kozelok]

Hello,

Any update on this functionality? It is hard to do rolling updates on large clusters without being able to easily control disconnect rate. Killing the service leads to a very high number of instant reconnects.