Fabio: Q: How does fabio handle consul agent outages?

[xiaods]

Becauz fabio is based on consul. if the consul is down, how about keep the fabio detect it and register it to another consul service.

[magiconair]

fabio talks to the local consul agent which you should keep running. Also, you should have multiple fabio instances running and sending traffic to them. If you have only one fabio and one consul then you have a single point of failure.

Can you explain the problem you're having a bit more?

[xiaods]

@magiconair i have run consul cluster in our infra. then you know the fabio one by one registered to local consul agent. then i manually stop a local agent, i need a way to let fabio know the local agent is down, then the fabio should be handle the case and can switch the fabio to another consul?

actually @magiconair could you please share some best practice to deploy fabio and consul. we need some guide on this production direction.

[magiconair]

The first thing is that you need to setup a proper consul cluster, i.e. some master nodes and local agents on all the machines handling traffic. From your description it is not clear whether you have a functional consul cluster or just individual consul instances.

I agree that fabio could be a bit smarter when the local consul agent is down but this hasn't been an issue for us. Other stuff breaks first. If you're concerned about that you could hide the consul agents behind a load balancer but I'm not aware of people doing that. Maybe someone from the community can comment?

I need to double check but AFAIR fabio will not update the routing table when the local consul agent is down. It will just continue to serve traffic with the last known good routing table.

So my advice is:

1. Setup 3 node consul cluster for the servers
2. Install the consul agent on all the nodes pointing to the masters
3. Install fabio on all the nodes (or just all frontend nodes)
4. Configure your external load balancer to forward all traffic to the fabio nodes

Please let me know whether that helps.

[xiaods]

@magiconair thanks for your patience guide. let me testing it again. then give more feedback on your suggest.

[pangzheng]

For example: the deployment of the three consul master the same number of fabio nginx, fabio front is haproxy, close one consul.

haproxy on fabio node is still 3, but has been working error exception. (Unable to service discovery)

Now think of ways to haproxy background set consul dns resolve exceptions fabio problem.

This deployment scenario is right? @magiconair

[magiconair]

As I've said before: we see other stuff break way before our consul agents die. As long as the setup does not change the fabio instance with the dead consul agent should still forward traffic with the last known routing table. Your monitoring should catch the consul agent failing and ideally just restart it. If the consul agent does not come up my guess is you have bigger issues since it then cannot access the consul servers or the configuration is fixed.

Yes, fabio could handle this more gracefully by trying a backup consul agent but first see whether that is a real issue on your platform.

[pangzheng]

@magiconair
Thank you for your answer。
How to set up a backup consul agent in fabio?

[magiconair]

Right now you can't.

[pangzheng]

@magiconair
Thanks

Setting a consul two health in a service, a health check is exception.
Why fabio this service is normal?

"checks": [{"id": "chk-borg-'${IP}'", "http": "http://'${IP}':5013/v1/health", "interval": "10s","timeout": "1s"}, {"id": "chk-fabio-'${IP}'", "http": "http://'${IP}':9998/health", "interval": "10s","timeout": "1s"}]

[magiconair]

@pangzheng If I understand your question correctly then you are asking why the fabio health check is OK if one of the registered services in consul is not OK, correct?

The fabio health check does not depend on the health of registered services. The fabio health check becomes not OK if fabio itself is not working anymore. If a service health check fails then fabio will not route traffic to it.

Does that answer your question?

[dcparker88]

I need to double check but AFAIR fabio will not update the routing table when the local consul agent is down. It will just continue to serve traffic with the last known good routing table.

I've been doing some testing here, and it seems like Fabio starts to 404 every route if we disable the local consul-client. Is there a way to cache the latest routing table in Fabio, or could I be doing something wrong with my test? I would expect Fabio to not add/remove routes at all if the local consul connection is down

[magiconair]

@dcparker88 That is the current behavior since the assumption is that your setup cannot function without Consul anyway. Having said that, we could add an option to not update the routing table when consul is gone. Anyone up for a PR?

[dcparker88]

yeah - we'll take a look (sorry for late response, was on vacation)

[scalp42]

super late but see #811

[leprechau]

Closing this with a reference to #811 that @scalp42 mentioned.