Swarmkit: Feature request: docker swarm nodes share a VIP (VIPS)

[mostolog]

Hi

As sketched here https://forums.docker.com/t/swarm-mode-vip-without-haproxy-using-vips/29202 actually swarm does HA services but doesn't HA docker hosts.

Requesting any node for service foo would work, due to wonderful mesh network. However, how would I know which host to ask if I have a few of docker swarms running? May I ask swarm1 hosts or swarm2?

It could be great if Docker creates a VIP ensemble to provide a "virtual host" to provide services, no matter if any host goes down.

Consider the following scenario:

• Docker swarm with 4 nodes
• Service foo with replicas=3
• Nodes configure VIPS
• Manager gets VIP (eg: 1.2.3.4)
• In the event of manager going down, other node will become manager and get the VIP

With the above, not only services will be on HA, but swarm/docker hosts too!

Hope I was clear on explanation.

[aluzzardi]

@mostolog Usually that would be handled by a load balancer on top of your docker hosts.

For instance, if you run on AWS, you could use ELB to load balance across your hosts.

/cc @mavenugo

[dongluochen]

Another (slower) way is to register all the managers to an (external) DNS record. The client would use the DNS name to access manager instead of IP. The client should retry different IPs until successful. For example, wget handles DNS name like this.

[mostolog]

@aluzzardi Yes, that COULD be handled by an external LB, but this is a feature request to avoid requiring an external piece and making Docker "self balanceable", not only services, but even hosts

@dongluochen That could have problems. If manager A and B are registered on the DNS and B crash, clients could still resolve DNS to B, where it won't work.

[dongluochen]

@mostolog DNS would return multiple IPs, including A and B. The client should try each IP until it's able to connect. Most web clients (wget, curl) work this way. DNS is a typical slow solution.

[mostolog]

@dongluochen As you said, client should try each IP until it's able to connect. ie: client must support that.

This proposal, on the other hand, will work out of the box, without any client requirement and for any protocol, not only HTTP.

[dongluochen]

@mostolog VIP is an IP address. It requires routing to deliver external requests to the VIP. For multiple swarm managers to share a VIP. It's either an external load balancer does the request forwarding, or you inject the VIP routing into routers which is outside the scope of Swarm.

[dongluochen]

One feasible solution is Docker client recognizes multiple managers. When a manager is unreachable, it automatically switches to next manager. Since docker client is part of docker, this should be possible.

cc @aluzzardi @aaronlehmann.

[mostolog]

@dongluochen It requires routing to deliver external requests to the VIP. For multiple swarm managers to share a VIP. It's either an external load balancer does the request forwarding, or you inject the VIP routing into routers which is outside the scope of Swarm.
One feasible solution is Docker client recognizes multiple managers. When a manager is unreachable, it automatically switches to next manager. Since docker client is part of docker, this should be possible.

Perhaps I'm confused but that's not how Kernel VIPS works. Isn't it? See https://www.youtube.com/watch?v=oFsJVV1btDU
IIUC, nodes create an ensemble and ONE of them get the IP, in the event of failure, another node will get the IP. No routing, not any external tools required, no routers needed.

[dongluochen]

@mostolog I take a look at the README of https://github.com/kobolog/gorb. It uses IPVS to route traffic to target servers. I think it uses IPVS to build a load balancer. IMHO I don't think it solves high availability problem for multiple Swarm managers. An external client has to resolve swarm managers into an IP, then routes requests to this IP. To make the managers highly available, you may

• use an external load balancer, which put the high availability requirement on external load balancer
• use Anycast, routing needs to be injected into network
• make client retry different IPs

Let me know if you are thinking a different scenario.

[mostolog]

AFAIK LVS can work this way:

• DockerHost1 eth0 has IP 192.168.0.2/24
• DockerHost2 eth0 has IP 192.168.0.3/24
• DockerHost3 eth0 has IP 192.168.0.4/24
• DockerHost4 eth0 has IP 192.168.0.5/24

LVS/IPVS is configured on hosts to "share" a VIP 192.168.0.1. One host will become master and get VIP (On a virtual mac?). eg:

• DockerHost1 eth0 has IP 192.168.0.2/24
• DockerHost2 eth0 has IP 192.168.0.3/24 and 192.168.0.1/24
• DockerHost3 eth0 has IP 192.168.0.4/24
• DockerHost4 eth0 has IP 192.168.0.5/24

On a switched/L2 network, any request to 192.168.0.1 will be sent to DockerHost2 normally
In the event of DockerHost2 going down, another host gets the VIP. eg:

• DockerHost1 eth0 has IP 192.168.0.2/24
• DOWN DockerHost2 eth0 has IP 192.168.0.3/24
• DockerHost3 eth0 has IP 192.168.0.4/24
• DockerHost4 eth0 has IP 192.168.0.5/24 and 192.168.0.1/24

Applied to docker, docker engine could make use of "--vip-address" parameter, so the ensemble will be HA and without SPOF. Even, this IP could be shared by managers or any node too.

http://www.ultramonkey.org/papers/lvs_tutorial/html/img3.png
http://www.austintek.com/LVS/LVS-HOWTO/mini-HOWTO/LVS-mini-HOWTO.html
http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/

[dongluochen]

On a switched/L2 network, any request to 192.168.0.1 will be sent to DockerHost2 normally

It works within a switched/L2 network. You can configure LVS to support this scenario. When the managers are in different subnets this wouldn't work. In Swarm we want to spread the managers into different failure domains, like across rack, building, availability zone to survive SPOF. There is no guarantee they are under one L3 subnet.

[mostolog]

I wouldn't consider adding this feature a bad option to solve "not needing an external balancing/HA piece" for same-networked clusters.

IMHO many users would love to have this feature to enable Docker host HA (as service HA is already achieved with swarm), and there's a HUGE numbers of users that don't cluster between datacenters that could make profit of this.

Anyhow, have you considered another "bundled/no need for external piece" alternatives?

Regards

Update:
Wouldn't NDP solve multiple network scenario?

[mavenugo]

@mostolog I agree that this is a very interesting and useful feature. But as you might have observed, most of the features that we develop that are bundled/no need for external piece follow a principle of portability. If a particular feature has some basic dependency on underlying infrastructure it becomes less portable. In case of networking (including Routing-mesh, Overlay networking, Service Discovery and even Bridge NAT) are all implemented with that portability in mind. In the case of providing docker host HA, it ties into how underlay is configured and becomes very hard to make it infrastructure agnostic. For example, one can use UCARP/VRRP to provide the host HA feature that you suggested. But that can be done externally by the users of docker swarm cluster.
As @aluzzardi and @dongluochen suggested, there are multiple ways of getting this done outside of the docker swarm depending on the environment : (ELB in AWS, Multiple A-record DNS (in a devops managed small/medium on-premises cluster), UCARP or VRRP (in a netops managed L2 underlay), etc...
If you have have the most portable way to achieve this without impacting the existing functionality such as routing-mesh, pls push a PR and we will review it and provide feedback.

[mostolog]

Thanks all for your replies.

[jroberts235]

I'm using keepalived and since swarm-mode has all ports on all nodes, there's no need for a load balancer. Just a VIP for the cluster.

Only thing you have to be sure to do is, assign a different priority on each node. I have 4 nodes, so I assigned them in descending order from 101, which is the Master.

I did just get this setup, so I can't say it's bullet proof, but it seems to work just fine.

Hope it's helpful!

[mostolog]

@jroberts235 Let me see if I understood your comment:
You have deployed a keepalived (cluster) out of docker swarm in order to manage docker VIP, right?

If so, that's exactly what this issues was intended to avoid, providing a native bundled load-balancer within docker engine.

[jroberts235]

Well for those who need it now, this is a much simpler solution than LVS, so I wanted to share my solution.

I totally agree that it should be included in swarm.

[jroberts235]

BUT - you don't need the complexity of a software load balancer to achieve this. You just need a heartbeat and the ability to attach/detach an IP, like keepalived does (and does very well), IMHO.

[mostolog]

@jroberts235 keepalived is based on IPVS.(=VIP=LVS)
In essence, I was asking to include that to enable docker-host to docker-host self configuration.