Habitat: Problem with Gossip exchange when nodes have multiple NICs

[janeczku]

I have a problem getting Gossip exchange to work in an environment where Docker containers are connected to multiple networks.
Containers have two NICs:
eth0 (docker0 bridge network)
eth1 (multi-host overlay network)

The supervisor seems to report the first routable IP as it's gossip address to other peers. This makes the Gossip exchange fail since the eth0 IP is not routed to the other containers.
And given that IP addresses of Docker containers are ephemeral, the supervisor's --listen-peer option can't be used to solve this.

In Docker container networks you really want to always refer to other containers/services by their name and not an IP address. Since peers already do report their hostname, please make it an option for the supervisor to use the hostname of the peer when establishing Gossip connections.

[adamhjk]

If you set --listen-peer 0.0.0.0, what happens?

[janeczku]

@adamhjk If i do that other members try to connect to this node using 0.0.0.0 as address.

DEBUG:habitat_sup::gossip::server: Processing member Member {
    id: Uuid("1a7ac586-30d5-4ad8-b0f4-2cac61d1b1ad"),
    hostname: "877bc9e4baf7",
    ip: "0.0.0.0",
    gossip_listener: "0.0.0.0:9634",
    incarnation: LamportClock {
        counter: 0
    },
    health: Alive,
    permanent: false
}

(...)

DEBUG:habitat_sup::gossip::server: PingReq from Peer { member_id: Uuid("8dd18d18-3474-494b-957e-ff5cdd3d710f"), listening_on: "0.0.0.0:9634", proxy_through: Some("0.0.0.0:9634"), proxy_to: Some("0.0.0.0:9634") }
DEBUG:utp::socket: Connecting to 0.0.0.0:9634

and eventually it loops these log messages over and over:

hab-sup(TL): I have lost quorum - getting rid of any leader
DEBUG:habitat_sup::gossip::detector: Detector State: Detector {
    open_requests: {
        Uuid("a0c1a886-b555-44ae-9dd5-57a0e5722400"): RequestState {
            status: PingReq,
            timeout: SteadyTime(
                SteadyTime { tv_sec: 1844, tv_nsec: 377414528 }
            )
        },
        Uuid("c3a8d2be-1689-40da-961a-d6561f6a49cc"): RequestState {
            status: PingReq,
            timeout: SteadyTime(
                SteadyTime { tv_sec: 1843, tv_nsec: 559914708 }
            )
        }
    }
}
DEBUG:habitat_sup::gossip::server: Skipping ping of c3a8d2be-1689-40da-961a-d6561f6a49cc due to already running request

[juliandunn]

Since peers already do report their hostname, please make it an option for the supervisor to use the hostname of the peer when establishing Gossip connections.

So if I understand what you mean, you would like --peer the-other-peer.example.com?

[janeczku]

@juliandunn Not exactly - passing a DNS name via --peer is already supported now. The problem is that supervisors report an incorrect IP address in the gossip_listener attribute of their own member rumor. As a result other peers fail to establish a connection back to them.

Their needs to be support for establishing any Gossip connections based on DNS names instead of IP addresses. For example, there could be an option that makes supervisors use their hostname when constructing the gossip_listener attribute of their own member information.

[janeczku]

@adamhjk @juliandunn

Steps to reproduce

Requires Docker client/engine >= v1.10.1

Create two non-routable networks (ext-1/ext-2) and one shared network (vpn) over which containers can communicate with each other.

In the container it will look like this:

/ # ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:AC:16:00:02           <------- ext-1
          inet addr:172.22.0.2  Bcast:0.0.0.0  Mask:255.255.0.0

eth1      Link encap:Ethernet  HWaddr 02:42:0A:32:00:04           <-------- vpn
          inet addr:10.50.0.4  Bcast:0.0.0.0  Mask:255.255.0.0

docker network create ext-1
docker network create ext-2
docker network create --subnet 10.50.1.0/16 vpn

Create seed and peer Habitat containers with leader topology:

docker create -ti --name hab-seed --net=ext-1 janeczku/rabbitmq --topology leader
docker create -ti --name hab-peer --link hab-seed --net=ext-2 janeczku/rabbitmq \
--topology leader --peer hab-seed

Attach both containers to the vpn network:

docker network connect vpn hab-seed
docker network connect vpn hab-peer

Start the containers:

docker start -ai hab-seed
docker start -ai hab-peer

[adamhjk]

Thank you so much for the clear guidelines to reproduce.

I think one answer might be to detect the interface we will use to send the rumor to the remote peer, and then automatically set that as the response. Another possible implementation is to simply reply to the address we received the connection from, which is probably even easier. We would simply need to pick up the port number in that case.

Either way, we'll fix this.

[janeczku]

@adamhjk Sounds like the right way to tackle this. Looking forward to the fix!

[isavcic]

One possible solution would be to provide a mechanism to override how the automatic detection of the listening address works, by a shell snippet or something similar. Or, to provide a mechanism to specify an interface instead of the address.

[isavcic]

Moreover, I really don't know why can't we use --listen-peer foobar:9634 if we've set --name foobar for the name of the container with docker run (since foobar expands to the IP of the eth1 on the overlay network and that's the one we want to use).

[psyhomb]

We really need for this issue to go "poof"

[isavcic]

The quick'n'dirty workaround we've created today is:

--- /hab/pkgs/core/hab-studio/0.8.0/20160729195909/libexec/hab-studio-type-baseimage.sh 2016-08-01 20:46:27.015921640 +0200
+++ hab-studio-type-baseimage.sh    2016-08-05 16:05:17.883631189 +0200
@@ -115,9 +115,14 @@
   $bb cat <<EOT > $HAB_STUDIO_ROOT/init.sh
 #!$busybox_path/bin/sh
 export PATH=$full_path
+
+if [[ -n "\$HAB_GOSSIP_IF" ]]; then
+  export LISTEN_PEER="--listen-peer \$(ifconfig \$HAB_GOSSIP_IF | awk -F'[: ]' '/inet addr/ {print \$13}'):9634"
+fi
+
 case \$1 in
   -h|--help|help|-V|--version) exec $sup "\$@";;
-  -*) exec $sup start \$(cat /.hab_pkg) "\$@";;
+  -*) exec $sup start \$(cat /.hab_pkg) "\$@" \$LISTEN_PEER;;
   *) exec $sup "\$@";;
 esac
 EOT

For a container on an overlay network, this works by adding -e "HAB_GOSSIP_IF=eth0" to the docker run parameters.

[sobakasu]

thanks @isavcic for posting that patch. I was having a similar problem with gossip exchange running services on rancher/docker. From rancher docs - "Under Rancher’s network, a container will be assigned both a Docker bridge IP (172.17.0.0/16) and a Rancher managed IP (10.42.0.0/16) on the default docker0 bridge". The --listen-peer 0.0.0.0 approach didn't work for me either. The container names in rancher resolve to the rancher subnet (10.42), so I needed the supervisor to bind to that address.

Updated patch: hab-studio-type-baseimage.patch.txt

I add the environment setting HAB_GOSSIP_SUBNET="inet 10.42" to the docker compose.yml to make the supervisor bind to an address in the 10.42 subnet. Now gossip communication is working.

The code in the patch loops a few times if it can't find the subnet in ip addr output because I notice it takes a few seconds for rancher to set up the 10.42 network, and the hab init script starts before that happens.

[eeyun]

@janeczku Thank you for opening this bug.

It's obviously been a few months since it was originally submitted and since it was opened we actually completely rewrote our gossip layer. During triage today I tested this out with your steps above (against redis, not against rabbitmq mind you) and it _appears_ as though it might be resolved.

For now I'm going to close this issue out. If you get a moment to test in your own environment with the latest version of Habitat (v0.16.0) I'd super appreciate hearing whether or not you're still running into this! Feel free to comment and reopen if the issue persists and if so we'll be sure to get it in front of the team asap.