Swarmkit: Manager reachable but context deadline exceeded always

[yoncise]

Docker version: Docker version 1.12.0, build 8eab29e

Docker info:

 Running: 3
 Paused: 0
 Stopped: 3
Images: 8
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 54
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host null overlay bridge
Swarm: active
 NodeID: cxaz16nmfzoaa5q30rmypb15e
 Is Manager: true
 ClusterID: e2ylkt4rd6blzmvj6jg1kp865
 Managers: 2
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot interval: 10000
  Heartbeat tick: 1
  Election tick: 3
 Dispatcher:
  Heartbeat period: 5 seconds
 CA configuration:
  Expiry duration: 3 months
 Node Address: 115.159.xxx.xxx
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-36-generic
Operating System: Ubuntu 14.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 994 MiB
Name: VM-39-112-ubuntu
ID: 2NC5:MTGA:LGRV:RI6U:7M35:AMBM:2PGG:QOI3:ASVU:QFJA:RWF6:ZAX6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

I have two hosts A and B and both of them have an internet IP, however, host A doesn't bind internet IP to interfaces (my cloud service provider map between internet IP and internal IP) that is to say, if I bind service to <internal ip>:<port> then my cloud service provider will create a map like <internet ip>:<port> => <internal ip>:<port>, so I can reach this service via <internet ip>:<port>

I init swarm in my host A with: docker swarm init --advertise-addr <internet ip> --listen-addr <internal ip>. Then my host B used docker swarm join \ --token <token>\ <A internet ip>:2377 to join swarm. Then I executed docker node ls in host A, I can see host B is active and reachable, however it seems A and B can not run Raft correctly. Host B will alert Error response from daemon: rpc error: code = 4 desc = context deadline exceeded always.

ping @aluzzardi

[aaronlehmann]

Does docker swarm join complete successfully on B? We've seen some problems when clocks are out of sync. We've made some changes since the release to help with this.

If the join does complete successfully, the problem sounds related to the 1-to-1 NAT. Can you try specifying --advertise-addr with the public address in the docker swarm join command? Generally this shouldn't be necessary, but I'm curious if it makes a difference.

[yoncise]

Thanks for your reply. I don't think join complete successfully, even though it doesn't alert any error and docker node ls on host A output <HOST B> Ready Active Reachable. Execute docker info in host B after joined swarm shows

Swarm: active
NodeID: dcsbmuxpm8g7k5af88wqxkva3
Error: rpc error: code = 4 desc = context deadline exceeded
Is Manager: true
ClusterID:
Managers: 0
Nodes: 0
Orchestration:
Task History Retention Limit: 0
Raft:
Snapshot interval: 0
Heartbeat tick: 0
Election tick: 0
Dispatcher:
Heartbeat period: Less than a second
CA configuration:
Expiry duration: Less than a second

add --advertise-addr when join swarm doesn't make any difference. BTW, host B doesn't live in a NAT network.

[aaronlehmann]

It would be useful to see the logs from both A and B. They might show failed connections or other problems.

Also, can you please verify that the clocks are correct on both machines?

[yoncise]

I checked out the logs on B:

time="2016-08-10T16:17:26.958326507+08:00" level=info msg="Initializing Libnetwork Agent Local-addr=120.26.43.xxx Adv-addr=120.26.43.xxx Remote-addr =10.105.39.112"
time="2016-08-10T16:17:36.958744588+08:00" level=error msg="Error in agentJoin : could not join node to memberlist: 1 error(s) occurred:\n\n* Failed to join 10.105.39.112: dial tcp 10.105.39.112:7946: i/o timeout"
time="2016-08-10T16:20:18.129821336+08:00" level=info msg="Waiting for TLS certificate to be issued..."
time="2016-08-10T16:20:18.295102084+08:00" level=info msg="Downloaded new TLS credentials with role: swarm-manager."
time="2016-08-10T16:20:18.369537331+08:00" level=info msg="Listening for connections" addr="[::]:2377" proto=tcp
time="2016-08-10T16:20:18.369719451+08:00" level=info msg="Listening for local connections" addr="/var/lib/docker/swarm/control.sock" proto=unix
time="2016-08-10T16:20:18.389250413+08:00" level=info msg="Initializing Libnetwork Agent Local-addr=120.26.43.xxx Adv-addr=120.26.43.xxx Remote-addr =115.xxx.xxx.xxx"
time="2016-08-10T16:20:18.839325585+08:00" level=info msg="Initializing Libnetwork Agent Local-addr=120.26.43.xxx Adv-addr=120.26.43.xxx Remote-addr =10.105.39.112"
time="2016-08-10T16:20:19.221024670+08:00" level=info msg="6f0191209b75cc6 became follower at term 0"
time="2016-08-10T16:20:19.221236372+08:00" level=info msg="newRaft 6f0191209b75cc6 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"
time="2016-08-10T16:20:19.221353919+08:00" level=info msg="6f0191209b75cc6 became follower at term 1"
time="2016-08-10T16:20:20.175372878+08:00" level=info msg="6f0191209b75cc6 [term: 1] received a MsgHeartbeat message with higher term from 6b3bd82431646f6e [term: 9]"
time="2016-08-10T16:20:20.175632244+08:00" level=info msg="6f0191209b75cc6 became follower at term 9"
time="2016-08-10T16:20:20.175748769+08:00" level=info msg="raft.node: 6f0191209b75cc6 elected leader 6b3bd82431646f6e at term 9"
time="2016-08-10T16:20:28.846854798+08:00" level=error msg="Error in agentJoin : could not join node to memberlist: 1 error(s) occurred:\n\n* Failed to join 10.105.39.112: dial tcp 10.105.39.112:7946: i/o timeout"
time="2016-08-10T16:20:58.844910582+08:00" level=error msg="Error getting node bsgmfnplotqmporr9nmx590nu: rpc error: code = 4 desc = context deadline exceeded"
time="2016-08-10T16:20:58.845162903+08:00" level=error msg="Handler for GET /v1.24/nodes/bsgmfnplotqmporr9nmx590nu returned error: rpc error: code = 4 desc = context deadline exceeded"

It seems host B are trying to estiblish connection to 10.105.39.112 which is the internal IP on host A and is not reachable from B. I init swarm on A with docker swarm init --advertise-addr <internet ip> --listen-addr <internal ip>. and docker swarm join-token manager is:

docker swarm join \
    --token <token>
    10.105.39.112:2377

I changed 10.105.39.112 to <internet IP> and executed join command on host A.

[yoncise]

After init swarm on host A, cat /var/lib/docker/swarm/state.json echos:

[{"node_id":"cxaz16nmfzoaa5q30rmypb15e","addr":"10.105.39.112:2377"}]

which I think should be the advertise addr instead of listen addr. Can I manually change this?

[aaronlehmann]

and docker swarm join-token manager is:

docker swarm join \
    --token <token>
    10.105.39.112:2377

That's definitely unexpected. You should see the public IP address in the output to this command.

When I try this, here's what I see:

# docker swarm init --listen-addr 172.17.0.3 --advertise-addr 1.2.3.4
Swarm initialized: current node (38315wwgylhg8d5lphc8agojr) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join \
    --token <token> \
    1.2.3.4:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

# docker swarm join-token manager
To add a manager to this swarm, run the following command:

    docker swarm join \
    --token <token> \
    1.2.3.4:2377

If we can figure out why you're seeing the private address here, that will probably explain the other problems.

I changed 10.105.39.112 to and executed join command on host A.

Do you mean host B?

[yoncise]

Yeah, I mean host B. Sorry for typo. What can I do for you now?

[yoncise]

Ok I figured out. I init swarm advertise addr with internal ip first time. Then I reinit swarm with force new cluster, however, it seems that if you specify force new cluster docker will ignore new advertise addr

[aaronlehmann]

Ah, was just looking at this when I saw your comment pop up. Glad you figured it out! The issue with --force-new-cluster does sound like an actual bug. I'll put that on my list to look at. It may be difficult to support changing the advertise address with --force-new-cluster because of the way it's implemented, but we could at least disallow the combination of both options.

[yoncise]

Yeah, output some infomation is more friendly. My last question, can I reinit swarm with new advertise addr without shutting down my service? I am running a single node swarm

[aaronlehmann]

Yeah, output some infomation is more friendly. My last question, can I
reinit swarm with new advertise addr without shutting down my service?

Unfortunately it's not possible right now.

[yoncise]

Ok, thanks for your reply

[edesai]

Are there any updates on this?