Swarmkit: docker daemon fails to restart after unsuccessful swarm join and reboot

[inetdavid]

Environment: Docker 1.13.1, macOS 10.12.3 (Sierra), Docker Toolbox

Using default Virtualbox VM I tried to join a swarm manager on another host, which failed.

Docker continued to run on VM fine for days afterwards, but after a reboot of the VM docker-machine was unable to connect:

$ dm ls
NAME              ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
default           -        virtualbox   Running   tcp://192.168.99.100:2376           Unknown   Unable to query docker version: Cannot connect to the docker engine endpoint

provision and regenerate-certs failed, but I was able to ssh into default with "docker-machine ssh default". "sudo /etc/init.d/docker status" showed the Docker daemon was stopped. Daemon restart failed with:

# /etc/init.d/docker start
Need TLS certs for default,127.0.0.1,10.0.2.15,192.168.99.100
-------------------

/var/log/docker.log showed:

time="2017-02-24T16:57:52.507926112Z" level=error msg="cluster exited with error: can't initialize raft node: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate signed by unknown authority (possibly because of \\\"x509: ECDSA verification failure\\\" while trying to verify candidate authority certificate \\\"swarm-ca\\\")\"" 
time="2017-02-24T16:57:52.507951325Z" level=fatal msg="Error creating cluster component: swarm component could not be started: can't initialize raft node: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate signed by unknown authority (possibly because of \\\"x509: ECDSA verification failure\\\" while trying to verify candidate authority certificate \\\"swarm-ca\\\")\"" 

With those clues I eventually found /var/lib/docker/swarm directory and once I renamed the two .json files docker was able to restart successfully.

root@default:/mnt/sda1/var/lib/docker/swarm# ls -l
total 20
drwxr-xr-x    2 root     root          4096 Feb 21 20:40 certificates
-rw-------    1 root     root            98 Feb 24 17:53 docker-state.json
drwx------    3 root     root          4096 Feb 21 20:40 raft
-rw-------    1 root     root            32 Feb 23 23:30 state.json
drwxr-xr-x    2 root     root          4096 Feb 21 20:40 worker
root@default:/mnt/sda1/var/lib/docker/swarm# mv docker-state.json docker-state-OLD.json
root@default:/mnt/sda1/var/lib/docker/swarm# mv state.json state-OLD.json
root@default:/mnt/sda1/var/lib/docker/swarm# /etc/init.d/docker status
Docker daemon is not running
root@default:/mnt/sda1/var/lib/docker/swarm# /etc/init.d/docker start
Need TLS certs for default,127.0.0.1,10.0.2.15,192.168.99.100
-------------------
root@default:/mnt/sda1/var/lib/docker/swarm# /etc/init.d/docker status
Docker daemon is running

Since swarm isn't required for the daemon to run, perhaps it would be better for the daemon to NOT exit so that docker-machine can reconnect? Then the user could run Docker commands and see that the swarm wasn't able to connect, etc. I doubt most users could diagnose this kind of issue and reading through some other issues here the most common advice is to just remove the VM and re-create it, which can end up having a lot more impact.

[aaronlehmann]

ping @cyli: Any idea what would cause that signature validation error?

[cyli]

@aaronlehmann I think I saw a similar error in https://github.com/docker/docker/issues/30393#issuecomment-279122611, except in that case it was trying to join a new swarm cluster when it had the previous cluster's root CA.

I'm not sure why it would even try to connect to any node, since if the join failed, that should mean that it didn't have any node credentials and on restart it should not attempt to reconnect to any node.

@inetdavid What were the contents of /var/lib/docker/swarm/certificates, if you still have that directory handy?

[inetdavid]

I don’t have that directory any more. My issue is that I’m working through some Proof of Concept (PoC) code and the swarm manager on another server isn’t available any longer. But rather than communicate that fact to me, Docker on my local VM just dies, leaving a cryptic log message. When using docker-machine most every command fails because the Docker daemon is no longer running.

The most common advice for when this happens is to just delete and rebuild the VM from scratch, which is drastic.

Docker shouldn’t die just because the swarm info is wrong or because it can’t validate the swarm CA, and if the daemon continues to execute it can then return clear error messages to docker-machine commands.

David Schmidt
[email protected] [e-mail]
509-869-0928 (cell, text messages preferred)

On Feb 27, 2017, at 9:52 AM, Ying Li [e-mail] wrote:

@aaronlehmann https://github.com/aaronlehmann I think I saw a similar error in docker/docker#30393 (comment) https://github.com/docker/docker/issues/30393#issuecomment-279122611, except in that case it was trying to join a new swarm cluster when it had the previous cluster's root CA.

I'm not sure why it would even try to connect to any node, since if the join failed, that should mean that it didn't have any node credentials and on restart it should not attempt to reconnect to any node.

@inetdavid https://github.com/inetdavid What were the contents of /var/lib/docker/swarm/certificates, if you still have that directory handy?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/docker/swarmkit/issues/1985#issuecomment-282796467, or mute the thread https://github.com/notifications/unsubscribe-auth/AAA-RN51RFuY22FgLlyz4bcAZyQmnxzFks5rgw10gaJpZM4MLh6T.

[cyli]

As an update to this issue, I've managed to replicate in the following manner (I don't necessarily disagree with the statement about the docker daemon dying - this is purely an informational comment for others looking to replicate or debug):

1. docker swarm join a cluster successfully as a manager, then kill the other nodes in the cluster and bring them back up with a new CA - the manager in question that wasn't restarted came back up successfully and just fail to connect with the other nodes. Shows that error message, but docker engine starts up correctly.

2. Wipe the /var/lib/docker/swarm/raft directory, but leave all the other swarm directories:

   /var/lib/docker/swarm
   ├── certificates
   │   ├── swarm-node.crt
   │   ├── swarm-node.key
   │   └── swarm-root-ca.crt
   ├── docker-state.json
   ├── state.json
   └── worker
       └── tasks.db
   

3. Then restart the docker engine again. It detects no raft state, so attempts to rejoin the other cluster, but fails because the other cluster has a different root CA:

   Feb 28 22:52:27 cyli-test3 dockerd[27804]: time="2017-02-28T22:52:27.526951432Z" level=info msg="Loading containers: done."
   Feb 28 22:52:27 cyli-test3 dockerd[27804]: time="2017-02-28T22:52:27.617227716Z" level=debug msg="loaded CA and TLS certificates" module=node
   Feb 28 22:52:27 cyli-test3 dockerd[27804]: time="2017-02-28T22:52:27.619808595Z" level=debug msg="next certificate renewal scheduled" module="node/tls" node.id=hgnfovcp05t0a5ijwzlkiqqrz node.role=swarm-manager time=2017-04-14 19:12:00.000000487 +0000 UTC
   Feb 28 22:52:27 cyli-test3 dockerd[27804]: time="2017-02-28T22:52:27.628096414Z" level=debug msg="(*Agent).run" module="node/agent"
   Feb 28 22:52:27 cyli-test3 dockerd[27804]: time="2017-02-28T22:52:27.632430500Z" level=info msg="Listening for local connections" addr="/var/run/docker/swarm/control.sock" proto=unix
   Feb 28 22:52:27 cyli-test3 dockerd[27804]: time="2017-02-28T22:52:27.632980899Z" level=info msg="Listening for connections" addr="[::]:2377" proto=tcp
   Feb 28 22:52:27 cyli-test3 dockerd[27804]: time="2017-02-28T22:52:27.647651071Z" level=info msg="shuting down certificate renewal routine" module="node/tls" node.id=hgnfovcp05t0a5ijwzlkiqqrz node.role=swarm-manager
   Feb 28 22:52:27 cyli-test3 dockerd[27804]: time="2017-02-28T22:52:27.649627154Z" level=debug msg="(*Agent).run exited" module="node/agent"
   Feb 28 22:52:27 cyli-test3 dockerd[27804]: time="2017-02-28T22:52:27.650221167Z" level=error msg="cluster exited with error: can't initialize raft node: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate signed by unknown authority (possibly because of \\\"x509: ECDSA verification failure\\\" while trying to verify candidate authority certificate \\\"swarm-ca\\\")\""
   Feb 28 22:52:27 cyli-test3 dockerd[27804]: time="2017-02-28T22:52:27.650720467Z" level=fatal msg="Error creating cluster component: swarm component could not be started: can't initialize raft node: rpc error: code = 13 desc = connection error: desc = \"transport: x509: certificate signed by unknown authority (possibly because of \\\"x509: ECDSA verification failure\\\" while trying to verify candidate authority certificate \\\"swarm-ca\\\")\""
   Feb 28 22:52:27 cyli-test3 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
   

   Feb 28 22:52:27 cyli-test3 systemd[1]: docker.service: Unit entered failed state.
   Feb 28 22:52:27 cyli-test3 systemd[1]: docker.service: Failed with result 'exit-code'.

So maybe the swarm node got into an inconsistent state somehow (having certificates, and existing state, but no readable raft data).

If you don't particularly care about the node being a member of the swarm, you can also wipe out the entire /var/lib/docker/swarm directory and restart.