Machine: No connection to Docker deamon on OS X Docker Toolbox

[franz-josef-kaiser]

Issue:

_Docker Compose_ can't connect to deamon.
Standard host via Docker Toolbox (boot2docker) / Docker Quickstart Terminal session
Cannot connect to the docker engine endpoint

$ docker-compose up; docker-machine start default; docker-compose up
ERROR: Couldn't connect to Docker daemon - you might need to run `docker-machine start default`.
Starting "default"...
Machine "default" is already running.
ERROR: Couldn't connect to Docker daemon - you might need to run `docker-machine start default`.

The same goes for _Docker_:

$ docker ps
Cannot connect to the Docker daemon. Is the docker daemon running on this host?

$ docker images
Cannot connect to the Docker daemon. Is the docker daemon running on this host?

$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
default   -        virtualbox   Running   tcp://192.168.99.100:2376           v1.10.2

$ docker-machine ip default
192.168.99.100

$ docker-machine start
Starting "default"...
Machine "default" is already running.

$ docker-machine status default
Running

$ docker-machine active
No active host found

It worked without any problem for weeks, stopped working today. Strange also is that the docker-machine lists default as running machine, but active shows _no_ active machine.

---

Diff of times:

$ docker-machine ssh default

$ time; hwclock; date
BusyBox v1.23.1 (2015-02-22 15:53:49 UTC) multi-call binary.
Tue Mar  8 14:30:13 2016  0.000000 seconds
Tue Mar  8 14:30:14 UTC 2016

It seems the time got stuck on the _Docker Machine_ as seen in #1954 and PR #2006 .

The box itself is "alive" and reachable:

ping -c 3 192.168.99.100
PING 192.168.99.100 (192.168.99.100): 56 data bytes
64 bytes from 192.168.99.100: icmp_seq=0 ttl=64 time=0.293 ms
64 bytes from 192.168.99.100: icmp_seq=1 ttl=64 time=0.321 ms
64 bytes from 192.168.99.100: icmp_seq=2 ttl=64 time=0.315 ms

--- 192.168.99.100 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.293/0.310/0.321/0.012 ms

_Docker Machine_ itself can't connect:

docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
default   -        virtualbox   Running   tcp://192.168.99.100:2376           Unknown   Unable to query docker version: Cannot connect to the docker engine endpoint

Exporting the correct environment variables is not possible:

$ eval $(docker-machine env default)
Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates for host "192.168.99.100:2376": dial tcp 192.168.99.100:2376: getsockopt: connection refused
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
Be advised that this will trigger a Docker daemon restart which will stop running containers.

Checking if the certs are not matching and messed up during provisioning – done for ca.pem, cert.pem

diff ~/.docker/machine/certs/ca.pem ~/.docker/machine/machines/default/ca.pem

Result: All diffs without any result, so no difference there. Also docker-machine ssh default works.

I then ran docker-machine regenerate-certs default to see if that would trigger _anything_ with the following result:

Regenerate TLS machine certs?  Warning: this is irreversible. (y/n): y
Regenerating TLS certificates
Waiting for SSH to be available...
Detecting the provisioner...
Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded

Running now a verbose _Docker Compose_ up call errors out missing file:

docker-compose --verbose up
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.auth.load_config: File doesn't exist
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/compose/compose/cli/main.py", line 55, in main
  File "/compose/compose/cli/docopt_command.py", line 23, in sys_dispatch
  File "/compose/compose/cli/docopt_command.py", line 26, in dispatch
  File "/compose/compose/cli/main.py", line 172, in perform_command
  File "/compose/compose/cli/command.py", line 52, in project_from_options
  File "/compose/compose/cli/command.py", line 85, in get_project
  File "/compose/compose/cli/command.py", line 68, in get_client
  File "/compose/venv/lib/python2.7/site-packages/docker/api/daemon.py", line 78, in version
  File "/compose/venv/lib/python2.7/site-packages/docker/utils/decorators.py", line 47, in inner
  File "/compose/venv/lib/python2.7/site-packages/docker/client.py", line 112, in _get
  File "/compose/venv/lib/python2.7/site-packages/requests/sessions.py", line 477, in get
  File "/compose/venv/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
  File "/compose/venv/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
  File "/compose/venv/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
requests.exceptions.ConnectionError: ('Connection aborted.', error(2, 'No such file or directory'))
docker-compose returned -1

Update: After restarting the computer and opening _Docker Machine_ (Docker Quickstart Terminal), I got the following happening without any user interaction:

bash --login '/Applications/Docker/Docker Quickstart Terminal.app/Contents/Resources/Scripts/start.sh'

Starting "default"...
(default) Check network to re-create if needed...
(default) Waiting for an IP...
Machine "default" was started.
Waiting for SSH to be available...
Detecting the provisioner...
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.
Regenerate TLS machine certs?  Warning: this is irreversible. (y/n): Regenerating TLS certificates
Waiting for SSH to be available...
Detecting the provisioner...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded
Error checking TLS connection: Error checking and/or regenerating the certs: There was an error validating certificates for host "192.168.99.100:2376": dial tcp 192.168.99.100:2376: getsockopt: connection refused
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
Be advised that this will trigger a Docker daemon restart which will stop running containers.

Question:

What file is missing in this error message (from above) that _Docker Compose_ complains about?

docker.auth.auth.load_config: File doesn't exist

---

Details:

OS X 10.11.3

$ docker-machine -v; docker -v; docker-compose -v
docker-machine version 0.6.0, build e27fb87
Docker version 1.10.2, build c3959b1
docker-compose version 1.6.0, build d99cad6

Please advice if there is anything I can do to further debug this one or add additional details.

[franz-josef-kaiser]

Update 2: After reading Docker Compose#1590, I realized that somehow the last update for Docker Toolbox seems to have killed either my ~/.docker/.dockercfg or my ~/.docker/config.json file (whatever currently is in use). To test if that could be the root of the problem, I added an example ~/.docker/.dockercfg file with the only example I could find – from CoreOS docs. It's obviously bogus, but it seems that it boils down to a missing config file, deleted during update.

Example Config:

{
  "quay.io": {
    "auth": "xXxXxXxXxXx=",
    "email": "[email protected]"
  },
  "https://index.docker.io/v1/": {
    "auth": "xXxXxXxXxXx=",
    "email": "[email protected]"
  },
  "https://index.example.com": {
    "auth": "XxXxXxXxXxX=",
    "email": "[email protected]"
  }
}

Now the Error message changes:

$ docker-compose --verbose up -d
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.auth.load_config: Couldn't find 'auths' or 'HttpHeaders' sections
docker.auth.auth.load_config: need more than 1 value to unpack
docker.auth.auth.load_config: Attempting to parse legacy auth file format
docker.auth.auth.load_config: list index out of range
docker.auth.auth.load_config: All parsing attempts failed - returning empty config
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/compose/compose/cli/main.py", line 55, in main
  File "/compose/compose/cli/docopt_command.py", line 23, in sys_dispatch
  File "/compose/compose/cli/docopt_command.py", line 26, in dispatch
  File "/compose/compose/cli/main.py", line 172, in perform_command
  File "/compose/compose/cli/command.py", line 52, in project_from_options
  File "/compose/compose/cli/command.py", line 85, in get_project
  File "/compose/compose/cli/command.py", line 68, in get_client
  File "/compose/venv/lib/python2.7/site-packages/docker/api/daemon.py", line 78, in version
  File "/compose/venv/lib/python2.7/site-packages/docker/utils/decorators.py", line 47, in inner
  File "/compose/venv/lib/python2.7/site-packages/docker/client.py", line 112, in _get
  File "/compose/venv/lib/python2.7/site-packages/requests/sessions.py", line 477, in get
  File "/compose/venv/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
  File "/compose/venv/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
  File "/compose/venv/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
requests.exceptions.ConnectionError: ('Connection aborted.', error(2, 'No such file or directory'))

Question: Is there some Command line function that allows me to regenerate the file? I read that docker login actually should generate that file, but this is not happening for me. Instead I still get the error that the Docker Deamon seems to be not running.

[nathanleclaire]

Hi @franz-josef-kaiser thanks for the issue. docker login will generate that file, but your problem is not missing Docker client config. It's that you can't reach the daemon in the first place. I'm not sure what's up with that Compose error, @aanand @dnephin any ideas ^^ ?

Anyway, first step is to try and connect to the Docker daemon.

• Can you successfully run docker commands when sshed into the VM?
• What does tail -f /var/log/docker.log in the VM say?

Since the VM IP is reachable that (mostly) rules out wonky network config. Sounds to me like the Docker daemon is not starting successfully for some reason.

Another thing to try: docker-machine rm -y default && docker-machine create -d virtualbox default (note: this will destroy the VM and any data on it)

[franz-josef-kaiser]

Hi @nathanleclaire thanks for getting on this.

• [x] Test if SSH-ing into docker-machine works: yes.
• [x] Run docker run hello-world inside docker-machine: no, does not work.

docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.

• [x] Get latest log entries from /var/log/docker.log:

```
time="2016-03-10T12:17:01.172068845Z" level=warning msg="driver error disconnecting container nodejs : endpoint not found: f7bb5e6829f19d9f904e661d095cd725f3f31453262056e43a85e0e84c4d86b8"
time="2016-03-10T12:17:01.175755931Z" level=warning msg="driver error deleting endpoint nodejs : endpoint not found: f7bb5e6829f19d9f904e661d095cd725f3f31453262056e43a85e0e84c4d86b8"
time="2016-03-10T12:17:01.175778422Z" level=debug msg="Releasing addresses for endpoint nodejs's interface on network front"
time="2016-03-10T12:17:01.175792222Z" level=debug msg="ReleaseAddress(LocalDefault/172.19.0.0/16, 172.19.0.2)"

---

/usr/local/bin/docker daemon -D -g "/var/lib/docker" -H unix:// -H tcp://0.0.0.0:2376
--label provider=virtualbox

--tlsverify --tlscacert=/var/lib/boot2docker/ca.pem --tlscert=/var/lib/boot2docker/server.pem --tlskey=/var/lib/boot2docker/server-key.pem -s aufs >> "/var/lib/boot2docker/docker.log"
time="2016-03-10T12:17:01.881371546Z" level=fatal msg="Error starting daemon: pid file found, ensure docker is not running or delete /var/run/docker.pid"
```

• [x] Try destroying and rebuilding the default VM: no, does not change anything

```
$ docker-machine rm -y default && docker-machine create -d virtualbox default
About to remove default
Successfully removed default
Running pre-create checks...
…
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!

$ docker-machine status
Running
$ docker-machine active
No active host found
$ docker-compose up
ERROR: Couldn't connect to Docker daemon - you might need to run docker-machine start default.

$ docker-machine ssh > tail -f /var/log/docker.log
time="2016-03-10T12:26:26.267396805Z" level=debug msg="Registering GET, /volumes"
time="2016-03-10T12:26:26.267443833Z" level=debug msg="Registering GET, /volumes/{name:.}"
time="2016-03-10T12:26:26.267520416Z" level=debug msg="Registering POST, /volumes/create"
time="2016-03-10T12:26:26.267582984Z" level=debug msg="Registering DELETE, /volumes/{name:.}"
time="2016-03-10T12:26:26.267666438Z" level=debug msg="Registering POST, /build"
time="2016-03-10T12:26:26.267739206Z" level=info msg="API listen on [::]:2376"
time="2016-03-10T12:26:26.267789720Z" level=info msg="API listen on /var/run/docker.sock"
2016/03/10 12:26:29 http: TLS handshake error from 192.168.99.1:49785: EOF
time="2016-03-10T12:27:06.512878162Z" level=debug msg="Calling GET /v1.15/version"
time="2016-03-10T12:27:06.512926482Z" level=debug msg="GET /v1.15/version"
```

Before destroying and rebuilding the VM, I dumped the complete contents of the docker.log file in the console. The TLS handshake error can be found multiple times – even in times where the VM was supposed to work. It has two variations: One is like above with EOF. The other one is (port and timestamp are obviously different):

2016/01/31 13:08:41 http: TLS handshake error from 192.168.99.1:65286: tls: first record does not look like a TLS handshake

[nathanleclaire]

@franz-josef-kaiser, In the last block posted you did not eval $(docker-machine env) (implicit VM name is default) before running docker-compose up. It makes sense the Docker Compose cannot connect in that case, since this is where Compose reads its connection information from. If you set those environment variables does it work?

[franz-josef-kaiser]

@nathanleclaire That was a good catch. And that solved it!

Can I still be of some help backtracing whatever caused this? Having a fix for the stack is nice, but it might hit others as well and in case I can I am happy to help with getting to the root of this.

[nathanleclaire]

@franz-josef-kaiser Unfortunately I think it's gone to the wind now, but if you encounter any more issues, feel free to open more tickets. Thanks!

[franz-josef-kaiser]

@nathanleclaire I guess you can reopen the issue … just had to repeat the destroying/creating the VM and sourcing the ENV again after a restart.

$ docker-machine rm -y default && docker-machine create -d virtualbox default && eval $(docker-machine env)

[franz-josef-kaiser]

@nathanleclaire and again it happened.

[franz-josef-kaiser]

This keeps happening over and over again…

[nathanleclaire]

@franz-josef-kaiser Please post --debug output or open a new issue with more details. Thanks

[saikojosh]

I have the same issue on OSX. Tried a few other things but @franz-josef-kaiser's command docker-machine rm -y default && docker-machine create -d virtualbox default && eval $(docker-machine env) fixed my problem for now.

[naoko]

All I needed was to run eval $(docker-machine env).
maybe inserting this line to bash_profile for mac user might be helpful?

[axlroden]

tried eval $(docker-machine env) first .. didn't work.

Had to recreate docker machine first.

[brandontamm]

@nathanleclaire - Hopefully I can shed some light - I think I have a use case that these guys probably are also having...

I created a docker-machine with digital ocean driver with the smallest droplet size* Why? because DO only lets you scale up or down if you keep the same storage space, so I start small and then manually go into DO and scale my server up. At the same time, I manually install docker-compose on DO droplet, which upgrades docker package on server.

I think the docker upgrade triggered from the docker-compose installation is what is breaking the link.

docker-machine ssh xxxx works, but no other docker-machine commands seem to work after this particular use case

[Stig124]

To verify that the env are correctly set go there and follow that It should work : Docker Environement Variables help