Moby: An error occurred trying to connect version 1.9.0

[chiavegatto]

Execute command:

docker ps -a

Error:

An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/containers/json?all=1: read unix /var/run/docker.sock: connection reset by peer

I'm using ubuntu and updated version 1.8.3 to 1.9.0.

docker version
Client:
Version: 1.9.0
API version: 1.21
Go version: go1.4.2
Git commit: 76d6bc9
Built: Tue Nov 3 17:43:42 UTC 2015
OS/Arch: linux/amd64
An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/version: read unix /var/run/docker.sock: connection reset by peer

[GordonTheTurtle]

Hi!

Please read this important information about creating issues.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

This is an automated, informational response.

Thank you.

For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

---

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:
uname -a:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

ENEEDMOREINFO

[chiavegatto]

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version: 1.9.0
docker info: An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/info: read unix /var/run/docker.sock: connection reset by peer
uname -a: Linux lab13-rafael 3.16.0-52-generic #71~14.04.1-Ubuntu SMP Fri Oct 23 17:24:53 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:

1. Updated version 1.8.3 to 1.9.0
2. Execute command docker ps -a or docker info or docker version

Describe the results you received:
Error:

An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/containers/json?all=1: read unix /var/run/docker.sock: connection reset by peer

----------END REPORT ---------

[chiavegatto]

These commands solved my problem:

If you want to make progress the following can be used as a workaround:

1. Stop docker daemon
2. sudo rm /var/lib/docker/network/files/local-kv.db
3. Start docker daemon

[kesopeso]

I had the same problem. Your solution helped me too. Thank you.

[thaJeztah]

ping @mrjana (I also saw the local-kv.db mentioned in https://github.com/docker/docker/issues/17760)

[kevin125]

@chiavegatto Thx, it works!

[mavenugo]

@chiavegatto @kevin125 @kesopeso by any chance you tried docker daemon from the master or RCs before doing the upgrade from 1.8 to 1.9 ?

[mrjana]

@chiavegatto @kevin125 @kesopeso Any chance, you have the docker logs during the time this happened. docker apparently panicked and if my guess is correct you hit https://github.com/docker/docker/pull/17639. If that is case, you can be assured that it will be available in 1.9.1

[ehohenstein-imvu]

I've found that the solution posted by @chiavegatto works for me as well.

I can reproduce this problem very easily. The following 3 line dockerfile causes this problem to occur on my system every time I try to build it:

FROM ubuntu
MAINTAINER test
RUN useradd testuser

Here is the output:

$ docker build -t test_img .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu
---> e9ae3c220b23
Step 2 : MAINTAINER test
---> Using cache
---> 6346b025a5ae
Step 3 : RUN useradd testuser
---> Running in c29f6f768b8f
unexpected EOF

After experiencing this problem, all attempts to interact with the docker daemon fail:

$ docker ps
An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/containers/json: read unix /var/run/docker.sock: connection reset by peer

$ docker version
Client:
Version: 1.9.0
API version: 1.21
Go version: go1.4.2
Git commit: 76d6bc9
Built: Tue Nov 3 17:40:31 UTC 2015
OS/Arch: linux/amd64
An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/version: read unix /var/run/docker.sock: connection reset by peer

$ docker info
An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/info: read unix /var/run/docker.sock: connection reset by peer

$ uname -a
Linux precise-sandbox 3.19.0-32-generic #37~14.04.1 SMP Fri Nov 6 00:01:52 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ dpkg -l | grep docker
ii docker-engine 1.9.0-0~precise Docker: the open-source application container engine

When building the following output is written to /var/log/syslog:

Nov 12 18:24:11 precise-sandbox kernel: [26947.901793] aufs au_opts_verify:1612:docker[30411]: dirperm1 breaks the protection by the permission bits on the lower branch
Nov 12 18:24:11 precise-sandbox kernel: [26947.915139] aufs au_opts_verify:1612:docker[30411]: dirperm1 breaks the protection by the permission bits on the lower branch
Nov 12 18:24:11 precise-sandbox kernel: [26947.930973] aufs au_opts_verify:1612:docker[30411]: dirperm1 breaks the protection by the permission bits on the lower branch
Nov 12 18:24:11 precise-sandbox kernel: [26947.937694] aufs au_opts_verify:1612:docker[30411]: dirperm1 breaks the protection by the permission bits on the lower branch
Nov 12 18:24:11 precise-sandbox kernel: [26947.945493] aufs au_opts_verify:1612:docker[30411]: dirperm1 breaks the protection by the permission bits on the lower branch

[ehohenstein-imvu]

This problem was happening for me continually but I've been able to resolve it.

I tried uninstalling (purging) the docker-engine package and then re-installing it. The problem remained.

I tried uninstalling the docker-engine package and then deleting the /var/lib/docker directory (this required a reboot). After re-installing the docker-engine package, everything seems to work as expected. It seems that the docker daemon had some invalid/corrupted data on disk that was preventing it from working correctly.

[scher200]

Thank you, chiavegatto
this worked for me too:

1. Stop docker daemon
2. sudo rm /var/lib/docker/network/files/local-kv.db
3. Start docker daemon

[saurabhnsonar]

Not running for me too, tried every possible solution:

Client:
Version: 1.9.1
API version: 1.21
Go version: go1.4.2
Git commit: a34a1d5
Built: Fri Nov 20 13:12:04 UTC 2015
OS/Arch: linux/amd64
An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/version: read unix /var/run/docker.sock: connection reset by peer

uname-a:
Linux portal 2.6.32-042stab102.9 #1 SMP Fri Dec 19 20:34:40 MSK 2014 x86_64 x86_64 x86_64 GNU/Linux

docker info:
An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/info: read unix /var/run/docker.sock: connection reset by peer

[thaJeztah]

@saurabhnsonar kernel 2.6 is too old to be running docker, and not supported, so docker failing on that is probably expected

[gilo]

@thaJeztah I am getting the same error with docker 1.9.1 on ubuntu 14.04 following a system restart. The workaround works for me but I need to automate it to happen after restart.

docker version 1.9.1, build a34a1d5
Linux testim-docker-01 3.13.0-68-generic #111-Ubuntu SMP Fri Nov 6 18:17:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[thaJeztah]

@gilo it may be that the daemon didn't shut down clean because of that restart, resulting in some corruption in the database. I think @aboch made changes in libnetwork to better recover from those situations. If you encounter that situation, could you backup the old local-kv.db file, so that we can analyze it?

[gilo]

@thaJeztah I will try to backup the file on the next encounter. btw, since the workaround here is about deleting it... is it a good practice to remove it at reboot (a cron job)?

[thaJeztah]

@gilo that shouldn't be needed (perhaps as a temporary workaround though), wondering why it would get corrupt on a regular reboot.

[zxlzh2015]

@chiavegatto you solve my problem,thanks

[muddman]

thanks @chiavegatto -- that worked for me too on Docker version 1.9.0, build 76d6bc9. I had the problem on ubuntu 14.04 after a clean reboot.

[rivkahcarl]

@chiavegatto thank you so much! I just had the same problem and that worked perfectly!

[tocttou]

I am also facing the same problem on Docker version 1.9.0 on Ubuntu 14.04. I have some containers that need to be up 24x7 but they stop working at least once a week. @chiavegatto 's workaround solved my problem but it keeps repeating.

[bgokden]

I can confirm the @chiavegatto's solution works on Docker version 1.9.1 on Ubuntu 14.04. Thank you.

[muddman]

Thanks @scher200 your solution worked for me on docker 1.9.1 ubuntu 14.04. cheers.

[thaJeztah]

Is anyone still encountering this on docker 1.10? Otherwise I think we should close this

[mlaitinen]

Occurred to me on 1.10.1 build 9e83765:

$ docker ps
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
$ sudo service docker start
docker start/running, process 5906
$ docker ps
An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.22/containers/json: read unix @->/var/run/docker.sock: read: connection reset by peer

Stopping the daemon, removing local-kv.db and starting daemon didn't help.

[thaJeztah]

@mlaitinen was that on a fresh 1.10.1 install, or an upgrade from 1.9.x ?

[mlaitinen]

An upgrade from 1.9.x. Unfortunately I don't have any more details because I removed docker-engine and /var/lib/docker which for some weird reason caused the deletion of nearly entire /var/lib.

[daaain]

In case someone else runs into the same thing (I found this issue via Googling An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.22/containers/json: read unix @->/var/run/docker.sock: read: connection reset by peer), on upgrading to 1.10.1 from 1.9.x on Ubuntu 15.10 I had the same error message and the problem was in /etc/systemd/system/docker.service which used -d instead of daemon so just had to edit that.

[ctammes]

Just happened to me, using 1.10.1. Yesterday, no problems, I could execute 'docker images' and 'docker ps' without errors. This was an upgrade from 1.9, together with OS update, a couple of days ago.
Error:

An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.22/version: read unix @->/var/run/docker.sock: read: connection reset by peer

OS information:

3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

After 'sudo rm /var/lib/docker/network/files/local-kv.db' it worked again. Then I found out that my network 'app' wasn't there anymore and containers were using it. So, maybe that might have been the problem.

Client:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

[Guisardo]

Removing the local-kv.db worked for me too

[patrick-mota]

Having trouble in Debian 8, just installed docker using docker-machine create generic... and I have the same problem as @daaain
so like he said just edit:
/etc/systemd/system/docker.service and change -d to daemon
after that
systemctl daemon-reload
sudo service docker restart
docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

[irvifa]

@patrick-mota thanks your solution also works on arch linux

[vskubriev]

To @scher200. Thank you for a tip.

Removing the local-kv.db worked for me too

[Guisardo]

FYI: After restarting my ubuntu, because of some system updates, the same problem came back. I'd to manually delete the local-kv.db again to make docker work again.

[LM1LC3N7]

sudo rm /var/lib/docker/network/files/local-kv.db

You've saved my life !

Thanks !! :)

[tjm-jimmy]

@chiavegatto Your solution saved me hours of research thank you very much.

[progre55]

Had the same error after upgrading to 1.10.3.
However, the removing local-kv.db solution did not help at all.

Checking the logs in /var/log/upstart/docker.log, I realized that my partition that holds the docker data dir (I use overlay and have a separate partition for /var/lib/docker) has run out of space. Specifically, inode space, even though there is enough disk space:

could not calculate checksum for "addfbe2ad2fa379369b1dd7b5a127396309ed3a9e1e20be918d322fd5a457e02", "open /var/lib/docker/graph/addfbe2ad2fa379369b1dd7b5a127396309ed3a9e1e20be918d322fd5a457e02/.migration-size: no space left on device"

:~# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/vgdocker-data      50G   21G   27G  44% /var/lib/docker

:~# df -i
Filesystem                    Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vgdocker-data    3276800 3276800       0  100% /var/lib/docker

So, I had to stop docker, delete the entire docker directory, and start it again.

Question: has docker started consuming more inodes (using more smaller files)?
Again, I use the overlay storage driver.

[cpuguy83]

@progre55 overlay consumes lots of inodes compared to other drivers.

[flaccid]

We ran into this on Docker version 1.10.2, build c3959b1 using btrfs.
As btrfs has no inodes, it seems to be a falsey and removing local-kv.db did not help.
df -i:

/dev/xvdc           0     0     0     - /var/lib/docker

df -h:

/dev/xvdc        20G   17G  3.4G  83% /var/lib/docker

Occurred on a fully updated RHEL 7.2 (AMI ID ami-e0c19f83). The exact log message from the daemon:

msg="Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to get bridge network configurations from store: error while populating kmap: open /var/lib/docker/network/files/local-kv.db: no space left on device"

[MaxBer]

Had this on 1.10.3 on ubuntu 16.04. removing local-kv.db helped.

[fogshot]

I get this error on my VServer hosted by Strato, the suggested workaround, however does NOT work for me. See available system information below:

$ docker version
Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:47:50 2016
 OS/Arch:      linux/amd64
An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.23/version: read unix @->/var/run/docker.sock: read: connection reset by peer

$ uname -a
Linux hxxxxxxx.stratoserver.net 3.13.0-042stab113.17 #1 SMP Wed Feb 10 18:31:00 MSK 2016 x86_64 x86_64 x86_64 GNU/Linux

[whitecolor]

I have the same problem with ubuntu 14.4 and I have docker 1.12
and I have no directories /var/lib/docker/network ( /var/lib/docker/network/files/local-kv.db)

any advice ho to run it?

[scher200]

Hi Alex,

I am also on ubuntu 14.04.3 and Docker 1.12 and my local-kv.db is at:
/var/lib/docker/network/files/
Maybe you could do a:
sudo cat /etc/_release_
and report the output here

[FumiakiKato]

Hi scher200 and Alex,

I have the same problem on ubuntu 14.04.3 and Docker 1.12.

An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json: read unix @->/var/run/docker.sock: read: connection reset by peer

The result of sudo cat /etc/*release* is as follows:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS"
NAME="Ubuntu"
VERSION="14.04.3 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.3 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

[scher200]

I am sorry I can't replicate this now with the exact same version of
ubuntu, maybe some DockerPeoples around who can an clearify this.

2016-08-08 14:17 GMT+02:00 FumiakiKato [email protected]:

Hi scher200 and Alex,

I have the same problem on ubuntu 14.04.3 and Docker 1.12.

An error occurred trying to connect: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json:
read unix @->/var/run/docker.sock: read: connection reset by peer

The result of sudo cat /etc/_release_ is as follows:

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS"
NAME="Ubuntu"
VERSION="14.04.3 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.3 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/17846#issuecomment-238218459,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJG6XS6Rsc5IfLjcKr_ulOh7FbDjQ0f7ks5qdx5ogaJpZM4GfBWT
.

[epaminond]

Get the same issue on my server. I could provide credentials for someone to debug it if needed.

[whitecolor]

@scher200
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS"
NAME="Ubuntu"
VERSION="14.04.3 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.3 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

If it helps)

I've no /var/lib/docker folder, but docker service is running

I believe the problem that it is on OpenVZ

[carlosmmelo]

any update on this? I'm getting this error on v1.12 and CentOS 6

[thaJeztah]

@carlosmmelo CentOS 6 is not a supported platform for Docker for over a year

[scher200]

yeah maybe, please add your errors, i think you'll help others too

2016-08-10 20:43 GMT+02:00 Alex [email protected]:

@scher200 https://github.com/scher200
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.3 LTS"
NAME="Ubuntu"
VERSION="14.04.3 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.3 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

If it helps)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker/issues/17846#issuecomment-238963260,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJG6Xe2f3ik7pVxMTQq0vJlMXlk3jaLMks5qehvtgaJpZM4GfBWT
.

[danorton-cubic-austin]

I was configured for aufs and was using a new system image, but forgot this step when creating the image:

apt-get install linux-image-extra-$(uname -r)

[epaminond]

I've also investigated this issue and it seems it's coming from system using old 2.6.x kernel.

[raisercostin]

I was able to fix it by running (see kenorb response http://stackoverflow.com/a/34887761/99248)

sudo service docker stop
sudo rm -rf /var/lib/docker
sudo service docker start 

[cwen-coder]

@raisercostin your solution worked for me on docker 1.12.1. ubuntu 14.04. I run ubuntu 14.04 with VirtalBox and Vagrant . thanks

[thaJeztah]

I'm closing this issue, because this has become a collection of multiple issues (most related to either a corrupted file, or an incomplete installation/upgrade), resulting in the docker daemon being unable to start. (Removing /var/lib/docker is basically a "factory reset" (all docker data is removed)).

If you run into a similar issue, please check / provide relevant daemon logs as well as those may give more information why the daemon is unable to start, and open a new issue if you suspect there's a bug.

[genggjh]

apt-get install linux-image-extra-$(uname -r)
This works for me.

[BrittMcDanel]

@genggjh Thanks this fixed it for me.

Doing:

sudo service docker stop
sudo apt-get install linux-image-extra-$(uname -r)
sudo service docker start 

[evandrousa]

Docker was working and then after not using for a while, it stopped working and gave me this same issue. I am using Ubuntu 14.04. Removing the local-kb.db file did not fix the issue.

FIXED! The issued seems to come from a Kernel upgrade I did(upgraded from 4.4.0 to 4.8.6). Docker worked again rebooting and selecting the previous kernel in GRUB(Kernel 4.4.0)

[mbylstra]

In my case I had to delete /etc/docker/key.json as suggested in https://github.com/docker/docker/issues/23985

It looks like there are a number of different issues that might be causing the following error message:

An error occurred trying to connect: Get http:///var/run/docker.sock/v1.21/containers/json?all=1: read unix /var/run/docker.sock: connection reset by peer

What makes this difficult to solve is that multiple different problems result in the same error message. There is a better way to troubleshoot.

As suggested by @caffeineaddiction in https://github.com/docker/docker/issues/20221, run the docker daemon manually to be able to view the error messages.

sudo service docker stop
sudo /usr/bin/docker daemon

You should get an error message which will help you google for a more specific solution (rather than randomly trying all suggested fixes in this thread)

I'd already done apt-get --purge docker-engine and deleted the entire /var/lib/docker directory, so I'm not sure if those steps are necessary for solving the particular corrupted key.json issue (probably not)

[dabianzhixing]

In my case, I have ever used the overlay network. so I have a configuration in /etc/default/docker that related to the network card eth1. I have ifdown the eth1 by mistake then I get the same error. After I ifup the network card eth1. All thing is ok.

[vigneras]

Launching the daemon from the command line helped a lot. Thanx @mbylstra.
In my case, the problem was:
FATA[0000] open /var/run/docker/libcontainerd/containerd/ddb2c3058fcdd9567d5ca97aacea2b3254ef07d5aa849d6dbfcdfe0548445567/state.json: no such file or directory

Doing: rm -rf /var/run/docker, solved the problem (but I didn't have images yet...)

[libp]

sorry ,我最后选择了重装docker....

[aarongorka]

Had to delete ~/.docker/config.json, possibly something to do with ECR.

[DMW007]

@chiavegatto Not a suiteable solution for me since all configuration like networks got lost, and I've to restart all containers. But what worked for me: Moving the local-kv.db file (e.g. local-kv.db.old), then restarting the docker-daemon. Now a new database is created. Then stop docker-daemon, delete the new one, and move the local-kv.db.old back to local-kv.db. Then start the daemon again and all is working again.

I have no clue what happens here and why... Using 17.05.0-ce, build 89658be. But it seems that something swallowed up on docker and this workaround can fix it. Never had this before, will keep an eye on it if it ever happens aganin.

[leandrorcombinato]

Hi, my name is Leandro Combinato

Follow solution bellow:

cd /var/run/
sudo chmod -R 777 docker.sock

Thank very much!