Compose: docker-compose up: ERROR: Cannot start container: failed to update store for object type *libnetwork.endpointCnt

[jeron-diovis]

Mac OS X 10.10.5

$ docker-compose up
Pulling redis (redis:2.8)...
2.8: Pulling from library/redis

1d53ee5b1800: Pull complete
74872ba2a3b0: Pull complete
b73eb536e646: Pull complete
733ba5d512e3: Pull complete
00b5a1f6aeae: Pull complete
65b0d3a3cd30: Pull complete
a46a54050037: Pull complete
344888176fa7: Pull complete
2380915c0807: Pull complete
f454f845714a: Pull complete
b04e3558e7f0: Pull complete
e25f48bc0233: Pull complete
acaeeb5be8f7: Pull complete
bdccb64d4c06: Pull complete
96288bc75a66: Pull complete
47130e6037f2: Pull complete
ca6553f7ffad: Pull complete
Digest: sha256:2e89addcd3b13e473e4302c202c0171f8e83f4d31af3bbcd8d6c3f68a31fde7c
Status: Downloaded newer image for redis:2.8
Creating rejum_redis_1
ERROR: Cannot start container 6f546315c9ad94337c201b95b22668549d27423b850c366a30de66c46a9c7372: 
failed to update store for object type *libnetwork.endpointCnt: Key not found in store

Didn't find any info about this error, can you explain what this means and what to do?

Do I need to show my docker-compose file?

[dnephin]

It would be good to share your docker-compose.yml to help debug this.

[jeron-diovis]

Here it is:

data:
    image: mediasapiens/python2-jum:1.8
    volumes:
        - .:/web
        - uploads:/uploads
        - static:/static
    links:
        - db:db
    environment:
        - DJANGO_SETTINGS_MODULE=settings.docker
    command: /bin/bash -c 'cd /web && pip install -r requirements.txt && fab run_data'

ask:
    image: mediasapiens/python2-jum:1.8
    volumes_from:
        - data
    links:
        - db:db
        - smtp:smtp
        - redis:redis
    ports:
        - "9900:9000"
    expose:
        - "9000"
    environment:
        - PY_AUTORELOAD=true
        - VENDOR_SLUG=ask
        - DJANGO_SETTINGS_MODULE=settings.docker
    command: /bin/bash -c "cd /web && pip install -r requirements.txt && fab runit"


indola:
    image: mediasapiens/python2-jum:1.8
    volumes_from:
        - ask
    links:
        - db:db
        - smtp:smtp
        - redis:redis
    ports:
        - "9901:9000"
    expose:
        - "9000"
    environment:
        - PY_AUTORELOAD=true
        - VENDOR_SLUG=indola
        - DJANGO_SETTINGS_MODULE=settings.docker
    command: /bin/bash -c "cd /web && pip install -r requirements.txt && fab runit"


celerybeat:
    image: mediasapiens/python2-jum:1.8
    links:
        - redis:redis
        - smtp:smtp
        - db:db
    volumes_from:
        - ask
    environment:
        - DJANGO_SETTINGS_MODULE=settings.docker
    command: /bin/bash -c 'cd /web && pip install -r requirements.txt && fab celerybeat'


celery:
    image: mediasapiens/python2-jum:1.8
    links:
        - redis:redis
        - smtp:smtp
        - db:db
    volumes_from:
        - ask
    environment:
        - DJANGO_SETTINGS_MODULE=settings.docker
        - C_FORCE_ROOT=1
    command: /bin/bash -c 'C_FORCE_ROOT=1 && cd /web && pip install -r requirements.txt && fab celeryd'


redis:
    image: redis:2.8
    expose:
        - "6379"


smtp:
    image: catatnight/postfix
    environment:
        - maildomain=localhost
        - smtp_user=user:pwd


db:
    image:
        mysql:5.6
    environment:
        - MYSQL_ROOT_PASSWORD=test37
    expose:
        - 3306


client:
    image: mediasapiens/npm
    volumes:
        - client:/client
        - static:/static
    command: /bin/bash -c "cd /client && npm install && npm start"

[jeron-diovis]

Actually, I tried to re-create docker VM, and suddenly it helped. Now everything just works, without any additional actions. So, probably, this issue is deprecated and can be closed.

But would be good to know at least what this error means and why it can appear.

[dnephin]

I'm not sure what would cause this, I can't find any other references to the error. I know it's coming from the docker engine, and the error is about a struct from https://github.com/docker/libnetwork, but I'm not sure how it got into that bad state.

[spencerkohan]

I'm running into the same error with docker build trying to build a simple node.js app. Here's the console output:

$ docker build .
Sending build context to Docker daemon 157.2 kB
Step 1 : FROM node:5
 ---> 2647b0da972f
Step 2 : ADD package.json package.json
 ---> Using cache
 ---> a03db49601c5
Step 3 : ADD app.js app.js
 ---> Using cache
 ---> 4b6bfcfbc5be
Step 4 : RUN npm install
 ---> Running in 541bb5e6ac43
failed to update store for object type *libnetwork.endpointCnt: Key not found in store

And here's my Dockerfile:

# this container is based on the node 5 container
FROM node:5
#add the package.json file
ADD package.json package.json
# copy app.js from the local directory to the container's root
ADD app.js app.js
#install npm dependancies
RUN npm install
#expose port 8000
EXPOSE 8000
#execute this command
CMD ["node","app.js"]

I recently updated the tools on my dev machine using the toolkit release 1.9.1:

$ docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

The default machine was generated using toolkit version 1.6.2, I'm not sure if that could be the problem.

@jeron-diovis how did you re-create the docker VM?

edit: Here's something interesting: I've got a project with an identical Dockerfile and identical package.json which does not use docker-compose and it's building fine. @dnephin even if I'm building this image without docker-compose, is it possible there's something cached from running docker-compose that could be causing a problem?

[dnephin]

This error is from the docker engine, I don't think it's related to compose

[raivisdejus]

Had this problem. Update and reboot helped me.

[CharlieR-o-o-t]

It's internal docker daemon error, some key not found in KV store:
./components/engine/vendor/github.com/docker/libkv/store/store.go: ErrKeyNotFound = errors.New("Key not found in store")

Got this on network disconnect api call:
POST networks/net_name/disconnect
data: '{ "Container": "'$CONTAINER_NAME'", "Force": true}'

Unable to reproduce...

[thaJeztah]

@CharlieR-o-o-t looks like in that case an API call was made with literally '$CONTAINER_NAME' for the container name (so the $CONTAINER_NAME env-var was not expanded); possibly due to the single quotes around the environment variable being there?

[CharlieR-o-o-t]

@thaJeztah , nope
look CONTAINER_NAME=test && echo '{ "Container": "'$CONTAINER_NAME'", "Force": true}' { "Container": "test", "Force": true}
single quotes in this case working as string limiter: 'string' $var 'string continue'

The exact error was:
Response code: 404
Docker response: {"message":"endpoint delete failed for container e7bba813fbe44726a49e1b636ce72557ae8cb6bc30adde3109465468aa2e2377 on network my_net: Key not found in store"}

My code is a part of docker_is_alive monitoring trigger. It's working all the time, but sometimes (1-2 per day) results in this error. No any errors in docker.log, and I can't switch to debug mode because it's production :)

[Svtter]

I moved my docker images folder when I faced this problem. If you use Ubuntu, just run sudo apt install --reinstall docker to fix this problem.

[vivekruhela]

Actually, I tried to re-create docker VM, and suddenly it helped. Now everything just works, without any additional actions. So, probably, this issue is deprecated and can be closed.

But would be good to know at least what this error means and why it can appear.

I am facing the same error in Linux/Ubuntu. I tried install --reinstall docker and virtualbox also but the error is still coming. Can you suggest me how to solve this??

[hazeem1991]

still having the same problem and install --reinstall docker did not solve it is there any way that i can fix it ?

[Gisleburt]

For those suffering this issue, I reset my docker _twice_ and it started working.

Your mileage may vary. For this and other reasons, I'm wondering if my SSD is shot although I'm not getting any errors when scanning it. 🤷🏻‍♂️

[marabesi]

More than one year later and I fixed this issue restarting the docker service as @Gisleburt commented.

Its worth mention that I am using the docker service in one partition and pointing the images to another one, here is my daemon.json:

{
    "storage-driver": "overlay",
    "graph": "/mnt/aabfaa12-123s-48ab-a25d-asdas121/docker",
    "cgroup-parent": "/docker_limit.slice"
}

Then I executed: systemctl restart docker and it worked again.