Compose: docker-compose up fails if network attached to container is removed

[adambro]

Container start fails, because network it was attached to has been removed. For some reason docker-compose tries to remove that container attached network first, which results with an error.

Steps to reproduce:

docker-compose up
docker network rm dockercomposenetworkrmfails_default
docker-compose up

[adambro]

The confusing part is: docker-compose created a new network first, but then tried to re-start already existing container attached to removed network. I'd expect it to use this newly created network instead. I've figured out what's happening only after comparing network Id hash, because network name was the same (generated automatically).

Maybe compose can try fail-safe network rm instead?

Minimal docker-compose.yml to replicate:

version: '3'

services:
  container:
    image: rwgrim/docker-noop

[shin-]

Thanks for the report! I think there are several things to note here:

1. First and foremost, you shouldn't delete networks managed by Compose unless you're doing through Compose (typically with docker-compose down). Outside of this particular problem, it will certainly lead to a slew of other issues down the line.
2. Containers always connect to networks using the network ID which is guaranteed to be unique, whereas no such guarantee exists for names.
3. Your container is being restarted because no changes in configuration were detected, and the recreation of the network is considered to be an independent event. You could actually workaround the issue by using the --force-recreate flag.
4. There's a case to be made for checking the network containers will connect to still exists, and reconnecting to the new network if necessary, but see item 1: this probably shouldn't happen in the first place, and I think raising an error is equally valid in this instance.

[adambro]

Thanks for detailed explanation. Indeed, it happened because I've run docker network prune some time before.

What I don't get still is why it tries to remove Network that does not exist any more? I guess that caused "not found" error. I haven't found any other instance of such error message.

[shin-]

It's not due to a removal attempt. Compose tells the engine to start , which the engine attempts and recognizes that the network it is configured to be connected to doesn't exist anymore, preventing the operation from succeeding. Compose simply reports that error.

[adambro]

You're right @shin- it's a message from failed container start command:

ERROR: for container  Cannot start service container: network 8a131ba522472461e38b5189f5fd489b22d4d7a850a6dcd64a227ceb687c35a6 not found

I agree with your points since I better understood how compose tries to bring up old containers instead of creating new ones every time. Thanks for explanation, appreciate that. It seems to be that way by design, so nothing more to do here I guess.

[dazinator]

It would be nice to have some "recreate if necessary" flag. I.e we want to ensure docker-compose up does whatever it takes to bring the environment up and if that means it needs to create a network then so be it!

[jlyman]

I'd second the suggestion from @dazinator for a "recreate if necessary" flag. I ran a docker network prune command just like @adambro, and since I run several different Docker setups for different projects, I didn't realize that pruning while working on one large project was going to negatively affect the others. (My fault for not reading and thinking through the warning from network prune more when I did so.) But a flag like that would be a nice way of getting a project back on its feet quickly.

[DonAurelio]

I was running the ubuntu version of docker holded in this respository (docker.io 18.06.0). I removed that version an install docker.io using apt-get install docker.io. So now I have this version:

Client:
Version: 17.12.1-ce
API version: 1.35
Go version: go1.10.1
Git commit: 7390fc6
Built: Wed Apr 18 01:23:11 2018
OS/Arch: linux/amd64

Server:
Engine:
Version: 17.12.1-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.10.1
Git commit: 7390fc6
Built: Wed Feb 28 17:46:05 2018
OS/Arch: linux/amd64
Experimental: false

Containers running with docker-compose doesn't have the file resolv.conf with the dns consiured as expected, but, each container can access Internet. I think that change solve mi problem.

[theilgaard]

How are we to handle this on a host crash? The containers will not start back up again as a non-existent network is configured for docker-compose.

[riteshcusat08]

@shin-

I am running docker container on linux machine.
Docker service is running.
I start the container with “docker-compose up -d ” command and it starts the container.

Now when I run any service which makes some entries in iptables, Container stops working.
I see below errors..

Creating network “seg_backend” with the default driver
Failed to Setup IP tables: Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i br-24d7aa7869f4 -j RETURN: iptables: No chain/target/match by that name.
(exit status 1)) (edited)
i run command “docker network ls” , And i could see that its deleting the network created by docker.
I had to run below set of command manually to fix the problem

systemctl stop docker
pkill docker
iptables -t nat -F
ifconfig docker0 down
brctl delbr docker0
systemctl start docker

[riteshcusat08]

this is my docker-compose.yml file, I have kept networks as it is, edited other entries

version: '3.4'

services:
serviceName:
image: ""
container_name: ""
ports:
volumes:
healthcheck:
networks:
- backend

networks:
backend:

[tflori]

My 2 cents: of course it is an issue with docker-compose. When there is no container it will be created by default. When there is no image it will be built by default. When there is no network it will fail. That feels wrong for a tool that helps managing docker containers, images, networks and volumes.

From compose help:

...
  down               Stop and remove containers, networks, images, and volumes
...

I strongly disagree @shin- that you should know what you can and can not do with containers, images, networks and volumes managed from a docker-compose file. In worst case you are working together on one host and don't know where the compose file is that created one of the things that you see with docker.

[jlost]

--force-recreate fixed my problem, but it's strange that the network reference persisted even after using down.

[dcsan]

I had a similar problem. caused by trying to create botfront projects in two separate directories, completely borked the project.

I did:

docker-compose rm
botfront up -e botfront

and was able to get started again.

FYI If you use the docker dashboard to start/stop/restart the project from the image is a good idea.

you can also see the sub-services there too.