Compose: Update to docker-compose version 1.21 breaks certain deployments

[skunde]

Description of the issue

docker-compose version 1.21.0 is not able to handle com.docker.compose.project labels set with an older version of docker-compose that contain - or _ in their container names.

Our deployment does the following:

// pull new images
$ docker-compose pull
// stop old containers
$ docker-compose rm -s -f
// replace docker-compose.yml
// start new containers
$ docker-compose up -d

Context

docker-compose version 1.21.0, build 5920eb0

Docker version 18.03.0-ce, build 0520e24

Steps to reproduce the issue

1. Deploy some containers with - or _ in their container names with docker-compose version 1.20
2. Change docker-compose to version 1.21
3. Try to restart the containers

Observed result

docker-compose does not recognize containers deployed with an older version.
It fails to stop already deployed containers and fails to start the new containers afterwards.

Expected result

docker-compose recognizes containers deployed with an old version and transparently rewrites existing com.docker.compose.project labels to the new format.

Stacktrace / full error message

$ docker-compose rm -s -f
No stopped containers
$ sudo -E docker-compose up -d
Creating container-name ... 

Creating container-name ... error

ERROR: for container-name  Cannot create container for service container-name: Conflict. The container name "/container-name" is already in use by container "e7dfd157fed12251ed21a70f8243d87baa6fb1e2768e872875c101d37cf99891". You have to remove (or rename) that container to be able to reuse that name.

Additional information

This issue was introduced with #5844 at https://github.com/docker/compose/blob/9cc30ad0e940453228d6d968e4827cae34eda770/compose/cli/command.py

[luigi-bellio]

same issue ... this version of compose changed the rule to build container and volume names prefix ... if the file "docker-compose.yaml" is in a folder with name "a-b" ... previous version of compose used "ab_" as prefix for containr and volume names ... this version use "a-b_" instead

[niQo]

probably related to https://github.com/docker/compose/pull/5788

[weijiekoh]

This problem can also be reproduced if you use docker load to restore an image and then docker-compose -f docker-compose.yml up -d to run it. It will try to rebuild the image from the Dockerfile specified in docker-compose.yml, even though the image already exists thanks to docker load.

[aPoCoMiLogin]

Same issue here. Since I can't wait for fix, did created services docker-compose up --no-start and then just copy data from previous/old volumes:

docker cp myproject_data_1:/data . && docker cp data my-project-data:/

[DenisKuplyakov]

Is this right that due to new naming scheme docker-compose won't pickup old volumes?

[ryanolson]

It also affects using options like ipc for sharing IPC namespaces. This naming change should be reverted.

[jedie]

What's the best work-a-round to handle that until 1.22.0 is out?!?
Downgrade to 1.20.1 seems not to work here :(

[skunde]

Because we can afford a little downtime of our applications we basically stop and remove the currently running container(s) and start our deployment afterwards.
It is not the most elegant solution.

We looked into changing the labels on the running containers but unfortunately you can not do that.

[sir-gon]

Also... there is no current way to rename volumes:
https://github.com/moby/moby/pull/33966

[shin-]

The 1.21.1 release we pushed out last week will pick up all pre-existing resources, even if they were created with an older version, so that shouldn't be an issue anymore.

[ajmueller]

@shin- I updated Docker for Mac this morning which included the 1.21.1 release and it ended up wiping out the new volume I had created to account for the change in 1.21.0. I'm guessing others will experience the same. Here's a breakdown of what happened:

• Project is contained in a folder called web-application
• Original volume named webapplication_database
• 1.21.0 update creates web-application_database volume, which is empty
• I delete web-application_database volume and try to move the data from webapplication_database to web-application_database. In the process, my data somehow gets wiped (I'm still not sure how this happened, but I believe it was my fault). At this point webapplication_database is an empty volume.
• I let Docker for Mac create web-application_database again. I populate the volume with new data and use it for a couple days with no issues.
• 1.21.1 update comes out today. I download and installed the related Docker for Mac update.
• Starting my local server now results in web-application_database getting recreated (presumably as a copy of webapplication_database), resulting in a now empty volume and all of the data I entered a couple days ago was lost.

There are some steps here that were definitely my fault, but I imagine others may encounter this scenario and it seems like one that could and should be avoided. Before the last step I should have deleted the webapplication_database volume. I thought I had, but I guess not.

Basically it seems that the 1.21.1 update didn't recognize that I had already created the web-application_database volume myself and went ahead and created it again anyway as a copy of webapplication_database.

[shin-]

@ajmueller Hmm, that seems odd ; but first off, Compose doesn't perform any volume copy (there's no way to do that that is driver-agnostic) and doesn't remove volumes unless you explicitly request it (typically with the down command). So there's a good chance your data's still out there.

You'll want to inspect your containers and check which volume is being mounted, and also check the physical location of each volume to see what data they contain. Chances are things got mixed up if you've got both the legacy and new naming in there, but it should be fixable.

[felixlinker]

1.21.1 doesn't seem to have resolved this issue for me.

I have to images created that are named wplocalclidocker_wp-dev_1 and wplocalclidocker_mysql_1 which are defined in a folder named wp-local-cli-docker. When they're stopped and I attempt to restart them via docker-compose up -d new containers are created named wp-local-cli-docker_..._1.

---

C:\Users\felix\git\wp-local-cli-docker>docker-compose version
docker-compose version 1.21.1, build 7641a569
docker-py version: 3.2.1
CPython version: 3.6.4
OpenSSL version: OpenSSL 1.0.2k  26 Jan 2017

C:\Users\felix\git\wp-local-cli-docker>docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:12:48 2018
 OS/Arch:      windows/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:22:38 2018
  OS/Arch:      linux/amd64
  Experimental: true

[adamhooper]

@shin- This change of behavior deserves a major version bump.

Picture this real-world scenario:

1. I tell power-users, "run this using docker-compose up"
2. Power users do so with docker-compose 1.20
3. Other power users do so with docker-compose 1.21
4. "Oh, no!" I say ... and I fix my command, docker-compose up --project-name myprojectwithoutdashes

Now some users have volumes with the wrong name. And I have scripts that use raw docker commands that expect a certain project name. So whom should suffer after I recover from this _undocumented_ change: the users from step 2, or the users from step 3?

My suggestion: mark 1.21 a "bad" release and publish a 1.22 that undoes the change to minimize the damage. Save this (very welcome) change for docker-compose 2.0.

[shin-]

@adamhooper As I said somewhere else, I'm not introducing another breaking change to undo a previous breaking change, as unfortunate as it may have been. I don't see a scenario where that doesn't make things worse for everyone involved.

@felixlinker I'm not sure what the issue is - as far as I can tell, what you're describing is the intended behavior.

[cpfeiffer]

I'm pretty sure there are many people still using versions < 1.21. At some point they are going to upgrade and suffer from the breakage, so reverting the breaking change would save them the hassle.

Additionally, since docker volumes cannot be renamed people are scripting around docker and compose, using the well-known docker-compose naming conventions. All these scripts and configurations are being broken.

[felixlinker]

@shin- Thanks for the clarification. I got this wrong then.
When there is no way to rename containers, is there a way to copy the volumes used by the "broken container" to a newly created container that adheres to the new naming scheme? That should be sufficient in my case but I couldn't find any hints on that.

[shin-]

@cpfeiffer As long as these hypothetical users upgrade to 1.21.1+, there shouldn't be any significant issues.

[...] using the well-known docker-compose naming conventions. All these scripts and configurations are being broken.

Naming conventions should never be treated as a guarantee. People doing so at their own risk will have to adapt their scripts when they upgrade – I believe that is a reasonable expectation. If external scripts depend on volumes being named a certain way, then those names should be explictly declared in the Compose file

@felixlinker Compose will look for the volume using the new naming convention first, then attempt to find it using the legacy naming convention. If you have created both (probably happened if you used 1.21.0) and want to retrieve your previous data, manually removing the volume with the new name and recreating the containers (docker-compose up --force-recreate) should do the trick.

[cpfeiffer]

@shin- Totally agree with that, but conventions shouldn't be broken in a minor release, even without any kind of warning or announcement.

Since you don't intend to revert the change, I'd suggest adding a warning/announcement somewhere, at least.

[coaxial]

That happened to me. I was running an older docker-compose version that didn't have hyphens. I have some services that were created and started before hyphens in names were a thing.

Now docker-compose ps etc do not work with those services anymore because it's looking for my-project_service_# names instead of the existing myproject_service_#.

And new projects are broken too. If I start a new project that didn't exist before updating to 1.21.2 with docker-compose up, it will still name without hyphens, and every docker-compose command is broken.

This happens with 1.21.2

For anyone with this issue, the only solution is to go back to 1.20.1 (the last version before the breaking change):

$ which docker-compose # to see where it's installed
$ sudo rm $(which docker-compose)
$ sudo curl -L https://github.com/docker/compose/releases/download/1.20.1/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose # or wherever your previous docker-compose was
$ sudo chmod +x /usr/local/bin/docker-compose # or wherever you put it
$ sudo docker-compose --version
docker-compose version 1.20.1, build 5d8c71b

[shin-]

@coaxial #5982

[coaxial]

@shin- Thanks, I'll stick to 1.20.1 in the meantime. Any ETA for 1.22.0?

[skrprabhu]

Is this fixed in 1.24.1, build 4667896b ??