Compose: Proposal: Stop using sequential container numbers

[aanand]

Sequential container numbers (myapp_web_1, myapp_web_2) are problematic.

1. They're costly to maintain: we have to query the names of all containers for a service to determine what the next number should be. As apps get bigger, this is going to get worse.
2. They're error-prone: if a Swarm node with myapp_web_LASTNUM goes down at the same time as Compose is querying the cluster, it will incorrectly think LASTNUM is the next number we should use. Once the node comes up, there'll be a clash of container names.
3. They make it impossible to do multiple docker-compose runs in parallel.

As such, I think we should start using random suffixes. This may well break some scripts that rely on container names, but now that we're using labels to track containers, we should be explicit about making _no guarantees_ about container names.

[villlem]

For scripts is useful ps command with -q option. If there is option for get my only one container there would be no need for some explicit naming.

[SBoudrias]

I just wondered what were the plans for this feature? Is it planned soon?

[dnephin]

Hey Simon!

We aren't doing anything about this in the 1.5 release (which will be out soon), but I think we may look to implement something in the 1.6 release which will be a few months away.

[murphyke]

+1 I want to run parallel copies of a container in a data processing pipeline. If there were a --container-name=NAME CLI flag, I would be in business, but random suffixes would work also.

[ibizaman]

Another use case where this would be useful is if the prefix - the directory of the docker-compose.yml - is the same and the same service name is used.

For example, having two apps with the following structure:

myapp/
    doc/
    src/
        docker-compose.yml
myapp2/
    doc/
    src/
        docker-compose.yml

and assuming they both use a mysql service, this will result in two identically named services src_mysql_1.

A not so great solution would be to use the complete $PWD as a prefix.
But in fact, why do we even need to name the machines in the first place? This is not a troll question. :)

EDIT: I just found out about the --projectname command line option which solves my particular problem. Although the question about why should we even give a name still stands.

[dnephin]

The project name is required to support isolated environments.

[caiofbpa]

:+1:

[rvernica]

@dnephin, just wondering, are you still planning to remove the sequential container numbers?

[kchudy]

I just came across this issue when CI executes tests in parallel (triggered by different commits). I handled it by generating a random name for the container like so:

docker-compose run --rm --no-deps --name `uuidgen` backend python3 ./tools/manage.py test

[EspadaV8]

Thanks @kchudy . I had to use cat /proc/sys/kernel/random/uuid instead of uuidgen since that wasn't available on my CI server so the full command looks something like

docker-compose \
  -f docker-compose.yml \
  -f docker-compose.override.yml \
  -f docker-compose.development.yml \
  run --rm \
  --name `cat /proc/sys/kernel/random/uuid` \
  dev composer install --no-progress --no-ansi --no-scripts

[shin-]

Related: #4688

[etc0de]

Is pure naive randomness enough to solve the parallel docker-compose run problem though? This could still lead to races, it just makes them more unlikely.

IMHO a mechanism is needed in addition to query a number for being free & reserve it in one atomic step. This should be less work performance-wise than going through all numbers to find the next higher one as of now, but it's still more complex than simply throwing a dice and using that number without any further considerations. However, I guess it begs the question who keeps track of the reservations - something in the file system? The docker daemon? ..

[shin-]

@JonasT That's just not an issue in practice: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions

[etc0de]

@shin- true if you use UUIDs. This wasn't confirmed before, and I kind of thought you might use a random 64bit int or similar. In hindsight, I suppose just using UUID4 solves this. Just as a relevant side note, are all modern systems equipped with sufficiently good entropy sources? I know some embedded systems aren't, I'm just pointing out in case it is relevant.

[adamkdean]

It's a shame that this has to be an issue. While it may not work in your use cases, there are so many other situations where this isn't a problem, sometimes simply testing services at scale. The use of hostnames in some cases can greatly help to identify services, and the ability to generate unique hostnames, whether sequentially numbered or randomly generated, is sometimes a requirement that cannot be ignored.

Take for example a service which identifies itself via it's hostname which must be unique within a cluster. The choice currently is: use randomly generated container hostnames, e.g. bf7c24f9f7c3, b817bffc9cd9, at the expense of clarity, or don't use docker-compose to scale. Supporting a simple feature doesn't mean docker-compose has to enforce this. Simply exposing a ${SCALE} variable would solve a lot of problems without introducing any.

I hope that you can see this, and perhaps in time, introduce it. We've come a long way since 2013, but we still have so far to go.

[kierankelleher]

What @adamkdean said : “Simply exposing a ${SCALE} variable would solve a lot of problems without introducing any.”

[sergiogiro]

Is there a way to programmatically query what the containers for a service are? How can I refer to them externally from a program if I don't know what the names are?

[ngrilly]

@sergiogiro docker ps --filter label=com.docker.compose.service=my_service_name

[sergiogiro]

Thanks, appreciated. Plus -q so that you only get the ids :)

This change to docker compose broke our integration tests (as CircleCI automatically updated to the new version), and it was quite difficult to see why. FWIW, I think that a less traumatic implementation path would have been:

• the random slug should have been disabled by default, and a warning displayed if the option to activate it wasn't set
• make the slug present by default, having an option to deactivate, with a warning displayed if the slugs are deactivated, indicating that the option will be deprecated in the future (so that users facing a break don't need to rewrite a lot of code immediately as to fix this, as there's a temporary fix)
• deprecate the option to deactivate

[sergiogiro]

Oh, and including @ngrilly 's one liner (plus -q :P) in the release notes where this was mentioned

[ngrilly]

@sergiogiro I forgot about this: if your run multiple projects, you should also specify the project: docker ps -q --filter label=com.docker.compose.service=my_service_name --filter label=com.docker.compose.project=my_project_name.

[flaushi]

The default naming scheme has changed from 1.22 to 1.23.
now there is a random suffix in the container name.
I bet this breaks thousands of scripts. IMHO, this must be undone!

[shin-]

Copying what I wrote in #6140 :

While this can be seen as a breaking change in the sense that container names can no longer be inferred the way they could before, it should be noted that

1. Compose never made any guarantees about the naming pattern of the containers it creates
2. Relying on inferred patterns can result in unexpected errors when the sequentiality breaks, making it something to avoid wherever possible
3. In the rare cases where it is necessary, scripts will still be able to fetch individual containers based on the predictable prefix (the part before the slug), with the caveats mentioned above. The prefix is also backward-compatible.