Compose: Order of containers starting up?

[prologic]

Does fig starts containers in the order as
written in the fig.yml? If not is it meant to?

I've run into an issue where I need one or two
containers to start before others.

Related but different Issue: https://github.com/orchardup/fig/pull/74

[bfirsh]

Containers start in dependency order. E.g.

web:
    build: .
    links:
     - db
db:
    image: orchardup/postgresql

db will start before web. Is there a case where you'd want another ordering?

[prologic]

There are however situations where containers
do not depend (link) with one another.

e.g:

skydns
skydock
hipache
... etc ...

cheers
James

James Mills / prologic

E: [email protected]
W: prologic.shortcircuit.net.au

On Thu, May 29, 2014 at 2:31 AM, Ben Firshman [email protected]:

Containers start in dependency order. E.g.

web:
build: .
links:
- db
db:
image: orchardup/postgresql

db will start before web. Is there a case where you'd want another
ordering?

—
Reply to this email directly or view it on GitHubhttps://github.com/orchardup/fig/issues/235#issuecomment-44431687
.

[bfirsh]

Perhaps you could create a fake link between the two services? That's quite a nice way of explicitly saying that one service depends on another.

Alternatively, we could add some other option? Or perhaps boot them up in the order they are specified in the configuration file?

[prologic]

I'm thinking it would be nice to use an OrderedDict
when reading and parsing the YAML sot ath the
order you specify the services are they order they start.

(Reglardless of links, etc)

cheers
James

James Mills / prologic

E: [email protected]
W: prologic.shortcircuit.net.au

On Thu, May 29, 2014 at 8:30 PM, Ben Firshman [email protected]
wrote:

Perhaps you could create a fake link between the two services? That's
quite a nice way of explicitly saying that one service depends on another.

Alternatively, we could add some other option? Or perhaps boot them up in
the order they are specified in the configuration file?

—
Reply to this email directly or view it on GitHub
https://github.com/orchardup/fig/issues/235#issuecomment-44518936.

[defunctzombie]

Fake links are not possible when using the "net: host" option because docker will not launch containers with net host and links. The two features are mutually exclusive.

[nicgrayson]

:+1: I would lover ordering and if there was some sort of health checking before it moved on to the next container

[anentropic]

will volumes_from enforce dependency order too, like links does? I guess it needs to, just checking as maybe I can remove dummy links from my fig.yml

[aanand]

@anentropic yes, it will.

[hamiltont]

To make this issue actionable, I'd propose that this be added to fig's website. The automatic dependency handling, based on (from this discussion) links and volumes, is definitely worth mentioning in the documentation.

Also, is there any lifecycle checking for containers (like mongodb) that take some time between "the container run command returned" and "the container is ready to be used because port X is responding"?

[baptistedonaux]

+1

[nicgrayson]

@hamiltont i wrap fig in a bash script and use this function:

wait_for_endpoint() {
until $(curl --output /dev/null --silent --head --fail $1); do
printf '.'
sleep 5
done
printf "n"
}

[arun-gupta]

There could be a dependency where a container publishes REST endpoints that may be needed for other containers. Linking may not be necessary, but defining an order would be important in that case.

[mbdas]

Any comment on the dependency ordering that does not involve link or volumes from?
Also the pattern mentioned from docker guys involving config volume containers to have config update containers to be consumed by app containers and in this case both the update container and app container will have volumes_from but the update container needs to run and finish first before app container runs. In terms of ordering there can be 2 use cases. One where one container exits before other runs and in other case the second may run on good health check of first.

[vgoldin]

+1 for explicit dependencies not related to any docker specifics. In addition, startup waiting time would be very much appreciated among with dependencies.

[cboettig]

Adding an extraneous volumes_from seems enough to workaround the net container:name issue, but a proper fix would be great.

[dnephin]

This would be addressed by #686

[aanand]

@mbdas If I have it right, you're asking for a different kind of dependency ordering.

links and volumes_from express dependencies between long-running containers that determine the order in which each is created and started. Whatever happens once they're running is irrelevant.

You're asking for the ability to express dependencies between long-running and ephemeral containers, where an ephemeral container needs to be created, started _and finish running_ before a long-running container is created.

Is that right?

[mbdas]

Both actually. So, you got it right on the ephemeral containers which is a common patter for data volume update containers for example. So, looking for generic on-exit keyword where a target container can start on successful exit of a list of containers.

The second use case is a dependency between long running containers where the target container starts on successful health check condition (user supplied say with basic defaults) of the other long running containers. Link does not help in ensuring health checks and workarounds are not clean.

[aanand]

My reluctance regarding the second use case boils down to: I don't think health checks should be Compose's job. In a robust distributed system, the services _themselves_ are responsible for checking their dependencies, retrying, backing off etc.

You wouldn't deploy a web app to production that refused to start if the database wasn't ready yet, and since half of the point of Compose is to close (or narrow) the gap between development and production, I don't think it's good to add complexity for the sake of enabling bad practices.

I'd prefer if Compose did _no_ ordering at all, but the way Docker is set up (container config being immutable) it's unfortunately necessary to force an ordering just so we can actually create the containers.

[defunctzombie]

There is a difference between deploying and using something in dev. One
might have to preconfigured dbs or other tasks and it is just easier to do
that if you can put in a simple delay or ordering. I don't use something
like compose for deployment but I do use it in dev where workflows are
sometimes different for simplicity.

On Thursday, February 26, 2015, Aanand Prasad [email protected]
wrote:

My reluctance regarding the second use case boils down to: I don't think
health checks should be Compose's job. In a robust distributed system, the
services _themselves_ are responsible for checking their dependencies,
retrying, backing off etc.

You wouldn't deploy a web app to production that refused to start if the
database wasn't ready yet, and since half of the point of Compose is to
close (or narrow) the gap between development and production, I don't think
it's good to add complexity for the sake of enabling bad practices.

I'd prefer if Compose did _no_ ordering at all, but the way Docker is set
up (container config being immutable) it's unfortunately necessary to force
an ordering just so we can actually create the containers.

—
Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/235#issuecomment-76206994.

[aanand]

Fair enough, but we're planning to take Compose beyond just development environments - see the roadmap.

Can you show me a use case where putting the logic in the container is significantly more hassle than getting Compose to do it?

[defunctzombie]

Most of that roadmap is dev oriented which is awesome. Helping people dev
micro services is already good enough.

I'll think about it more, maybe will just write scripts to wait as you say.

My feedback: keeping compose dev oriented seems great to me. Vagrant had
this idea that you could do dev and then turn that into production. This is
not so simple and ends up being useless for any real production setup so
people just use vagrant for dev and testing and that is perfectly alright.

On Thursday, February 26, 2015, Aanand Prasad [email protected]
wrote:

Fair enough, but we're planning to take Compose beyond just development
environments - see the roadmap
https://github.com/docker/compose/blob/master/ROADMAP.md.

Can you show me a use case where putting the logic in the container is
significantly more hassle than getting Compose to do it?

—
Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/235#issuecomment-76214518.

[roytruelove]

+1

Check out this quote from the 'mysql' repo:

If there is no database initialized when the container starts, then a default database will be created. While this is the expected behavior, this means that it will not accept incoming connections until such initialization completes. This may cause issues when using automation tools, such as docker-compose, which start several containers simultaneously.

This is a showstopper for me for docker-compose, unf

[dnephin]

I regularly use docker-compose with mysql, but I don't use that image. You probably want to extend that image, and create your database and tables as part of the "build" step. That way the work is cached, and you don't have to repeat it every time you start up your container.

Any other container using the mysql image just needs to retry until the container is available. It needs to do that to be resilient to network failures anyway.

https://github.com/dnephin/readthedocs.org/blob/docker_compose/dockerfiles/database/Dockerfile is an approximate example of that setup using postgresql, it should be similar for mysql.

[gtmtech]

+1 for depends_on.

[rafaeltuelho]

When possible I'm using some kind of 'test/retry connection test' with the service that a given container/service depends on.

To workaround this limitation (service startup ordering/dependency), in my container's service startup script (eg: run.sh) I use this snippet of code to wait for: https://gist.github.com/rafaeltuelho/aff24eefe8c6e48a6b49

[MikeMichel]

is the startorder with "links" just an order or is compose really checking if the linked container (db) is in status "running" and only then starts the other one (web)?

something like

dependencies:
    - db
    - redis

would be nice

as a user of marathon i like how dependencies was solved https://mesosphere.github.io/marathon/docs/deployments.html
in combination with health checks https://mesosphere.github.io/marathon/docs/health-checks.html this makes sure db is "really" ready before web starts.

[dnephin]

There's a bunch of interesting discussion in this ticket, but it's unfortunately not terribly related to the original question.

Service ordering is currently based on links and volumes_from because that's a hard requirement for docker, but I don't think "order" fully describes the concern. The concern is more about availability, which is covered by #374.

I'm going to close this ticket, as I think it overlaps with #374.
There is also #686 for discussion about depends_on

If you feel there are issues unrelated to those ticket, please feel free to open a new issue.