Compose: Proposal: One-off tasks

[bfirsh]

Background & problem

Users quite often define maintenance scripts, test suites, debugging systems in their Compose files which they don't want to run when they do docker-compose up.

Personally, this is something I've wanted since almost the start. My daily paper cut is writing docker-compose run web ./test.sh instead of docker-compose run test.

I wrote a diplomatic aggregation of potential solutions and data points in #1896. This is an opinionated proposed solution we can use as a strawman.

Proposed solution

I propose we add a top-level tasks configuration option, which takes the same options as the services option. It's a bit like a service, but doesn't start when you run docker-compose up. It only works with docker-compose run. Service and task names must be unique so they don't collide.

As an example, let's add a task to the Django getting started guide app:

services:
  db:
    image: postgres
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
  web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - "8000:8000"
    depends_on:
      - db
tasks:
  test:
    build: .
    entrypoint: python manage.py test
    depends_on:
     - db

You would run the tests by running docker-compose run test.

Because we've specified entrypoint in the task definition, we can also pass arguments to the tests: docker-compose run test myapp/test_views.py

Questions

• What am I not thinking about?
• tasks is a nice bike shed -- go at it
• How does this interact with Compose spec?
• Having to specify entrypoint instead of command is a bit clumsy. Users will automatically write command then wonder why their arguments aren't working as expected. Maybe there is a better solution here?

/cc @aanand @shin- in case there is any historical knowledge I am not aware of. (hi!)

[ndeloof]

I like the idea compose file format can be used to define tasks/jobs that are expected to stop after completion, but imho this is orthogonal to the initial intent to only run some of them in a specific context.

For this purpose, I'd prefer a "profile" approach like I used to do with Apache Maven on java projects: Just like docker-compose by default will load a docker-compose-override.yaml file if it exists aside the main yaml config, we can introduce support for --profile flag that will automatically pick an override file to be loaded.

i.e
docker-compose up => docker-compose.yaml + docker-compose-override.yaml
docker-compose --profile test up => docker-compose.yaml + docker-compose-test.yaml

so, your sample becomes:

$ docker-compose --profile test run test myapp/test_views.py`

or

expose DOCKER_COMPOSE_PROFILE=test
docker-compose run test myapp/test_views.py`

How does this interact with Compose spec?

This is indeed the right place to have this discussion :)

[thaJeztah]

(copying from a slack conversation we had)

Alternatively we could make "profile" a property of services/objects. Implicitly all objects would be in the default profile, but could have custom profiles specified.

For example, a service that has:

profiles: ["debug"]

Would only be started if the --profile=debug profile is selected, whereas;

profiles: ["default", "prod"]

Would be started if _either_ no profile was selected, or the --profile=prod profile was selected. But would not run if --profile=dev is picked.

The above would facilitate "run definitions"; e.g., services that are defined in the compose-file but "never" run automatically. This could be achieved by setting a "dummy" profile (e.g. "run-never") and only running those manually.

At the same time, it would allow for a "set" of tasks to be specified that can be run as a group, e.g. setting profile "migration" and running docker-compose up --profile=migrate would start all services that are in the migration profile.

Note: "profile" as a name would have to be looked at; there also was a proposal somewhere to have "container profiles" that allowed certain default configurations for a container, so we need to look carefully if "profile" is the best name.

[bfirsh]

I am thinking of this primarily from a user's point of view. I can understand how profiles might make sense from an architectural point of view, but from a user's point of view, frankly it is confusing and doesn't map with my intent.

I have a test script, and I want to run the test script. I'm not switching between modes or profiles.

Could you explain profiles to me as if I were a user? How would I use them to implement my example above, step-by-step?

Perhaps another way of helping me understand: What other use-cases are there for profiles? replicate.yaml examples of those use-cases would be helpful too.

[ndeloof]

From a user point of view, I agree "profiles" can smell over-engineering, maybe could be considered separately as a generic improvement to flexibility.

Regarding your proposal, I'm just concerned with the use of tasks as a name for this new section as we might want to support "batch" processes set by compose in the future, not just long-running services. Maybe "one-off" as this is the noon used in the code (not being a native english speaker I can't tell what would better reflect the intent)

Alternatively, with lower (?) impacts, "services" could also get a new disabled attributes that can be set so that service is not started by default, but can be enabled either by some env variable or by an explicit run command. This is option #2 "Add an option to services to make them not start by default" in https://github.com/docker/compose/issues/1896. If we decide to adopt this approach, @thaJeztah's proposal makes more sense to me as this both allow to easily disable a service _and_ add support to just switch between two deployment profiles, typically running --profile debug to run a set of debugging sidecars.