Terraform: Use Docker Compose syntax in aws_ecs_task_definition?

[brikis98]

Is there any way to use a docker-compose.yml definition in an aws_ecs_task_definition instead of the AWS container definition JSON? I believe the AWS CLI supports it, but when I try it with terraform, I get an "error decoding JSON" error.

[gjohnson]

Seems like a higher level tool to me. The aws-cli doesn't support it actually, it's a more specialized tool called ecs-cli and I think you could apply the same specialization in terraform by just writing a script that parses a compose file and writes out a valid terraform json file.

[radeksimko]

@brikis98 Thanks for opening this issue.

@gjohnson is right - ECS CLI is a tool which is in reality just translating docker-compose.yml into the ECS task definition:
https://github.com/aws/amazon-ecs-cli/blob/82b379a6226f15d2fdeab0a393ac2105bb12f89f/ecs-cli/modules/compose/ecs/utils/convert_task_definition.go#L65-L69

There is no direct support in the ECS API hence no support in the AWS CLI either.

The implication of that is also that they're transparently ignoring some fields which are not supported in ECS, e.g. env_file:
https://github.com/aws/amazon-ecs-cli/blob/82b379a6226f15d2fdeab0a393ac2105bb12f89f/ecs-cli/modules/compose/ecs/utils/convert_compose_yml.go#L29-L53
https://github.com/aws/amazon-ecs-cli/blob/82b379a6226f15d2fdeab0a393ac2105bb12f89f/ecs-cli/modules/compose/ecs/utils/yaml.go#L122-L126

I first thought we could do some work to make ecs_task_definition to make it transparently convert between the two formats, but we would run into issues because of volumes - these are managed separately in the ECS API & Terraform DSL, but together in the Docker Compose YML.

Considering the above ^ I think it would probably make sense to create a new resource, e.g. ecs_docker_compose_task_definition or something like that, but I'm not sure about that either ...

I personally decided to go ahead with two separate configs, one for ECS TD and second one for Docker Compose. Now, when I see some (intentional) differences between the two - e.g. volumes (not) being attached, I'm thinking it wasn't a wrong decision.

[brikis98]

@radeksimko: Thanks for the detailed reply.

I've been using the JSON container definition format and you're right, there are some differences, such the env_file, as well as as CPU/memory settings. That said, it feels like 95% duplication, and the other 5% could probably be handled using a concept similar to the docker-compose.override.yml file which, as the name implies, allows you to specify overrides that get merged together with the underlying docker-compose.yml file.

I suppose the other direction is worth pondering too: is it possible to generate a docker-compose.yml file from the container definition JSON? The latter seems to be a superset of the former, so this should be straightforward, and such an approach would allow you to maintain a single source of truth and reuse the existing docker-compose.override.yml functionality.

[radeksimko]

@brikis98

I suppose the other direction is worth pondering too: is it possible to generate a docker-compose.yml file from the container definition JSON?

Agreed, conversion in that direction would be easier, but I don't think that's something related to Terraform - if you want the _end product_ to be docker-compose.yml, then your concern is mostly local environment I think and Terraform's goal (AFAIK) isn't to manage local environments.

Have a look at Vagrant and https://github.com/leighmcculloch/vagrant-docker-compose - u could add such functionality in there via PR. Or maybe you could start such discussion in the docker-compose itself https://github.com/docker/compose - not sure whether either of these would want to accept ECS TDs as the input format, but it's worth trying.

[tilgovi]

The ECS-CLI actually only supports Docker Compose version 1 format right now, which doesn't have network or volume definitions. That might be how it's easy for them to support it.

See aws/amazon-ecs-cli#86

[tilgovi]

I would propose maybe tracking that issue, and if support for version 2 lands in ECS maybe it will come with support for an expanded set of options that make the transformation easier.

[eedwardsdisco]

Docker Compose v2 Syntax is now supported natively by the ECS CLI

It would be really nice to have support for this in terraform

Use case:

• local environments powered by Vagrant + Docker Compose
• remote environments powered by Terraform + ECS
• ideally both using the same docker compose file...

[apparentlymart]

Hi everyone! Thanks to @brikis98 for opening this issue and everyone else for the great discussion.

This definitely seems like a useful feature, but due to the complexities involved (and our preference for not laying too much "extra abstraction" on top of provider APIs) we on the Terraform team are not planning to work on this.

As @gjohnson noted, it's possible to make an external tool that does the conversion between formats and then pass the result to Terraform. That is the path I'd suggest for now, possibly using the external data source to call it from Terraform.

Since there is an (albeit less convenient) way to achieve this, I'm going to close this issue. Thanks again to everyone, and thanks for your patience.

[jmahowald]

For others who stumble across, it appears that someone made a non generic external data source and has a full blown provider for it. https://github.com/jritsema/terraform-provider-compose2ecs . Thanks @jritsema

[ee07dazn]

@jmahowald - did you try/test that out? or did any else here tried that out?

[binarymist]

did you try/test that out? or did any else here tried that out?

Looks like the owner has lost interest

[tamsky]

did you try/test that out? or did any else here tried that out?

Looks like the owner has lost interest

Not just the developer of terraform-provider-compose2ecs,
but also the developers of the upstream library on which it was based, namely:

• https://github.com/docker/libcompose
  have lost interest...

libcompose was tagged "unsupported" in October 2018:

• https://github.com/docker/libcompose/pull/516

Which (imho) puts a few extra nails into the idea of continuing it.
Lack of support for libcompose was what stopped me this week, as I was examining updating the compose2ecs provider.

Despite all that, aws continues to develop and support their amazon-ecs-cli tool... which still relies on docker/libcompose

---

I did examine

• https://github.com/ashald/terraform-provider-yaml (which is now also marked obsolete)

But in a stroke of good news, the yaml provider is obsolete because its core features have been added as core interpolation functions as of terraform-v0.12.2. 🥳

Can anyone imagine a way for the newly introduced yamldecode() interpolation to allow docker-compose schema to sneak into the container_definitions schema?

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.