Copilot-cli: Docker build args sourced from Shell environment variables

[flyinprogrammer]

So we'd like to have copilot source environment variables from the shell that copilot is run in as build arguments into the docker build command. And it either already does this, and I am dumb, or it does not.

In my mind some form of the following might work:

manifest.yml:

image:
  build:
    dockerfile: path/to/dockerfile
    context: context/dir
    args:
      SECRET_PASSWORD: {{ .Env.SECRET_PASSWORD_IN_SHELL_ENV }}

In my shell:

export SECRET_PASSWORD_IN_SHELL_ENV="ssshthisissecret"
copilot deploy
# at some point this would happen:  docker build -f Dockerfile --build-arg SECRET_PASSWORD=ssshthisissecret

I thought maybe, perhaps, we were clever and ran the manifest file through go template before shoving it into a struct, but i'm either stupid, or we aren't doing this? Happy to provide a PR and shame on me for not running this in a debugger before opening this issue, but alas here we are.

[bvtujo]

Hey @flyinprogrammer, you're right that Copilot doesn't do this currently. This is a super interesting use case though, so thank you so much for bringing it to our attention.

Also, you're right that we run docker build under the hood and it's not great to require that you expose your secrets in the manifest if you have to pass them in as build args.

Depending on your use case, though I can think of a couple of workarounds of varying complexity that will allow you to keep using Copilot for infrastructure management 😄

If you need secrets inside your deployed container:

We recommend you use SSM and the secrets section of the manifest to specify credentials that you need to access from inside your running containers.

If you need secrets from the terminal environment as build arguments

You can run docker build manually using whatever environment variables suit you, then use the image.location field of the manifest to specify an external repository (Dockerhub or ECR, both work 😄 ) from which to grab the built secret-containing image.

HUGE CAVEAT that if you do this, be really careful not to specify any secrets in the variables section of the manifest as they'll be exposed as plaintext in the Environment Variables section of the Container Definition inside the Task Definition on the AWS console.

Do either of those work for you?

[flyinprogrammer]

This is a super interesting use case though, so thank you so much for bringing it to our attention.

Yah, the primary use case is that we have private python/java repos that tools like pip and gradle need access to at build time, as we build our code in the container. Building our image outside of Copilot seems reasonable for now.

image.location

So this would require us to manage the version in the manifest.yaml file - which is fine, but it still might be nice to have a way to make that dynamic since most of our use cases already have CI tools providing Git hashes and Tag names as environment variables. At some point it might be nice to have copilot acknowledge these ideas as Default Environment variables, but perhaps it's out of scope.

I'll close this for now as your backlog of issues will either force this requirement naturally, or it will be able to be delayed indefinitely, which is no problem either way.

[acarlton]

I would also like to see this feature.

[perevernihata]

bump on this one, things like nextjs require build time env vars, hence deploying from local using copilot deploy becomes impossible