Copilot-cli: Feature Request: Run tests inside a container

[nathanpeck]

In the current Copilot pipelines the only option for running tests is to run commands directly inside the CodeBuild stage. This doesn't really fit with the container-first spirit of Copilot.

There are a few things that need to be done to enable containers for testing:

1. Copilot currently runs the test stage on a plain CodeBuild environment, that does not have to toggle turned on for supporting Docker in Docker, so it is impossible for users to call a docker daemon at all. This should be an easy fix, to make Docker available in the test stage so people can implement their own tests using Docker.
2. Ideally Copilot would allow users to specify a Dockerfile to build and run for tests. Copilot would build and push both containers (the application container and the test container)
3. Copilot would ideally support both unit and integration tests, with the unit test stage running an "exec" type command on the test container, in order to execute unit tests, and integration tests running the container with an environment variable which points at the external URL of the service, allowing the integration tests to interact with the deployed service for that stage as if it was a real user.
4. With tests as containers Copilot could optionally do regression testing to ensure that backwards compatibility has not been broken. This would run both the latest version of the test container and the previous version of the test container, to ensure that both the latest tests and the previous version of the tests pass against the environment.

Some or all of these approaches would encourage users to implement both unit and integration tests, and it would allow them to keep every aspect of the release pipeline containerized.

[ceagan]

I am attempting to run Selenium tests in the end-to-end test phase. To make this as portable as possible, I configured my tests to run inside a Docker container based on bitnami/java that communicates with another Docker container (selenium/standalone-chrome).

Unfortunately, the build environment doesn't have Docker installed, and it doesn't seem possible to enable it from the copilot configuration files. Is it possible to add the following to the generated buildspec.yml so that Docker is available in the end-to-end test environment?

phases:
  install:
    runtime-versions:
      docker: 18

[efekarakus]

This is now released in https://github.com/aws/copilot-cli/releases/tag/v1.5.0! 🚀