Copilot-cli: Task Run Command Design

[kohidave]

Some parts of customers workflows don't fall neatly into services, cron jobs, or triggers. Occasionally, customers need to run one-off tasks (think creating a database in a newly minted RDS instance, running a query against their instance, or just for debugging). This design talks about how we can enable this for customers.

What category is this command?

I believe the tasks domain should be under a new category called Operate. These commands are focused around one-off executions that are a normal part of operations, but may not be part of a customers day to day workflow.

How will it work?

copilot task run 

By default, this will build the local image. Then, if the customer is working within a workspace associated with a project, we can prompt them to choose an environment.

copilot task run 
Which environment would you like to run this task?
> Test
> Prod
> None (runs in your default VPC)

If they select an existing environment, we can set up an ECR repo for on-off tasks real quick (or
bake the one-off repo into our environment template), push the image, and launch the task on Fargate with a sensible default configuration.

If the customer doesn't have any projects set up (or they are not in a workspace) - we can just deploy the task to the Default ECS cluster and the Default VPC. Since this is a lower level command, we can expose flags like subnet or security group.

For this command, you don't have to have set up any ecscli projects.

Optional Flags

--num (the number of tasks to set up)
--cpu (the cpu-units)
--memory (the amount of memory)
--image (an optional image to run, instead of building a dockerfile)
--task-role (optional role for the task to use when running. If running in an environment, the default can be one of our roles like the env task role)
--env, --project (for selecting the environment and project to run this in)
--subnet-id, security-group-id (for choosing a vpc)
--env-vars (key/value list of environment variables)
--command (list of commands which are passed to docker run)

Some examples might be

# Builds, and pushes the local docker file to an ECR repo in environment test. This is run within
# a workspace, so we don't need to provide the --project flag. The VPC will be the env's VPC
copilot task run --env test
✔️ Great, we started your task _i3df4cg43_
$ run copilot task logs --task _i3df4cg43_ to view your task's logs. 

# If not run in a workspace, builds and executes the image in the default VPC/Cluster.
copilot task run

# Runs in the default VPC (if not in a WS). Spins up 4 tasks with 256vcpu, 2GB mem. Runs a 
# particular image (kohidave/ddb-parallel-scan-backup), passes in environment roles, and uses
# a particular role for the task to executed under. 
copilot task run --num 4 
                     --cpu 256 --memory 2048
                     --image kohidave/ddb-parallel-scan-backup 
                     --env-var bucket,davids-ddb-output-s3-bucket 
                     --task-role ddb-backup-role

# Same as above, but overwrites the default commands in the Dockerfile
copilot task run --num 4 
                     --cpu 256 --memory 2048
                     --image kohidave/my-service 
                     --env-var bucket,davids-ddb-output-s3-bucket 
                     --task-role ddb-backup-role
                     --commands rake db migrate

Isn't this like the Fargate CLI?

Yeap! Almost exactly like that. But we can also include integrations into our view of environments so folks can run their tasks _in_ a particular environment (ECS cluster / VPC). This is super useful if you have resources within a particular VPC that can only be accessed from within the VPC.

What next?

We'll have to follow up with visibility features for tasks. Logs, list, PS, etc.

Questions

• Should we do anything to limit folks from running commands in production environments?

[kohidave]

Other Example Usecases

• Calling a private service API (not exposed through the LB, but through service discovery)
• Querying a memcached/redis instance within an environment's VPC
• Running a task on Fargate quickly
• Running a one-off script which requires secrets, but you don't want to copy secrets to your dev machine

[JoeAlamo]

Being able to override the default command would be great.

For example, I have an image / task definition for a Django container that I use for my API. By default it will serve HTTP requests. But if I can pass an override, I can run admin tasks using manage.py, on the same codebase etc. just as if I was running it locally or on a server.

[kohidave]

Being able to override the default command would be great.

For example, I have an image / task definition for a Django container that I use for my API. By default it will serve HTTP requests. But if I can pass an override, I can run admin tasks using manage.py, on the same codebase etc. just as if I was running it locally or on a server.

All right - added a "commands" flag to address this. Good call. Thanks a bunch.

[aidansteele]

This would be amazing.

First thought: It would be amazing to have a "synchronous" mode. I'm thinking something like ecs task run --wait .... It would do the following:

• Execute run as described by @kohidave.
• Stream logs from the task.
• Waits until the task terminates. Because this is for tasks that terminate, right?
• The CLI returns the task's container's exit code itself. So a failure ECS-side causes failure on CLI-side. Great for running within a pipeline.

Second thought: It would be 🤯 if this was combined with ECS Cloud Debug's support for shelling into a running task. Then I could run a handful of commands, type exit and have the container be terminated.

[Louise15926]

Some thoughts on validating flags such as subnet-id, security-group-ids, and task-role.

1. It'd be easier to wait until Execute stage to "validate" these flags - the API call that deploys the task using the task definition will return an error if any of these arguments are invalid and prevent deployment.

2. Validate the arguments at Validate stage: several API calls need to be made in order to validate these arguments. For example, to validate subnet-id, we'd probably need to make API call to DescribeSubnets filtered by VPC id (which is either found via environment or is the default VPC).

3. Other options that I don't know 💭 💭 ...

Any thought or comment?

[efekarakus]

1. It'd be easier to wait until Execute stage to "validate" these flags - the API call that deploys the task using the task definition will return an error if any of these arguments are invalid and prevent deployment.

Your recommendation sounds good to me 👍 let's go for it :shipit:

[Louise15926]

Current behavior of the --app and --env flags:

If --app is specified:
--- If currently under a workspace
--- Inconsistent with current workspace (workspace’s app is a, but specified b) => is a valid input; use b instead of a as app
--- consistent => nothing special to do
--- If not in a workspace
--- the specified app exists => valid input
--- if doesn't exist => invalid input; throw error

If --env is specified:
--- If --app is specified and is valid
--- check if env exists under the app; error if it doesn’t
--- If --app is not specified
--- if currently under a workspace => Check if env exist; error if not
--- If not under a workspace => invalid input; throw error

[efekarakus]

One thing that would be neat after we deliver the proposed version is adding a new flag like --spot-count. That leverages the FARGATE_SPOT capacity provider: https://aws.amazon.com/blogs/compute/deep-dive-into-fargate-spot-to-run-your-ecs-tasks-for-up-to-70-less/

I'm thinking:

1. copilot task run --count 5 runs 5 regular fargate tasks.
2. copilot task run --spot-count 5 tries to run up to 5 fargate spot tasks.
3. copilot task run --count 5 --spot-count 5 runs 5 regular fargate tasks and up to 5 fargate spot tasks.

(Related https://github.com/aws/amazon-ecs-cli-v2/pull/1044)

[toricls]

I'm not sure it's related to this issue or not, but as an example use-case of standalone ECS task, sometime I use Step Functions' state machine to wrap a task to ensure the task to be retried when it fails.

[efekarakus]

This feature will be released in v0.3.0, we'll have a separate design to incorporate step functions to a task :).