Copilot-cli: [Resources] Environment Resource Import

[kohidave]

Overview

Customers sometimes want to bring their own resources (VPCs, Security Groups, Policies). This task is to make that possible.

Scope

The scope of this design (including use cases and following sections) only includes Milestone 1. Once it is accomplished, we’ll proceed to the rest of milestones, which might also be adjusted in the future.

Target Use Cases

Use case 1:
I have an existing VPC with subnets where legacy applications exists. I want to add new Copilot services to these subnets so that they communicate with legacy apps.

Use case 2:
I want to do VPC peering so that I need to specify the CIDR ranges of the new copilot environments (VPC and subnets).

Requirements

1. Use an existing VPC.
2. Use existing public/private subnets.
3. Override subnet CIDR ranges

Design Proposal

Documentation
This feature requires following documentation changes:

1. Add a new page “Customizing your env resources” under “Developing”.
2. Update env init page accordingly.

Help Menu

$ copilot env init

Required Flags
  -a, --app string       Name of the application.
  -n, --name string      Name of the environment.
      --prod             If the environment contains production services.
      --profile string   Name of the profile.

Import Existing Resources Flags
      --import-vpc-id string                 use an existing VPC ID.
      --import-public-subnets stringSlice    use existing public subnet IDs.
      --import-private-subnets stringSlice    use existing private subnet IDs.

Configure Default Resources Flags
      --override-vpc-cidr ipNet                   global CIDR to use for VPC (default 10.0.0.0/16).
      --override-public-subnets-cidr ipNetSlice   CIDR to use for public subnets (default 10.0.0.0/24,10.0.1.0/24).
      --override-private-subnets-cidr ipNetSlice   CIDR to use for private subnets (default 10.0.2.0/24,10.0.3.0/24).

Optional Flags
      --use-default bool          skip prompting and use default environment configuration.

Note that vpc importing and adjusting are mutually exclusive with each other since we cannot edit any existing resources created by users. And when they specify only --vpc-from-id we’ll take all subnets in the vpc by default. However, if they specify both --vpc-from-id and --vpc-public-subnets or --vpc-private-subnets then we’ll only use specific subnets.

UI Changes
Env resources importing:

$ copilot env init
What do you want to call your environment? [prod]

Is this a prod environment? [y]

Would you like to use the default configuration for a new production environment? 
  - A new VPC with 3 AZs, 2 public subnets and 2 private subnets
  - A new ECS Cluster
  - New IAM Roles
  - Termination Protection for resources
[y/n]: n

Ok no problem - would you like to adjust the environment config or import your own resources?
> Import resources (VPC, subnets)
  Adjust environment configuration (CIDR)
  Skip and create default

All right, which VPC would you like to use?
> vpcName1
  vpcName2

Which public subnets would you like to use?
> [x] subnet1
  [x] subnet2
  [ ] subnet3

Ok great, we’ll create an environment using __subnet1__, __subnet2__ in __vpcName1__.

All right, creating your production environment called prod.
...

Env resources adjustment:

$ copilot env init
What do you want to call your environment? [prod]

Is this a prod environment? [y]

Would you like to use the default configuration for a new production environment? 
  - A new VPC with 3 AZs, 2 public subnets and 2 private subnets
  - A new ECS Cluster
  - New IAM Roles
  - Termination Protection for resources
[y/n]: n

Ok no problem - would you like to adjust the environment config or import your own resources?
  Import resources (VPC, subnets)
> Adjust environment configuration (CIDR)
  Skip and create default

All right, what VPC CIDR would you like to use? (10.0.0.0/16) [10.2.0.0/16]

What CIDR would you like to use for the first public subnet? (10.0.0.0/24) [10.2.0.0/24]

What CIDR would you like to use for the second public subnet? (10.0.1.0/24) [10.2.1.0/24]

Ok great, we’ll create an environment using adjusted environment configuration.

All right, creating your production environment called prod.
...

When --use-default=yes is set:

$ copilot env init
What do you want to call your environment? [prod]

Is this a prod environment? [y]

Ok great, we’ll create an environment using default environment configuration.

All right, creating your production environment called prod.
...

Milestones

Milestone 1:

• VPC and subnets importing
• UX prompting changes (breaking changes for scripting users)

Milestone 2:

• Blog post for this feature, including an example for creating a Copilot service with existing infrastructures. For example we can migrate part of the monolith ElasticBeanstalk service to Copilot service.

Milestone 3:

• Security group importing
• Customized environment manager role

Milestone 4:

• Manifest for environments (if needed)

[kohidave]

Similar issue from our friends over on Amplify.

[jdhom]

We have demand for this as well. Larger org where cloudGov manages VPCs and works with separate network team to extend our IP space to internal VPCs. As it currently stands... we will not be able to use ecs-cli v2 without support for mapping to existing VPCs.

We have existing Ansible automation, based on ecs-cli v1 and see a lot of value in the new features of v2.

[khrno]

I think this would be a great help for AWS Copilot adoption in current existing workloads. Because behind Copilot is Cloudformation, and that service has the feature to import existing resources, could use that to implement this feature.

[efekarakus]

Related https://github.com/aws/copilot-cli/issues/1125

• Usecase VPC peering

[adanemayer]

I would like to see support for importing a manifest that contains shared elements (VPC, subnets, security groups, ALB rules) and only requires a user to specify the parameters unique to their application (image name, ports, variables). The Operations team can provide a couple manifests for different use cases (frontend, backend, isolated) based on deployed resources already present in AWS. This would allow developers to use CoPilot to run their application within existing infrastructure and ensure non-containerized resources are available if needed.

[dreamorosi]

As already mentioned above larger orgs have specific requirements around security and compliance when it comes to VPCs, Subnets and SGs so for us, at least when it comes to internal use, we would have to retouch all the resources created by Copilot for them to be compliant.
Another use case that I don’t see being mentioned is the one in which developers are working in an environment in which they don’t have rights to provision/modify network related components (this is very often the case in client’s environments).
For us being able to reference existing network resources would be great.

[iamhopaul123]

As already mentioned above larger orgs have specific requirements around security and compliance when it comes to VPCs, Subnets and SGs so for us, at least when it comes to internal use, we would have to retouch all the resources created by Copilot for them to be compliant.
Another use case that I don’t see being mentioned is the one in which developers are working in an environment in which they don’t have rights to provision/modify network related components (this is very often the case in client’s environments).
For us being able to reference existing network resources would be great.

Hello @dreamorosi, thanks for your suggestion! We are currently working on allowing users to bring their existing SGs and would like to know more about the use cases on that. Could you put more detail on it (like what kind of SGs and how many of them do you have)? Thanks!

[iamhopaul123]

Hello, we just updated the description of this issue to be the design for customized env resources. We are planning to support customized VPC resources first, then security groups and adding more policies to environment manager role.

Please leave your feedback if you have some thoughts or suggestions on this feature or if you think we’re missing any use cases which you would love to see as a part of this feature.

[kohidave]

This feature will be in the next release 🎊

[efekarakus]

Heya! With v0.3.0 the env init command now asks you if you'd like to use an existing VPC and subnets. Please cut us a new issue, if you'd like to import any other resources from your VPC to the environment! Thanks :)