Algo: Fails to deploy EC2 cloudformation template

[jtnt]

OS / Environment

MacOS 10.12.4

Ansible version

ansible 2.2.0.0

Version of components from requirements.txt

Name: msrestazure
Version: 0.4.7

Name: setuptools
Version: 35.0.1

Name: dopy
Version: 0.3.5

Name: boto
Version: 2.46.1

Name: boto3
Version: 1.4.4

Name: azure
Version: 2.0.0rc5

Name: msrest
Version: 0.4.1

Name: apache-libcloud
Version: 1.5.0

Name: six
Version: 1.10.0

Name: pyOpenSSL
Version: 17.0.0

Name: Jinja2
Version: 2.8

Summary of the problem

Deployment fails with during cloudformation template deployment on Amazon EC2:

Steps to reproduce the behavior

1. Download latest version of Algo.
2. Go through full deployment procedure for MacOS.
3. Run Algo.
4. Get error

Note that I used Amazon root access keys during deployment.

The way of deployment (cloud or local)

Cloud: Amazon EC2

Expected behavior

Cloudformation deployment complete successfully

Actual behavior

See above

Full log

(env) NMP:algo-master jtnt$ ./algo

What provider would you like to use?
1. DigitalOcean
2. Amazon EC2
3. Microsoft Azure
4. Google Compute Engine (only for testing, see issue #369)
5. Install to existing Ubuntu 16.04 server

Enter the number of your desired provider
: 2

Enter your aws_access_key (http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html)
Note: Make sure to use an IAM user with an acceptable policy attached (see https://github.com/trailofbits/algo/blob/master/docs/deploy-with-ansible.md).
[pasted values will not be displayed]
Enter your aws_secret_key (http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html)
[pasted values will not be displayed]
Name the vpn server:
What region should the server be located in?
1. us-east-1 US East (N. Virginia)
2. us-east-2 US East (Ohio)
3. us-west-1 US West (N. California)
4. us-west-2 US West (Oregon)
5. ap-south-1 Asia Pacific (Mumbai)
6. ap-northeast-2 Asia Pacific (Seoul)
7. ap-southeast-1 Asia Pacific (Singapore)
8. ap-southeast-2 Asia Pacific (Sydney)
9. ap-northeast-1 Asia Pacific (Tokyo)
10. eu-central-1 EU (Frankfurt)
11. eu-west-1 EU (Ireland)
12. eu-west-2 EU (London)
13. ca-central-1 Canada (Central)
14. sa-east-1 São Paulo
Enter the number of your desired region:
Do you want macOS/iOS clients to enable "VPN On Demand" when connected to cellular networks?
Do you want macOS/iOS clients to enable "VPN On Demand" when connected to Wi-Fi?
List the names of trusted Wi-Fi networks (if any) that macOS/iOS clients exclude from using the VPN (e.g., your home network. Comma-separated value, e.g., HomeNet,OfficeWifi,AlgoWiFi)
:

Do you want to install a DNS resolver on this VPN server, to block ads while surfing?
Do you want each user to have their own account for SSH tunneling?
Do you want to apply operating system security enhancements on the server? (warning: replaces your sshd_config)
Do you want the VPN to support Windows 10 or Linux Desktop clients? (enables compatible ciphers and key exchange, less secure)
Do you want to retain the CA key? (required to add users in the future, but less secure)
PLAY [Configure the server] ****************

TASK [setup] *********************
ok: [localhost]

TASK [Generate the SSH private key] **************
changed: [localhost -> localhost]

TASK [Generate the SSH public key] ***************
ok: [localhost -> localhost]

TASK [Change mode for the SSH private key] ***********
ok: [localhost -> localhost]

TASK [Ensure the dynamic inventory exists] ***********
changed: [localhost]

TASK [cloud-ec2 : set_fact] ****************
ok: [localhost]

TASK [cloud-ec2 : Locate official AMI for region] **********
ok: [localhost]

TASK [cloud-ec2 : set_fact] ****************
ok: [localhost]

TASK [cloud-ec2 : Make a cloudformation template] **********
changed: [localhost]

TASK [cloud-ec2 : Deploy the template] *************
fatal: [localhost]: FAILED! => {"changed": true, "events": ["StackEvent AWS::CloudFormation::Stack algopvpn1 ROLLBACK_COMPLETE", "StackEvent AWS::EC2::VPC VPC DELETE_COMPLETE", "StackEvent AWS::EC2::InternetGateway InternetGateway DELETE_COMPLETE", "StackEvent AWS::CloudFormation::Stack algopvpn1 ROLLBACK_IN_PROGRESS", "StackEvent AWS::EC2::VPC VPC CREATE_FAILED", "StackEvent AWS::EC2::VPC VPC CREATE_IN_PROGRESS", "StackEvent AWS::EC2::InternetGateway InternetGateway CREATE_FAILED", "StackEvent AWS::EC2::InternetGateway InternetGateway CREATE_IN_PROGRESS", "StackEvent AWS::CloudFormation::Stack algopvpn1 CREATE_IN_PROGRESS"], "failed": true, "output": "Problem with CREATE. Rollback complete", "stack_outputs": {}, "stack_resources": [{"last_updated_time": null, "logical_resource_id": "InternetGateway", "physical_resource_id": null, "resource_type": "AWS::EC2::InternetGateway", "status": "DELETE_COMPLETE", "status_reason": null}, {"last_updated_time": null, "logical_resource_id": "VPC", "physical_resource_id": null, "resource_type": "AWS::EC2::VPC", "status": "DELETE_COMPLETE", "status_reason": null}]}

PLAY RECAP ***********************
localhost : ok=9 changed=3 unreachable=0 failed=1

[dguido]

Can you describe the permissions of the user you gave Algo the access keys for? Is it a root user or an IAM user? If it's an IAM user can you show us the access policy for it?

[dguido]

@aboutte Any chance you know off the top of your head why this might be happening?

[jtnt]

I was using root access keys.

[dguido]

jtnt: We just switched to Cloudformation templates this week. Can you try with an IAM user and the specified policy attached, as described in the installer, and try again? I wonder if the issue is because you're using root keys.

[jtnt]

Sure. What permissions do I need to give the IAM user?

[jtnt]

I just tried again with an IAM user with AdministratorAccess, and got the same result.

[aboutte]

It looks like the InternetGateway resource is what failed but the error does not include why.

@jtnt have you deleted this failed deployment? If not can you please:

• log into the AWS console
• go to the CloudFormation service
• select the failed deployment
• click on the Resources tab
• scroll down till you see CREATE_FAILED
• send me the details of all resources that have a status of CREATE_FAILED

[aboutte]

I just tested master (commit 31d6bd3) in us-east-1 like you did (I primarily use us-west-2) and my CloudFormation stack deployed successfully.

[patmcd]

+1 for a failure for me on the CFN deployment as well.

23:30:20 UTC-0400 CREATE_FAILED AWS::EC2::Route Route Route did not stabilize in expected time
23:30:19 UTC-0400 CREATE_FAILED AWS::EC2::Route RouteIPv6 Route did not stabilize in expected time

[aboutte]

@patmcd what region did you deploy to? And can you send me your CloudFormation template ([email protected])? It is located in algo/configs/${name provided during deploy}.yml.

[patmcd]

@aboutte us-east 1.

Worked after I wiped everything.

CFN sent.

[jtnt]

@aboutte Here are the events that failed:

22:58:34 UTC-0400 CREATE_FAILED AWS::EC2::VPC VPC The maximum number of VPCs has been reached.
22:58:34 UTC-0400 CREATE_FAILED AWS::EC2::InternetGateway InternetGateway The maximum number of internet gateways has been reached.

[patmcd]

@jtnt you hit your AWS Soft Limits. You either have to delete a VPC or Internet Gateway that you are not using, or contact support to increase your limit. Or even deploy to another region for testing.

[aboutte]

@jtnt take a look at this documentation if you are not famailiar with the limits and this document for increasing service limits.

@dguido this can be closed.

[jtnt]

Thanks, guys. Once I saw the errors, I started poking around and found the VPCs, etc. AWS newbie, so don't know what they are in the context of instances, etc. or that Algo created them. :)

[aboutte]

@jtnt every AWS resource created by Algo should be tagged with Environment => Algo for easy identification. FYI.

[dguido]

Great, thanks for the help @aboutte! I'll incorporate more of this into the documentation later this week.