Cookiecutter-django: Deploying to beanstalk - elasticache issues.

[hairychris]

Hey =)

I understand that Beanstalk support is experimental and I'm not an expert with it by any means. That said I was hoping to understand why I can't get this deployed to Beanstalk. Hopefully by documenting my progress here I'll either solve my own issue and put in a pull request or catch the passing attention of someone who knows more about this sort of stuff.

Expected Behaviour

The project successfully deploys to Elastic Beanstalk on completion of the steps outlined here.

Actual Behaviour

An error occurs during the EB step, the output on the console is below (ignore redacted):

Creating application version archive "app-161212_<redacted>".
Uploading <redacted>/app-161212_<redacted>.zip to S3. This may take a while.
Upload Complete.
Environment details for: <redacted>
  Application name: <redacted>
  Region: us-west-2
  Deployed Version: app-161212_<redacted>
  Environment ID: e-mp<redacted>
  Platform: 64bit Amazon Linux 2016.09 v2.2.0 running Python 3.4
  Tier: WebServer-Standard
  CNAME: UNKNOWN
  Updated: 2016-12-12 13:26:50.261000+00:00
Printing Status:
INFO: createEnvironment is starting.
INFO: Using elasticbeanstalk-us-west-2-709<redacted> as Amazon S3 storage bucket for environment data.
INFO: Created load balancer named: awseb-e-m-AWSEBLoa-1EOS48<redacted>
INFO: Environment health has transitioned to Pending. Initialization in progress (running for 34 seconds). There are no instances.
INFO: Created security group named: awseb-e-mpt8dsnutp-stack-AWSEBSecurityGroup-1P<redacted>
INFO: Created Auto Scaling launch configuration named: awseb-e-<redacted>tp-stack-AWSEBAutoScalingLaunchConfiguration-VMX<redacted>
ERROR: Stack named 'awseb-e-mp<redacted>-stack' aborted operation. Current state: 'CREATE_FAILED'  Reason: The following resource(s) failed to create: [MyElastiCache].
INFO: Created security group named: awseb-e-mpt<redacted>tp-stack-MyCacheSecurityGroup-6W<redacted>
INFO: Launched environment: <redacted>. However, there were issues during launch. See event log for details.
 -- Events -- (safe to Ctrl+C)

More information is available on the Elastic Beanstalk console:

2016-12-12 13:28:00 UTC+0000    ERROR   Stack named 'awseb-e-mpt<redacted>-stack' aborted operation. Current state: 'CREATE_FAILED' Reason: The following resource(s) failed to create: [MyElastiCache].

If we go to the CloudFormation console there is some better debugging info:

13:27:51 UTC+0000   CREATE_FAILED   AWS::CloudFormation::Stack  awseb-e-mpt<redacted>-stack The following resource(s) failed to create: [MyElastiCache].

13:27:50 UTC+0000   CREATE_FAILED   AWS::ElastiCache::CacheCluster  MyElastiCache   Cache subnet group name must be specified along with security group IDs.

From this I looked at the amazon documentation for AWS::ElastiCache::CacheCluster which led me to the following config:

MyCacheSubnetGroup: 
    Type: "AWS::ElastiCache::SubnetGroup"
    Properties: 
      Description: "Cache Subnet Group"
      SubnetIds: 
        - Ref: "Subnet1"

CacheSubnetGroupName:
        Ref: "MyCacheSubnetGroup"

At this point I know I'm fast disappearing down the creating a VPC and Subnets etc so let's try and do that programmatically:

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.4.0.0/16
      EnableDnsHostnames: 'true'
      EnableDnsSupport: 'true'

  Subnet1:
      Type: AWS::EC2::Subnet
      DependsOn:
        - VPC
      Properties:
        CidrBlock: 10.4.0.0/24
        MapPublicIpOnLaunch: true
        VpcId:
          Ref: "VPC"

  MyCacheSecurityGroup:
    Type: "AWS::EC2::SecurityGroup"
    Properties:
      GroupDescription: "Lock cache down to webserver access only"
      SecurityGroupIngress :
        - IpProtocol : "tcp"
          FromPort :
            Fn::GetOptionSetting:
              OptionName : "CachePort"
              DefaultValue: "6379"
          ToPort :
            Fn::GetOptionSetting:
              OptionName : "CachePort"
              DefaultValue: "6379"
          SourceSecurityGroupName:
            Ref: "AWSEBSecurityGroup"

  MyCacheSubnetGroup: 
    Type: "AWS::ElastiCache::SubnetGroup"
    Properties: 
      Description: "Cache Subnet Group"
      SubnetIds: 
        - Ref: "Subnet1"

  MyElastiCache:
    Type: "AWS::ElastiCache::CacheCluster"
    Properties:
      CacheNodeType:
        Fn::GetOptionSetting:
          OptionName : "CacheNodeType"
          DefaultValue : "cache.t1.micro"
      NumCacheNodes:
        Fn::GetOptionSetting:
          OptionName : "NumCacheNodes"
          DefaultValue : "1"
      Engine:
        Fn::GetOptionSetting:
          OptionName : "Engine"
          DefaultValue : "redis"
      VpcSecurityGroupIds:
        -
          Fn::GetAtt:
            - MyCacheSecurityGroup
            - GroupId
      CacheSubnetGroupName:
        Ref: "MyCacheSubnetGroup"

Outputs:
  ElastiCache:
    Description : "ID of ElastiCache Cache Cluster with Redis Engine"
    Value :
      Ref : "MyElastiCache"

Unfortunately I can't get this working so I decided to cheat and use the GUI to setup a VPC with a public and a private subnet. The following ID is the ID of the private subnet, I replaced this block:

MyElastiCacheSubnetGroup: 
    Type: "AWS::ElastiCache::SubnetGroup"
    Properties: 
      Description: "Cache Subnet Group"
      SubnetIds: 
        - "subnet-0a54<redacted>"

Now I seem to get an error complaining about security groups:

16:00:11 UTC+0000    CREATE_FAILED    AWS::ElastiCache::CacheCluster    MyElastiCache    Cannot use EC2-Classic security group sg-2e<redacted> with a VPC cluster.

I then started getting these errors and gave up:

    16:25:48 UTC+0000    CREATE_FAILED    AWS::CloudFormation::Stack    awseb-e-<redacted>-stack    The following resource(s) failed to create: [MyCacheSecurityGroup].
Physical ID:arn:aws:cloudformation:eu-west-1:709<redacted>:stack/awseb-e-cmf<redacted>-stack/d9f11940-62-16-8b<redacted>5cd2
16:25:46 UTC+0000    CREATE_FAILED    AWS::EC2::SecurityGroup    MyCacheSecurityGroup    Invalid id: "awseb-e-cmfrp<redacted>p-stack-AWSEBSecurityGroup-1PPS<redacted>8" (expecting "sg-...")
Physical ID:sg-e<redacted>
16:25:45 UTC+0000    CREATE_IN_PROGRESS    AWS::EC2::SecurityGroup    MyCacheSecurityGroup    Resource creation Initiated
Physical ID:sg-e8<redacted>

If anyone has any guidance or ideas on how to get this deployed please let me know. I'm off to learn a bit more about what is going on here and I will update this issue as I understand more. Thanks for reading this far.

[hairychris]

This now works for me without making any changes to the code in the repository. I'm confused but thankful. Sorry to waste your time. If anyone else gets stuck with this feel free to reopen / tag me and I will try and help out.

[mihow]

Hello, I'm receiving the same error, exactly as you described it! I'm testing with a fresh cookiecutter-django project.

Do you have any new ideas on this? @pydanny? Should Elastic Beanstalk create a VPC for you without any additional configuration?

Thanks for any help!

[pydanny]

AFAIK Elastic Beanstalk shouldn't be creating VPNs without you giving instructions. That said, because we've been moving more and more stuff to AWS Lambda, it's been 2 months since I worked with EB. In fact, if I get the time, I was going to convert the EB support of Cookiecutter Django to use @miserlou's excellent Zappa instead.

[mihow]

Hey deployment with Zappa would be awesome. I have a cookiecutter-django project using it now.

I believe the issue above might have something to do with the age of the AWS account and if it is using EC2 Classic or not. I am able to get it working on a newer test account that uses the "default VPC", but not on my client's account that uses EC2 Classic.

[kaidokert]

There's a zappa skeleton project here :
https://github.com/narfman0/cookiecutter-mobile-backend