Chalice: [proposal] Add a new cfn/SAM based deployment backend

Created on 28 Nov 2016 · 12Comments · Source: aws/chalice

This is a proposal to add a new deployment backend that’s based on cloudformation and the AWS Serverless Application Model (AWS SAM). This addresses several points of customer feedback and makes several feature requests easier to implement.

Motivation

When chalice deploys your application, it uses boto3, the AWS SDK for Python, to make the necessary coordinated API calls to IAM, Lambda, and API Gateway. There’s several benefits with this approach. Deployments are fast as we're making direct API calls to API gateway and Lambda. Nothing to setup (including S3 buckets because we send the zip file contents directly to lambda).

However, there’s several customer issues that I’d like to address:

I think that most of these issues can be addressed by leveraging the serverless application model along with cloudformation stacks. Consider if the whole deployment was broken down into into two explicit and separable components:

Generate assets that describe what to deploy
Take those locally generated assets and deploy them to AWS.

This gives users insight into what we’re about to deploy, allows them to customize what’s going to be deployed (by manipulating the manifest), allows them to see a list of resources that were created, and also makes deletion trivial.

Specification

Customers will now have the option of creating a chalice application with the new cfn deployment backend (the defaults will remain unchanged). This choice must be made at project creation time. It will not be possible to migrate from the current deployment backend to the cfn deployment backend.

The eventual goal would be to make this the default (and maybe only) deployment backend.

A new parameter is added to the new-project command that indicates you want to use Serverless Application Model templates: chalice new-project --use-sam-templates --s3-path. Note you'll also have to specify an S3 location where the deployment package can be uploaded (this is needed when using SAM/cfn).

Once you’ve created a new application with SAM templates enabled, the chalice deploy command will internally use SAM templates and cloud formation to deploy your application.

There will also be a new chalice package command, which can only be used for projects created with --use-sam-templates. This command creates local assets for your chalice app that can then be deployed at a later time (or even by other tools that can work with SAM templates). This package command takes an output directory and writes the following files to that directory:

app.json - The generated SAM template
deployment.zip - The deployment zip file to send to lambda

In order to create a lambda function with cfn, we’ll need to upload your deployment package to S3 and update the app.json with a path to that remote path. By default, the chalice package command does not make any remote
API calls, but you can provide the --remote-assets flag to indicate you’d like chalice to handle uploading your lambda deployment package to S3 for you. It will also take care of updating the SAM template with a reference to that S3 location when --remote-assets is specified.

Example

Here’s an example of using this new functionality to create a deployment package that can be deployed with the AWS CLI using the newly introduced aws cloudformation package/deploy commands:

$ chalice new-project —use-sam-templates cfn-based-app --s3-path s3://bucket/

# This will generate a SAM template, and upload your deployment package to S3
$ chalice package --remote-assets /tmp/deploy

# You can now deploy the app using your preferred cfn tools
# Here we're using the AWS CLI to deploy our app:
$ aws cloudformation deploy 
        --template-file /tmp/deploy/app.json \
        --stack-name cfn-based-app --capabilities CAPABILITIES_IAM

# We can grab the endpoint url from the stack outputs:
$ aws cloudformation describe-stacks \
        --stack-name cfn-based-app \
        --query "Stacks[0].Outputs[?OutputKey=='EndpointUrl'] | [0].OutputValue" \
        --output text
https://api-id.execute-api.us-west-2.amazonaws.com/dev/

Future enhancements

One other thing I'd like to add in the future is the ability for users to modify the existing SAM template, possibly with their own application specific resources. A really simple example would be to allow them to specify additional cfn/SAM resources that can merged together with the template we generate:

# Project root directory (where app.py is located):
$ cat app.json
{
  "Resources" : {
    "myTable" : {
      "Type" : "AWS::DynamoDB::Table",
      "Properties" : {
        "AttributeDefinitions" : [...],
        "TableName": "mytable"
        ...
      }
    }
  }
}

proposals

Source

jamesls

👍11

Most helpful comment

Following up here, this is now available in the 0.8.0 release of chalice, via the new chalice package command: chalice package /tmp/packaged-dir. I'll keep this open until I finish writing docs for this, but you it's available if you want to try it out.

jamesls on 1 May 2017

🎉3 👍1

All 12 comments

@jamesls

I like the direction of making the chalice compatible with the other tools like the CLI that support SAM. One question that I do have is for the package command when you do not specify --remote-assets how difficult is it to modify your app.json to point to the s3 bucket that you upload it to? The only reason I ask is because I wonder if the package command should be uploading to s3 by default?

Also is the --s3-path argument required for new-project? And if so, if you do not specify it, does the workflow change in getting your application deployed and if so, how?

kyleknap on 28 Nov 2016

how difficult is it to modify your app.json to point to the s3 bucket that you upload it to?

Are you asking how hard it is to manually update? If so, there's only one line you change.
In general though, you would be able to leverage the aws cloudformation package command to handle that for you. It generates a reference to the local deployment package path.

$ chalice package --local-assets /tmp/packaged
$ cd /tmp/packaged
$ ls -la
-rw-r--r--   1 jamessar  wheel     8547 Nov 29 11:12 app.json
-rw-r--r--   1 jamessar  wheel  3358734 Nov 29 11:12 deployment.zip

$ jp Resources.APIHandler < app.json
{
  "Properties": {
    "CodeUri": "./deployment.zip",
    "Events": {
      "indexget": {
        "Properties": {
          "Method": "get",
          "Path": "/",
          "RestApiId": {
            "Ref": "RestAPI"
          }
        },
        "Type": "Api"
      }
    },
    "Handler": "app.app",
    "Runtime": "python2.7"
  },
  "Type": "AWS::Serverless::Function"
}

# Then to deploy:

$ aws cloudformation package ...
$ aws cloudformation deploy ...

Also is the --s3-path argument required for new-project?

Yeah, I'll update the issue.

jamesls on 29 Nov 2016

Ahh that makes sense did not realize you could do:

chalice package --local-assets /tmp/packaged
aws cloudformation package --template-file app.json --s3-bucket mybucket

That's nice. Seems straightforward.

kyleknap on 29 Nov 2016

Love the direction +1

pauldeden on 2 Dec 2016

This all looks very good. What my team needs soon will be the ability to deploy via update rather than delete/add in APIG and deploying to prod rather than dev. The latter is easy to workaround for now, at least.