Serverless-application-model: [Feature request] Support for YAML anchors in SAM/CF templates

[bluepeter]

Any chance we can get YAML anchor support for SAM?

As it is, YAML anchors are supported in Amazon Swagger S3 files referenced in SAM files via, e.g., DefinitionUri. This is awesome as it lets us very succinctly define new API Gateway endpoints. For example:

    /forgot:                                                                                                      
      put:
        <<: *scaffold                                                                                             
        x-amazon-apigateway-integration:                                                                          
          <<: *integration
          uri: "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:XXX/invocations"                                                                   
      options: *options

Beyond Swagger YAML anchor support, anchors don't work in CF/SAM files currently.

[bluepeter]

I figured out a way to get YAML anchors working in CloudFormation templates. Sweet! Here's how:

You cannot include YAML anchors directly in your primary CF template (the one you are directly deploying). Doing so will yield: "An error occurred (ValidationError) when calling the CreateChangeSet operation: Template error: YAML aliases are not allowed in CloudFormation templates"

However, you can use YAML anchors in imported template snippets. Then, in the deployable primary CF file, you bring in the snippet (with YAML anchors) as follows:

Resources:
  'Fn::Transform':
    Name: 'AWS::Include'
    Parameters:
      Location: !Sub s3://xxx/MyYamlAnchoredResources.yaml

Anchors allow CF templates to become radically smaller.

[bluepeter]

The benefit? Radically shrink CF definitions:

Before:

  /forgot:
  ¦ put:
  ¦ ¦ produces:
  ¦ ¦ - "application/json"
  ¦ ¦ responses:
  ¦ ¦ ¦ 200:
  ¦ ¦ ¦ ¦ description: "200 response"
  ¦ ¦ ¦ ¦ schema:
  ¦ ¦ ¦ ¦ ¦ $ref: "#/definitions/Empty"
  ¦ ¦ ¦ ¦ headers:
  ¦ ¦ ¦ ¦ ¦ Access-Control-Allow-Origin:
  ¦ ¦ ¦ ¦ ¦ ¦ type: "string"
  ¦ ¦ x-amazon-apigateway-integration:
  ¦ ¦ ¦ responses:
  ¦ ¦ ¦ ¦ default:
  ¦ ¦ ¦ ¦ ¦ statusCode: "200"
  ¦ ¦ ¦ uri: "arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:xxx:function:BigOnion_forgotAboutDre/invocations"
  ¦ ¦ ¦ passthroughBehavior: "when_no_match"
  ¦ ¦ ¦ httpMethod: "POST"
  ¦ ¦ ¦ contentHandling: "CONVERT_TO_TEXT"
  ¦ ¦ ¦ type: "aws_proxy"
  ¦ options:
  ¦ ¦ consumes:
  ¦ ¦ - "application/json"
  ¦ ¦ produces:
  ¦ ¦ - "application/json"
  ¦ ¦ responses:
  ¦ ¦ ¦ 200:
  ¦ ¦ ¦ ¦ description: "200 response"
  ¦ ¦ ¦ ¦ schema:
  ¦ ¦ ¦ ¦ ¦ $ref: "#/definitions/Empty"
  ¦ ¦ ¦ ¦ headers:
  ¦ ¦ ¦ ¦ ¦ Access-Control-Allow-Origin:
  ¦ ¦ ¦ ¦ ¦ ¦ type: "string"
  ¦ ¦ ¦ ¦ ¦ Access-Control-Allow-Methods:
  ¦ ¦ ¦ ¦ ¦ ¦ type: "string"
  ¦ ¦ ¦ ¦ ¦ Access-Control-Allow-Headers:
  ¦ ¦ ¦ ¦ ¦ ¦ type: "string"
  ¦ ¦ x-amazon-apigateway-integration:
  ¦ ¦ ¦ responses:
  ¦ ¦ ¦ ¦ default:
  ¦ ¦ ¦ ¦ ¦ statusCode: "200"
  ¦ ¦ ¦ ¦ ¦ responseParameters:
  ¦ ¦ ¦ ¦ ¦ ¦ method.response.header.Access-Control-Allow-Methods: "'OPTIONS,PUT'"
  ¦ ¦ ¦ ¦ ¦ ¦ method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
  ¦ ¦ ¦ ¦ ¦ ¦ method.response.header.Access-Control-Allow-Origin: "'*'"
  ¦ ¦ ¦ passthroughBehavior: "when_no_match"
  ¦ ¦ ¦ requestTemplates:
  ¦ ¦ ¦ ¦ application/json: "{\"statusCode\": 200}"
  ¦ ¦ ¦ type: "mock"

After...

  /forgot:
  ¦ put:
  ¦ ¦ <<: *scaffold
  ¦ ¦ x-amazon-apigateway-integration:
  ¦ ¦ ¦ <<: *functionCode
  ¦ ¦ ¦ url:
  ¦ ¦ ¦ ¦ Fn::Sub: arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:BigOnion_forgotAboutDre/invocations
  ¦ options: *options

[sanathkr]

TL;DR - Do aws cloudformation package followed by aws cloudformation deploy to support Anchors :)

So here is the deal. YAML Anchors are not supported in AWS CloudFormation. They are explicitly blacklisted by CloudFormation, which we can't change.

But if you use the aws cloudformation package command before calling deploy, it will read the YAML file using regular YAML parser which will support anchors. The output of package command will be a flat file without anchors, but at least the code you write will support anchors. package is useful to bundle Lambda function code artifacts to S3. Even if you don't use any of the package functionality (ie. set CodeUri to S3 URI yourself), you can run the package command to produce a no-op output but flatten anchors.

[sanathkr]

Closing this issue as it is technically not possible to support this in SAM. Feel free to open if you have questions.

[mruckli]

The suggested workaround by @sanathkr seems not to work for me. The aws cloudformation package command does create a separate and flat yaml but it still contains my anchors.
I use the anchors inside my swagger api gateway doc. Might it be impossbile in that scenario?

[jjmschofield]

@mruckli is correct @sanathkr - package simply replaces &anchorName with something like &id001

[jjmschofield]

@bluepeter - did you get any joy combining your technique with aws cloudformation package?

[bluepeter]

I am using YAML anchors without issue in the solution I specified above. I have not tried the solution proposed by @sanathkr as mine is working well as is.

Here's more detail on what I am doing. The core CloudFormation template has next to nothing in it. Besides a few other scaffolding items, all it does is AWS::Include another like so in file BigOnion.yaml (or whatever else you want to call it):

Resources:
  Fn::Transform:
    Name: AWS::Include
    Parameters:
      Location: !Sub s3://my-s3-location-${AWS::AccountId}-${AWS::Region}/resources.yaml

Then, in resources.yaml (or whatever else you call it), I can use YAML to radically reduce CloudFormation templates. Here's an example of a Lambda function definition:

accountDelete:
  <<: *lambdaDefaults
  Properties:
    <<: *lambdaProps
    FunctionName: myLambdaFunction

Sweet!

And just wait to you can boil down API Gateway endpoints to 3 lines! (Which you can w/ prudent use of YAML anchors.)

Then, (and assuming you're using Node.js) to deploy, I add the following in package.json to simplify the command line CloudFormation deploy step:

  "scripts": {
    "cf-deploy": "./bin/update-resources.sh && aws --profile MyAWSProfileName cloudformation deploy --stack-name MyStackName --template-file CloudFormation/BigOnion.yaml --capabilities CAPABILITY_IAM"
  }

In the referenced bin/update-resources.sh I have the following:

#!/usr/bin/env bash

# Creates the S3 deployment helper bucket if it doesn't already exist.
# Uploads resources.yaml to the bucket.

PROFILE=${1:-BigOnion}
echo "Using AWS profile ${PROFILE}"
MY_REGION=`aws --profile ${PROFILE} configure get region`
echo "AWS region is ${MY_REGION}"
ACCOUNT_ID=`aws --profile ${PROFILE} sts get-caller-identity --output text --query "Account"`
echo "AWS account id is ${ACCOUNT_ID}"
S3="my-s3-location-${ACCOUNT_ID}-${MY_REGION}"
echo "AWS S3 is ${S3}"

aws --profile ${PROFILE} s3 mb s3://$S3
aws --profile ${PROFILE} s3 cp CloudFormation/resources.yaml s3://$S3/resources.yaml
if [ ! $? -eq 0 ]; then
  echo "FAILED"
  exit
fi

You can probably figure out what is going on here. This script basically creates the S3 bucket if it doesn't already exist, and uploads the new/updates resources.yaml there.

As a result, I can edit resources.yaml which contains all my YAML anchor'd stuff, and when I am ready to deploy, I just type: yarn run cf-deploy. This will create the S3 bucket (in my case I include region and account ID as I am using this across multiple regions and accounts), upload the new resources.yaml file which is referenced in BigOnion.yaml, and then deploy/update a new CF stack using the aforementioned in BigOnion.yaml

YMMV.

[vjeffz]

If you call the YAML anchor Metadata then it will generate a valid CFN key in the output YAML that can be ignored.

For e.g the following cloudformation with anchors:

---

Metadata: &default_vars
  ENV: production
  BUCKET: s3-prod-blah
  DB_HOST: prod-reader.db.com
  DB_PORT: 3306

Metadata: &properties
  Code:
    ZipFile: |
      var aws = require('aws-sdk')
      var response = require('cfn-response')
      exports.handler = function(event, context) {
          console.log("hello")
      }
  Description: My App Function
  Handler: "function.lambda_handler"
  Environment:
    Variables:
      << : *default_vars
  Role: !GetAtt MyDBReaderFunctionRole.Arn
  Runtime: "python3.6"
  Timeout: 60

Metadata: &role_properties
  ManagedPolicyArns:
    - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
  AssumeRolePolicyDocument:
    Version: "2012-10-17"
    Statement:
      -
        Effect: "Allow"
        Principal:
          Service:
            - "lambda.amazonaws.com"
        Action:
          - "sts:AssumeRole"
  Path: "/"

Resources:

  MyDBWriterFunction:
    Type: AWS::Lambda::Function
    Properties:
      <<: *properties
      Role: !GetAtt MyDBWriterFunctionRole.Arn
      Environment:
        Variables:
          << : *default_vars
          DB_HOST: prod-writer.db.com

  MyDBReaderFunction:
    Type: AWS::Lambda::Function
    Properties:
      <<: *properties

  MyDBWriterFunctionRole:
    Type: "AWS::IAM::Role"
    Properties:
      <<: *role_properties

  MyDBReaderFunctionRole:
    Type: "AWS::IAM::Role"
    Properties:
      <<: *role_properties

When run with aws cloudformation package this will generate:

Metadata:
  ManagedPolicyArns:
  - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
  AssumeRolePolicyDocument:
    Version: '2012-10-17'
    Statement:
    - Effect: Allow
      Principal:
        Service:
        - lambda.amazonaws.com
      Action:
      - sts:AssumeRole
  Path: /
Resources:
  MyDBWriterFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: "var aws = require('aws-sdk')\nvar response = require('cfn-response')\n\
          exports.handler = function(event, context) {\n    console.log(\"hello\"\
          )\n}\n"
      Description: My App Function
      Handler: function.lambda_handler
      Environment:
        Variables:
          ENV: production
          BUCKET: s3-prod-blah
          DB_HOST: prod-writer.db.com
          DB_PORT: 3306
      Role:
        Fn::GetAtt:
        - MyDBWriterFunctionRole
        - Arn
      Runtime: python3.6
      Timeout: 60
  MyDBReaderFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: "var aws = require('aws-sdk')\nvar response = require('cfn-response')\n\
          exports.handler = function(event, context) {\n    console.log(\"hello\"\
          )\n}\n"
      Description: My App Function
      Handler: function.lambda_handler
      Environment:
        Variables:
          ENV: production
          BUCKET: s3-prod-blah
          DB_HOST: prod-reader.db.com
          DB_PORT: 3306
      Role:
        Fn::GetAtt:
        - MyDBReaderFunctionRole
        - Arn
      Runtime: python3.6
      Timeout: 60
  MyDBWriterFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: /
  MyDBReaderFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      ManagedPolicyArns:
      - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: /

You can now use this with aws cloudformation deploy or aws cloudformation create-stack. It'll save you having to use Fn::Transform

[bluepeter]

@vjeffz thanks for the update... is this officially documented somewhere, or just another "side-effect" (like my original "discovery" of Fn::Transform?

[vjeffz]

@bluepeter not officially documented anywhere, just a simple workaround I discovered :)