Serverless-application-model: AWS::Lambda::Permission for log-group subscription filter fails to apply lambda:InvokeFunction action on CloudWatch logs

[rvypandey]

Hi team,

Description:

I'm trying to automate my stack by doing:

• Create Function
• Create ApiGateway
• Create LogGroup
  ------------------------> need AWS::Lambda::Permission here (for complete automation)
• Create a subscription filter for the logGroup to stream the logs to a different lambda.

Problem Statement:

When I try the 4th step inside the template, the stack fails by saying:

Template.yml:

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31

Parameters:
  executionRole:
    Type: String
    Default: execution_role
  s3Bucket:
    Type: String
    Default: functions_bucket
  s3ObjectKey:
    Type: String
  functionName:
    Type: String
    Default: poc_lambda
  logCollectorLambdaName:
    Type: String
    Default: cloudwatch-log-collector-lambda
  stageName:
    Type: String
    Default: myTestStage
  aliasName:
    Type: String
    Default: test

Resources:
  ApiGatewayApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: !Sub ${stageName}
      DefinitionBody:
        swagger: "2.0"
        info:
          title: !Sub ${functionName}
        paths:
          "/commerce":
            get:
              responses: {}
              x-amazon-apigateway-integration:
                uri: !Sub "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${functionName}:${aliasName}/invocations"
                httpMethod: POST
                type: aws_proxy
  Commerce:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Ref functionName
      AutoPublishAlias: !Ref aliasName
      Description: "poc lambda"
      Handler: src/index.handler
      Runtime: nodejs6.10
      CodeUri:
        Bucket: !Ref s3Bucket
        Key: !Ref s3ObjectKey
      Events:
        Catalog:
          Type: Api
          Properties:
            RestApiId: !Ref ApiGatewayApi
            Method: GET
            Path: /commerce
      MemorySize: 128
      Timeout: 3
      Role: !Sub "arn:aws:iam::${AWS::AccountId}:role/${executionRole}"
      Tags:
        name: !Ref functionName
      AutoPublishAlias: !Ref aliasName
  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/lambda/${functionName}"
      RetentionInDays: 14
  LogGroupLambdaInvokePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${functionName}"
      Action: "lambda:InvokeFunction"
      Principal: !Sub "logs.${AWS::Region}.amazonaws.com"
      SourceArn: !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${functionName}:*"
      SourceAccount:
        Ref: "AWS::AccountId"
    DependsOn: "LogGroup"
  LogsSubscriptionFilter:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      LogGroupName: !Sub "/aws/lambda/${functionName}"
      FilterPattern: ""
      DestinationArn: !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${logCollectorLambdaName}"
    DependsOn: "LogGroupLambdaInvokePermission"

Research:

I read a lot of blogs that implement this design -

1. https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html
2. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-permission.html#cfn-lambda-permission-sourceaccount _which is a subset of_ https://docs.aws.amazon.com/lambda/latest/dg/API_AddPermission.html#SSS-AddPermission-request-SourceArn
3. https://medium.com/@Prestomation/serverless-slackops-real-time-linking-to-logs-when-your-lambda-function-crashes-edbd8e0353e1
4. http://blog.davidablack.net/2016/08/17/a-log-reflector-for-aws-lambda/
5. https://rockytech.blogspot.com/2017/02/massively-integrating-aws-cloudwatch.html

went through some of the open/closed issues in SAM -

• https://github.com/awslabs/serverless-application-model/issues/144, https://github.com/awslabs/serverless-application-model/issues/257 and few others (that talk about AWS::Lambda::Permission implementation).

and also did more than 50 Cloudformation deployments with different combinations to make this work.

Workaround:

The only workaround for having the subscription filter in the template and still make this work, is to:

1. Remove the subscription filter on the first deployment. This will only create the logGroup (because that's in the template).
2. After this, you can add the subscription filter manually by navigating to Cloudwatch and putting the subscription on the logGroup. AWS will create the AWS::Lambda::Permission for you, which you cannot view (?).
3. Once the subscription filter is in place, you can add back the code in the template that creates the subscription filter.
4. On all the subsequent runs, even if the stack is cleared out and recreated, the subscription filter permission is intact with AWS and you don't get the error message anymore.

In short, you won't need the AWS::Lambda::Permission in the template anymore, because AWS has that automatically created for you.

#

It would be very helpful if anyone can point out the issue (if any) in the template, OR assure me that this is a bug, and that I am missing something obvious.

Thanks,
Ravi Pandey.

[rvypandey]

The issue got resolved after opening a ticket with AWS support.
The problem was that the FunctionName under LogGroupLambdaInvokePermission is not the lambda function that is generating the log, but the one ingesting it. Just by changing that everything falls in place. It also makes sense that the FunctionName should be the destination arn.

I should have tried this before raising this ticket (as I already tried many other), but the reason I failed to do this was because I got a bit lost because of the documentation on this feature (steps 5 and 6 specify different log-groups). I might be misunderstanding the docs, and might have missed something straight-forward.

Anyways, this issue is resolved!