Serverless-application-model: Create a LogGroup when AWS::Serverless::Function is parsed

[Nr18]

When you create a AWS::Serverless::Function and you want to have control over the retention of the invocation logs you now need to do something like this:

  FunctionLogGroup:
    Type: AWS::Logs::LogGroup
    DependsOn: [ Function ]
    Properties:
      LogGroupName: !Sub /aws/lambda/${Function}
      RetentionInDays: 7

  Function:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler

But it would be great to have something like this:

  Function:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      LogRetention: 7

[sanathkr]

Interesting. By default, you get a "infinite" log retention. Why do you reduce this period? Cost?

[Nr18]

For development environment you often will spin up the templates multiple times creating various log groups that will clutter your overview with (in time) non relevant log groups. Creating the Log Group from the CloudFormation template will remove the logs automatically.

For production environments it's always good to know what the retention is and explicitly setting it makes it real clear, if you want to archive them glacier would be a cheaper solution when dealing with large amounts of logs.

That being said if you decide to implement this you probably always want to deploy a LogGroup because if the log group has been created by an invocation you cannot create it in a update on a later time. So to keep the existing behavior a DeletionPolicy: retain (Not sure if this is supported) can be set and then you need to be able to control it something like:

Function: Type: AWS::Serverless::Function Properties: Handler: index.handler LogRetention: 7 LogDeletionPolicy: retain

[jfuss]

@Nr18 Do you think these Properties belong in the outer most level [1] or nested in on level [2]?

[1]

 Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      LogRetention: 7
      LogDeletionPolicy: retain

[2]

 Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Log: # Naming to be determined
        Retention: 7
        DeletionPolicy: retain

[jfuss]

Discussed on #sam-dev slack channel:
We landed on the only option to support this would be what @Nr18 suggested above (adding them to the Properties) but clearly document these new Properties will not work unless they are new functions. Existing stacks/functions through SAM that add these properties will fail due to the LogGroup already existing.

We should consider documenting a method to migrate existing functions to allow customers to use these new Properties. The migration will need to happen by customers though since SAM can't remove and re-add these confidently.

[Nr18]

@jfuss i would go for the nested option because now we have 2 options but i can imagine that CloudWatch logs will get KMS support or other useful options and in that case a specific block of options are easier to maintain.

Do you agree?

[jfuss]

Makes sense to me. Allows this feature to grow as the Log Group capabilities expand, which is important.

Ping on the #sam-dev slack channel if you are stuck or not sure where to start.

[thijsdev]

Hi there, any of you made any progress on this?

[Nr18]

He @thijsdev, i have not started yet... but will start shortly just rounding up some other stuff

[juve]

I was having occasional issues with the workaround for this issue where the dependency between the log group and the function was creating a race condition, viz:

1. CF begins deleting stack
2. CF deletes log group
3. Something triggers the lambda to run (a cloudwatch event in my case)
4. Lambda recreates the log group
5. CF deletes the lambda
6. CF finishes deleting stack
7. CF begins recreating stack
8. CF tries to create log group and fails because the log group already exists from step 4

In my case I was observing this issue when deleting and recreating the stack, but I suppose the same issue could happen on creation:

1. CF begins creating stack
2. CF creates lambda
3. Something triggers the new lambda to run
4. Lambda creates the log group
5. CF tries to create the log group and fails because the log group already exists from step 4

This happens because the log group depends on the lambda (in the proposed solution above), and because the function gets AWSLambdaBasicExecutionRole, which has logs:CreateLogGroup and enables lambda to create the log group.

I fixed the problem in my stack by inverting the dependency and being careful to set the function name and the log group name to match.

[ngmiller]

@juve You could probably solve that problem by assigning a role to the function that only allows logs:PutLogEvents and not logs:CreateLogGroup, logs:CreateLogStream, etc. This way the lambda would fail when the log group doesn't exist. Depending on your use case, this might not be desirable, but it seems reasonable that a lambda should fail if it's dependencies are not in order (including logging)

[jlhood]

@keetonian and I were discussing this issue this morning. As @jfuss said above, the primary blocker to making this change in SAM is that it's backwards-incompatible for existing stacks. If SAM added an AWS::Logs::LogGroup resource and you went to update an existing stack containing a Lambda function that had already created that log group, the AWS::Logs::LogGroup would fail on creation since the log group already exists.

An alternative solution we discussed is to create a custom resource that acts just like AWS::Logs::LogGroup, but has a "create or update" behavior where if the log group already exists, it will update its attributes to match what you pass in. While it isn't ideal for everyone to write this, one person could write this as a public SAR app and then others could use it via SAM nested apps.

Closing this issue since it's not a change we can make in SAM.

[hoang-innomizetech]

If we deploy the SAM from the beginning, we can add LogGroup resource to the template, but we need to use a fixed function name

ExampleFunction:
    Type: AWS::Serverless::Function
    Properties:
      Description:  A sample lambda function
      FunctionName: !Sub 'example-${Environment}'
      ...
CatalogsLogGroup:
    Type: AWS::Logs::LogGroup
    DependsOn:
      - ExampleFunction
    Properties:
      LogGroupName:
        !Join [
          '',
          ['/aws/lambda/', !Sub 'example-${Environment}']
        ]
      RetentionInDays: 3

Then we can deploy new stack and run subsequence deploy command without issue.

[billyshambrook]

Could the new Cloudformation resource import feature help here, I see that AWS::Logs::LogGroup is supported - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-supported-resources.html