Aws-cdk: bucket event notifications creates placeholder lambda function

[onedotover]

Describe the bug
When you use function.addEventSource(new S3EventSource(...)) or bucket.addObjectCreatedNotification(...) it creates a BucketNotificationsHandler lambda function with some placeholder code.

To Reproduce

    const bucket = new Bucket(this, 'TestBucket');
    const lambda = new Function(this, 'TestFunction', {
      code: Code.asset('./test-src'),
      runtime: Runtime.NodeJS10x,
      handler: 'index.handler',
      description: 'Example function',
    });
    lambda.addEventSource(new S3EventSource(bucket, {
      events: [EventType.ObjectCreated],
    }));

Expected behavior
Should only connect the TestFunction to the lambda function not create a new placeholder function.

Version:

• macOS 10.14.5
• Typescript
• 0.33

[mirskiy]

Running in to the same issue. Tried downgrading to 0.29, didn't help.

[made2591]

I was able to reproduce the error. If it's ok for everyone, I can work on this (I'm investigated already a bit)

[made2591]

Ok I think I found the reason of this problem, and it's described here. Actually, what CDK (version 0.35) implements, is discussed also here.

If you create the target resource and related permissions in the same template, you might have a circular dependency. For example, you might use the AWS::Lambda::Permission resource to grant the bucket permission to invoke an AWS Lambda function. However, AWS CloudFormation can't create the bucket until the bucket has permission to invoke the function (AWS CloudFormation checks whether the bucket can invoke the function). If you're using Refs to pass the bucket name, this leads to a circular dependency.

This is because AWS::Lambda::Permission needs to know the ARN of the AWS::S3::Bucket before it can create AWS::Lambda::Function. However, the AWS::S3::Bucket's NotificationConfiguration property needs to know the ARN of AWS::Lambda::Function before it can create the AWS::S3::Bucket, thus creating a circular dependency. The solution discussed and implemented avoid the circular dependency demonstrated earlier by creating the S3 bucket without any notification configuration. Then it sets up the notification after the Lambda function, Lambda permission, and S3 bucket resources have been created.

In the code, you can follow the whole process starting here: the createResourceOnce() call the NotificationsResourceHandler.singleton(this);. Then, a stack-singleton lambda function with the logic for a CloudFormation custom resource provisions bucket notification configuration for a bucket with a well-known logicalID (source). The line 35 verifies that a Lambda with that ID doesn't exist yet - it will never exist. Then, the bucket notifications resources in the stack are defined only once and lazily, so that if notifications are not added, there is no notifications resource.

To conclude... I don't think there's a way to do avoid this (@eladb @rix0rrr am I wrong?), excluding having two separate Stacks that follow the same reasoning - thus, by defining the two resource (exposing the Lambda from one Stack to another). Not sure if it could work, I will try and let you know asap.

Hope to be helpful!

[made2591]

@NGL321 what do you think about this?

[NGL321]

@made2591 I will take a little more time to review your research either this evening or Monday morning (thank you for being so thorough!).
In the meantime, if you think you can fix it yourself, please feel free to work on the issue!

[jthornbrue]

My stack contains an object created notification that triggers an SNS topic. See below (using the Python API):

my_bucket.add_event_notification(
        aws_s3.EventType.OBJECT_CREATED,
        aws_s3_notifications.SnsDestination(my_topic),
        aws_s3.NotificationKeyFilter(
            prefix=OUTPUT_PREFIX,
            suffix='.csv'
        )
    )

This creates a dummy lambda function in the stack--even though there is not and never will be a lambda target. This is annoying. Any progress on this bug?

[alukach]

I think @made2591 is correct. Any time that you use my_bucket.add_event_notification, a NotificationsResourceHandler will be created: https://github.com/aws/aws-cdk/blob/cd0e9bd53e37f4b5824d64a0234e28c3a86b1ed9/packages/%40aws-cdk/aws-s3/lib/notifications-resource/notifications-resource.ts#L51-L52
https://github.com/aws/aws-cdk/blob/cd0e9bd53e37f4b5824d64a0234e28c3a86b1ed9/packages/%40aws-cdk/aws-s3/lib/notifications-resource/notifications-resource.ts#L101-L116

As @jthornbrue mentions, it does not matter if you intend to wire up an event to a lambda target.

I'd love to have a way to have an S3 bucket insert events into SNS/SQS without all of these extra (and likely unnecessary) resources.

[LechuckThePirate]

This should be solved. I don't understand how this went out of beta with this problem. If you run into this, it's not just that you have a lambda function there that shouldn't be there... the thing is you are actually PAYING for every S3 bucket event...

[n-is-one]

For me this problem seems even worse. Along with the placeholder lambda, also a placeholder role and policy are created, as cdk diff shows:
...
[+] AWS::IAM::Role BucketNotificationsHandler050a0587b7544547bf325f094a3db834/Role BucketNotificationsHandler050a0587b7544547bf325f094a3db834RoleB6FB88EC
[+] AWS::IAM::Policy BucketNotificationsHandler050a0587b7544547bf325f094a3db834/Role/DefaultPolicy BucketNotificationsHandler050a0587b7544547bf325f094a3db834RoleDefaultPolicy2CF63D36
[+] AWS::Lambda::Function BucketNotificationsHandler050a0587b7544547bf325f094a3db834 BucketNotificationsHandler050a0587b7544547bf325f094a3db8347ECC3691

My company enforces that all roles have a permission boundary, and the placeholder role does not have one. So my entire stack fails. I see no way to get around this (but I am new to CDK)

[Roman-L]

Are there any workarounds for this?

@made2591 Did you test the multiple stacks solution?

[dz902]

Circular reference is annoying, but could be avoided with specifying ARNS instead of using refs. At least we should have an option to avoid the extra Lambda and document the behavior.

Although I see that extra Lambda was not invoked by anything anywhere IF you bind the event to another Lambda.

[nija-at]

the thing is you are actually PAYING for every S3 bucket event...

@LechuckThePirate - how do you mean? As far as I can tell, we're adding the additional lambda as a CustomResource on the CloudFormation stack. This lambda is invoked once per deployment and not once per S3 event.

My company enforces that all roles have a permission boundary, and the placeholder role does not have one. So my entire stack fails. I see no way to get around this (but I am new to CDK)

@erikaadvisser - Here is the additional policy that is applied to the bucket notification lambda function. Is "Resource": "*" the bit that restricted by your company?

[n-is-one]

@nija-at , I think that it is the creation of the IAM:Role part that failed, not the creation of the policy. The role needs a permission boundary. However, I no longer work for this company, so I cannot check it for sure.

[nija-at]

Opened this issue to track the permission boundary - https://github.com/aws/aws-cdk/issues/5925

Unfortunately, the placeholder lambda is unavoidable. See made2591's comment for details.

I don't see any other specific issue reported here. Feel free to reopen if I missed something something or a separate issue if it hasn't been reported here.

[dballance]

@nija-at - I'm new to the CDK - but ran into this in my first experimentation. Some googling led me to this issue.

How is the CDK deployment different from the SAM CFN deployments? I only ask because via a SAM template, I can deploy both a bucket resource and establish my lambda event without triggering a circular dependency.

Resources:
  Lambda:
    Type: AWS::Serverless::Function
    Properties: 
      Events: 
        SomeS3Event:
          Type: S3
          Properties:
            Bucket: !Ref SourceBucket
            Events: s3.ObjectCreated:*
  SourceBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: test-bucket

Edit: Is it solely for policy attachment? In my SAM templates I'm using computed ARNs and Parameters (so not directly !Ref'ing the bucket) to apply lambda policies:

Policies:
        - AWSLambdaExecute
        - S3CrudPolicy:
            BucketName: !Ref SourceBucketName
        - Version: '2012-10-17' 
          Statement:
            - Effect: Allow
              Action:
                - s3:GetObjectTagging
              Resource: 
                - !Sub 'arn:aws:s3:::${SourceBucketName}/*'