Serverless: Support for Lambda@Edge

[horike37]

This is a (Feature Proposal)

Description

For feature proposals:

• What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.

https://aws.amazon.com/blogs/aws/lambdaedge-intelligent-processing-of-http-requests-at-the-edge/
Today, Lambda@Edge has been GA released. When CloudFormation will support for this, would be great that implement this as a new event source.

• If there is additional config how would it look
  So I think it would be something like this:

functions:
  index:
    handler: handler.hello
    events:
      - cloudFront: originRequest

[pmuens]

Great idea! Thanks for keeping an eye on the feature and opening this @horike37 👍

Hopefully CloudFormation will support it soon!

[jthomerson]

I posted something on the CloudFormation forum asking if they could give us a clue when they'd be supporting it: https://forums.aws.amazon.com/thread.jspa?threadID=262327

[pmuens]

I posted something on the CloudFormation forum asking if they could give us a clue when they'd be supporting it: https://forums.aws.amazon.com/thread.jspa?threadID=262327

Great! Thanks for reaching out to the AWS Forums @jthomerson 👍

[jthomerson]

FYI all: I wrote a plugin to support Lambda@Edge for now: https://github.com/silvermine/serverless-plugin-cloudfront-lambda-edge.

[horike37]

@jthomerson
AWSOME!!! great work!!!!

[pmuens]

Great stuff 🎉! Thanks for sharing @jthomerson 👍

[grancalavera]

I believe this should be already supported in CloudFormation, unless something else besides LambdaFunctionAssociation is required... Is that the case?

[katspaugh]

@grancalavera LambdaFunctionAssociation requires an ARN with a specific version of the lambda.
I guess the proposal is to somehow automate the version update for a given CloudFront distribution when deploying a new lambda version.

[grancalavera]

@katspaugh right... would that mean that the CloudFormation template specifying the CloudFront distribution would have to be changed on the fly whenever a new version of the function is deployed?

I don't really know Serverless internals, but I've seen generated CloudFormation templates...

[Birowsky]

How do you guys do this with CloudFormation? You can help me out here: https://stackoverflow.com/q/48137499/592641

[PatrickJS]

+1
Lambda@Edge is a game changer for Web Apps

[horike37]

Does anyone want to implement this functionality?
We are always welcome your contribution anytime 😄

[laardee]

@horike37 Do you have more specific thoughts about the implementation? Should the event source create a new or update an existing CloudFront distribution? Or both?

If event source is updating existing distribution, that should be done with AWS SDK getDistribution & updateDistribution as the LambdaFunctionAssociation is a part of AWS::CloudFront::Distribution and not a separate CloudFormation resource. A pretty much same way that the serverless-plugin-cloudfront-lambda-edge is implemented.

The event source should contain the distribution id and path pattern along with the event type.

functions:
  index:
    handler: handler.hello
    events:
      - cloudFront: 
          distribution: distribution-id
          eventType: event-type
          pathPattern: path-pattern # for other than default cache behaviours

But if event source creates a new distribution, all the other distribution parameters (origins, certificates, aliases etc.) should be defined somewhere. I would personally prefer resources block when creating a new CloudFront distribution.

I can start working on this if you still want this to be implemented to the framework.

[horike37]

@laardee
Thank you for jumping in :smile:

We would adopt a way that event source creates a new distribution. In general, the framework core completely relies on CloudFormation behind the scene, so we can't update an existing distribution.(we want that as well if CloudFormation supports for updating an existing one though)

But if event source creates a new distribution, all the other distribution parameters (origins, >certificates, aliases etc.) should be defined somewhere. I would personally prefer resources block >when creating a new CloudFront distribution.

Fully agreed! it would be good to specify an additional settings for cloudfront on resources section.

I can start working on this if you still want this to be implemented to the framework.

AWESOME :+1: :tada: People who is in the serverless community still hope that!!
Let @serverless/framework-maintainers know if you need any helps!

[laardee]

@serverless/framework-maintainers I have a question about the stack deployment region. As the Lambda functions triggered with CloudFront has to be deployed into us-east-1, should the stack to be restricted so that it cannot be deployed to other regions or should lambda@edge functions have a separate CloudFormation stack that is deployed to us-east-1?

[horike37]

@laardee

should the stack to be restricted so that it cannot be deployed to other regions

I think it would be good to this idea. Additionally, would be good to add the attention to documentation of this feature as Current Gottcha.

In general, the framework core launch one CF stack for each one deployment.
It may introduce various any side effects if we adopt a separating stack way.

[laardee]

@horike37 FYI, currently deleting the CloudFormation stack that has lambda@edge functions seem to be impossible. The master version of Lambda function cannot be removed at all, and that prevents stack removal -> https://forums.aws.amazon.com/thread.jspa?threadID=260242&tstart=0#jive-message-824818. Anyways, I'll continue with the implementation. I have high hopes that AWS fixes this issue any day now. 😁

[horike37]

Oh... well :sweat_smile:
So that means we should wait for fix the issue on AWS end to go ahead with this feature.

Anyways, I'll continue with the implementation. I have high hopes that AWS fixes this issue any day now.

Great thanks! Yes, I hope that too.

[debanjanbasu]

The delete functionality seems to be working now. Is there any chance of updates now?

[laardee]

@debanjanbasu Nice, I'll try to open the PR today.

[laardee]

I managed to delete the lambda after I manually remove the lambda association from the CloudFront distribution and wait for a while (maybe 15 minutes). CloudFormation still fails to remove the function when the stack is deleted.

Lambda was unable to delete arn:aws:lambda:us-east-1:XXXXXXXXX:function:serverless-hello-world-dev-helloWorld3:1 because it is a replicated function.

The stack remains to the "delete failed" state. After 15 minutes I'm was able to delete the rest of the stack. :unamused:

@debanjanbasu is there more info about the fix somewhere?

@horike37 because the manual removal seems to work, we could first remove the associations and lambdas with the aws-sdk and then remove the stack. As a temporary solution before CloudFormation gets fixed...

[horike37]

Thank you for testing @laardee :+1:
Imo, we should wait for the implementation until a lifecycle from sls deploy to sls remove will work fine via CloudFormation.

We consistently have been implementing without operating deployed resources with aws-sdk(i.e operating deployed resources are done completely via CloudFormation), otherwise, consistency of deployment would be broken.

[laardee]

@horike37 ok, I'll check if I get more info about the removal from AWS.

[mfrankwork]

AWS responded to an open support ticket today and said they have enabled the deletion of Lambda@Edge functions. I was able to successfully delete a master Lambda@Edge function that previously could not be removed.

[laardee]

I still get an error when I try to delete replicated function. Maybe they make it available gradually. I'll do more testing.

[yvele]

Simple question.. How do you manage the fact that CloudFront distribution and Lambda Edge association are tightly coupled.

Like if you manage the CloudFront distribution in a separate CloudFormation stack, then the Lambda Edge association might be removed.

🤔

See also the question on AWS Forum or on SO.

[laardee]

That could be an option, possibly a bit complex to handle, but if the CloudFront and invoke permissions were in a separate stack, removal process could stop to wait for deletion of replicated functions. I check if it is possible to check when the replicated functions are deleted.

I think the issue is that the CloudFormation doesn't wait long enough for replications to be deleted. When I delete the CF stack, first round always fails because it cannot delete the replicated functions, but if I wait ~15 minutes and then try to delete the stack again, it will delete the functions that were left from the previous round. Removing the stack takes least 30+ minutes total... Debugging this is really time-consuming :grin:

[jthomerson]

@laardee @mfrankwork @horike37 I described a bit more about why the delete doesn't work in a similar issue opened on our plugin that supports Lambda@Edge via Serverless: https://github.com/silvermine/serverless-plugin-cloudfront-lambda-edge/issues/6

[jthomerson]

All: please note that I'm updating our serverless-plugin-cloudfront-lambda-edge plugin to use the new CloudFormation support for Lambda@Edge. The principle is that the plugin will merely make the required modifications to the CloudFormation stack after Serverless has initially created the stack template, but before it has deployed. Those modifications include:

• remove environment variables from functions that have a Lambda@Edge event trigger
• modify the permissions granted to the function's role since it will need to be able to create log groups that AWS controls the name of (which typically include the region name)
• modify the CloudFront distribution resource (which you define in your serverless.yml file) to add the LambdaFunctionAssociations parameter to it.

The changes planned for v2.0 can be found in the issues listed below. Please let us know if you have other requirements as we hope to ship 2.0 in the next week or so.

• https://github.com/silvermine/serverless-plugin-cloudfront-lambda-edge/issues/7
• https://github.com/silvermine/serverless-plugin-cloudfront-lambda-edge/issues/5
• https://github.com/silvermine/serverless-plugin-cloudfront-lambda-edge/issues/2

[horike37]

@jthomerson
Awesome :tada: :+1: :smile: Thank you for the infomation!

[claytonpiccinin]

Awesome work @jthomerson!!! Is there some way we can automate Cloud Front Distribution to set up a new Lambda@Edge version?

Currently, I'm using a proxy with multiple behaviors, and I'd like to update a specific cache behavior with a new version of Lambda@Edge.

Additionally, can someone explain me why AWS just permit to add triggers for a numbered version, not for $LATEST or for aliases?

[jthomerson]

@claytopiccinin thanks. I'm not sure I understand your question. Serverless automatically creates new versions of your function when they change. Combine that with serverless-plugin-cloudfront-lambda-edge and you can deploy Lambda@Edge functions to your CloudFront distributions easily with Serverless. Check out the plugin's documentation and see if it answers your question.

[joseoscargarciajr]

Is there a way for the plugin to use the Arn of the cloudfront instead of the name, I have functions and api in the ap-northeast-1 and lambdaegde is only available in us-east-1, I want to create the function and just reference the Arn of the cloudfront via Fn::ImportValue using Arn, is this possible?

[ejaylegaspi]

@jthomerson I'm using your plugin and it works great! However, is there a way to rollback a lambda deployment using Serverless and your plugin? Like to revert back to the old version of Lambda@Edge if there's a problem? I'm trying to find ways how to do it. My ideal deployment would be to release it to Production but with a flag to go and use the latest version of Lambda and see if everything looks good. If there's a problem, i can easily rollback to the previous version without doing another whole new deployment just to rollback.

[jthomerson]

@ejaylegaspi unfortunately with Lambda@Edge, any change to your function (including rollback) is constrained by the amount of time it takes the CloudFront distribution to roll out a change ... which for me averages 30-45 minutes. So, there's really no quick method. They don't support pointing to a Lambda alias (presumably because of the function replication "magic") (see [1]), so you're a bit stuck there. There's not much we could do at a plugin level for you to help with this.

[1] https://forums.aws.amazon.com/thread.jspa?messageID=816124

[kmjennison]

In case it's helpful, I wanted to share the somewhat convoluted setup I used. Our constraints were: we want to use Serverless to manage the Lambda@Edge functions, our existing services are not in us-east-1, and and our CloudFront distribution is defined in a separate Serverless service.

Unfortunately, serverless-plugin-cloudfront-lambda-edge does not support cross-service references, so I chose to define the Lambda@Edge associations manually via LambdaFunctionAssociations in the CloudFront resource.

I created a separate Serverless config file for the Lambda@Edge functions, because they must be in us-east-1 (unlike the rest of our stack) and Serverless does not support multi-region deployments. Serverless also does not support customizing the yml filename, so we simply rename each yml file to "serverless.yml" before deploying each one.

When we update the Lambda@Edge functions, we manually increment the version number in the CloudFront resource definition.

I'd love any suggestions if someone found a simpler approach, especially if I overlooked functionality in Serverless or serverless-plugin-cloudfront-lambda-edge. A new plugin could probably address this issue.

Hopefully Serverless core can eventually handle this multi-service, multi-region use case.

[garethmcc]

One thing to bear in mind throughout all of this is that practically it doesn't matter in which "region" you define your CloudFront distribution. It only matter what price option you pick to support the edges you want. Cloudfront is essentially global otherwise which is why there is the restriction on us-east-1. What may have made it easier is to move the CloudFront definition from the other service to the same service as your lambda.

[kmjennison]

@garethmcc True. I didn't do that because we have other resources/variables defined in the CloudFront service, so it seemed like a lot of trouble—but, at the end of the day, maybe less trouble than the approach I outlined :)

[Maxwell2022]

@kmjennison

[...] I created a separate Serverless config file for the Lambda@Edge functions, because they must be in us-east-1 [...]

Is that because of your architecture or is it an AWS requirement?

EDIT
https://aws.amazon.com/about-aws/whats-new/2017/07/lambda-at-edge-now-generally-available/

"Lambda@Edge functions can now be authored in US East (N. Virginia), and will be replicated globally for invocation in response to CloudFront events."

I understand that if you want replication, then you need to deploy to us-east-1. It's not really clear here. If you don't care about replication, could you deploy it in other regions? I guess yes. However, if you enabled CloudFront with A-Z replication, then you probably need to deploy this Edge functions to us-east-1.

[kmjennison]

@Maxwell2022 To my knowledge, Lambda@Edge functions must be in us-east-1.

AWS docs tutorial for Lambda@Edge:

Make sure that you’re in the US-East-1 (N. Virginia) Region. You must be in this Region to create Lambda@Edge functions.

AWS docs on Lambda limitations:

You can add triggers only for functions in the US East (N. Virginia) Region.

[c-ameron]

Hi @horike37 , thanks for your work with this. I can see that you've done https://www.npmjs.com/package/serverless-lambda-edge-pre-existing-cloudfront which fixes it - however I can't see it on Github anywhere? Could you please upload it to Github if you don't mind? Thanks!

[horike37]

@c-ameron
Hi
Thank you for seeing my plugin! Already uploaded it here.
https://github.com/serverless-operations/serverless-lambda-edge-pre-existing-cloudfront

Please check it 👍

[c-ameron]

ah great - thank you! :)