Serverless-python-requirements: Avoiding recreating an AWS Layer

[jnicho02]

Got my first AWS Layer working: geopandas. Thank you!

I had difficulty with the upload timing out, so copied the 59Mb pythonRequirements.zip, re-ran with an empty requirements.txt, uploaded the original zip manually to S3, and created a new Layer version using it.

Dumb question, but now that I have a Layer when I deploy I would want to skip Layer creation. Do I simply comment out the custom pythonRequirements and add in the layer ref ARN for the function? ...or should Layer creation only kick in if the requirements.txt has changed?

[dschep]

Alas, if a CloudFormation stack (what serverless uses under the hood) containing a layer and is updated, it creates a new version of the layer, even if it isn't updated.

The way I recommend handling this is unfortunately not super clean, you'll want a separate stack(service/project) for your layer and then use a CloudFormation export & import to use it. See the bottom of the tips section in this blog post

[jnicho02]

Many thanks for the quick reply. Hard to keep clean when you're on the cutting edge. The pattern of using a separate stack isn't unusual. I've seen it needed for databases to avoid destroying the data on a deploy.

[vitale232]

@jnicho02 - Sorry to bring this back from the dead, but this is the only hit on geopandas when searching the issues.

Have you had success with geopandas in Lambda recently? Do you have a public repo that is working that could serve as an example?

I keep running up against the size limits, even when I try to employ all of the space saving options from serverless-python-requirements readme. Here's the serverless.yml I'm using for testing:

service: lambda-geojson

provider:
  name: aws
  runtime: python3.8

functions:
  numpy_test:
    handler: handler.main
    layers:
      - { Ref: PythonRequirementsLambdaLayer }

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    layer: true
    zip: true
    slim: true

[jnicho02]

haven't run this for 12 months.... requirements-prod.txt only has geopandas in. I did an 'sls deploy' to create the layer the first time, and ctl-c when it started to upload the zip file. Then manually uploaded to S3 to deploy to the layer from there.

service: geopandas-layer

provider:
  name: aws
  runtime: python3.6
  region: eu-west-1

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    fileName: requirements-prod.txt
    dockerizePip: non-linux
    layer: true
#    slim: true

functions:
  hello:
    handler: handler.hello
    layers:
      - {Ref: GeopandasLambdaLayer}

[vitale232]

Thanks for the quick reply and additional details! I can successfully import geopandas using your method. It's not as clean as a simple sls deploy, but at least you can play around with it this way. Stay healthy out there!

[jnicho02]

I believe that a solution for using S3 to is coming to serverless https://github.com/serverless/serverless/pull/6196 In fact, it might be there already seeing as that PR was on 5 Jun 2019