Aws-cdk: [lambda] deployment failure on updates to cross-stack layers

[chorniyn]

Here is the application structure:

• Stack A:
  
  
  
  • lambda layer
  
  
• Stack B: Depends on A:
  
  
  
  • Lambda which is using the layer
  
  

new lambda.Function(this, 'some-lambda'
   {
     ...
        layers: [stackA.lambdaLayer]
   }
)

where stackA.lambdaLayer is an instance of lambda.LayerVersion.

The initial deployment works well.
However, when the layer code is changed and the second deployment takes place, Stack A deployment fails with error message:

lambdas-layer (...) Requested update requires the creation of a new physical resource; hence
creating one.
Resource creation Initiated
dev-StackA Export dev-CoreStack:corelambdaslayerLayerVersionArn261FB3DB
cannot be updated as it is in use by dev-StackB

[lapair]

@nchorniy is this still broken out of interest?... I was about to try set it up, but thought I'd just ask instead :)

[alexdilley]

@lapair still broken; I have the same issue.

[eladb]

Okay, I see what's going on. I don't have a solution yet, let me first explain what causes this: the problem here is that the layer version ARN changes when you update the layer (makes sense so far), which causes a change in the exported value which is imported from Stack B. CloudFormation protects against changes in export values that are being used by other stacks in order to protect against breaking the depending stack.

[alexdilley]

@eladb I've worked around this by using a SSM parameter store to reference the latest
layerVersionArn and then accessing the layer instance via Layer.fromLayerVersionArn(). (Nod to @rhboyd for this idea.)

I wrote a custom construct to act as such a proxy:

const cdk = require('@aws-cdk/core');
const lambda = require('@aws-cdk/aws-lambda');
const ssm = require('@aws-cdk/aws-ssm');

// Proxy to the LATEST version of the Lambda Layer shared between stacks via a
// SSM paramter store.
module.exports = class SharedLayer extends cdk.Construct {
  static parameterName(construct) {
    return `/${cdk.Stack.of(construct).stage}/shared/layerVersionArn`;
  }

  static of(construct) {
    const layerVersionArn = ssm.StringParameter.valueForStringParameter(
      construct,
      this.parameterName(construct)
    );

    return lambda.LayerVersion.fromLayerVersionArn(
      construct,
      'SharedLayer',
      layerVersionArn
    );
  }

  constructor(scope, id, props) {
    super(scope, id, props);

    new ssm.StringParameter(this, 'VersionArn', {
      parameterName: SharedLayer.parameterName(this),
      stringValue: props.layerVersionArn,
    });
  }
};

[jogold]

Okay, I see what's going on. I don't have a solution yet,

@eladb how about allowing the users to specify how they want to resolve their cross-stacks references: either with Fn::Import (the normal case) or with a custom resource that will read the output value by calling the CloudFormation API. This would allow to have cross-stacks references for resources for which an update sometimes requires a replacement.

[jogold]

This would result in a downtime for the consuming stack but can be maybe mitigated with a DeletionPolicy in the producing stack together with some sort of clean up at the end of the deploy command...

[ranguard]

https://serverless.pub/sar-layers/ Maybe of interest

[fabio-vitali]

@alexdilley could you post a sample code showing how your construct solves the scenario described in this issue?

[rhboyd]

@fabio-vitali That solutions works by breaking the coupling between one stack's export value and another stack's import value (which is enforced by cloudformation). Instead they are using one stack to put a value into SSM Parameter store then another stack is reading values from the same SSM Parameter store.

[lapair]

@fabio-vitali

The code @alexdilley wrote breaks down into:

Creating layer

        const paramName = '/layers/BaseLayer';

        // Create Layer
        const baseLayer = new lambda.LayerVersion(this, 'BaseLayer', {
            code: lambda.Code.asset(__dirname + '/layer/base'),
            compatibleRuntimes: [
                lambda.Runtime.NODEJS_8_10
            ],
            description: 'Our base layer',
        });

        // Save to SSM
        new ssm.StringParameter(this, 'VersionArn', {
            parameterName: paramName,
            stringValue: baseLayer.layerVersionArn,
        });

Using layer:

        const paramName = '/layers/BaseLayer';


         // fetch the Arn from param store
        const baseLayerArn = ssm.StringParameter.valueForStringParameter(
            this,
            paramName
        );
       // generate layerversion from arn 
        const layer1 = lambda.LayerVersion.fromLayerVersionArn(
            this,
            'BaseLayerFromArn',
            baseLayerArn
        );

       // Then supply when you create a lambda
       new lambda.Function(this, 'SomeName, {
              ...,
             layers: [layer1],
      });

[lapair]

@fabio-vitali but I think the @alexdilley version might create the relationship that my example above does not? Either way.. updating the layer deletes the previous version (with CDK).... but...

_"To avoid breaking functions, a copy of the version remains in Lambda until no functions refer to it."_
https://docs.aws.amazon.com/cli/latest/reference/lambda/delete-layer-version.html

[chrisgel15]

Not using CDK but I have been affected by this CF issue as well. My stack is created via CodePipeline + CloudFormation templates and I have a LayerVersion resource declared on it.

When I used the Output section to export my layer, the first time worked correctly but the second one showed the same error message.

I ended up declaring my LayerVersion resource in both templates to avoid using the cross reference value, as everytime the pipeline runs, it creates a new LayerVersion (no matter if I set the RetentionPolicy to Delete), and that version modification is preventing the update of the 2nd stack.

[pmaedel]

see #4014 for a feature request that would solve this issue

[nija-at]

To anyone who has or had this issue -

Do you have any stacks running production systems that is affected by this issue?

If not, how have you worked around this? Did you work around this by keeping the lambda function and its layer in the same stack?

(Asking to get a sense of what state people are in today)

[shaftoe]

@nija-at yeah, personally I've worked around that not sharing layers between stacks, in the end it's not that bad with the little helper function utility and folder structure, so I define the layer once for the project and I create a new one per stack whenever needed. Easier showed than said:

• the codebase
• the utility function
• example of usage in a stack

[ranguard]

I split the layers into a stack of their own, and share the ARN via SSM.Params, to then be used by a lambda in another stack. So no hard dependency...

You can see both the layer creation and then usage (which would usually be in a separate stack)

https://github.com/ranguard/cdk-talk-examples/tree/master/code/lambda_layer_example/lib

We have a 'layers' stack

AWS keeps layer versions around, whilst any lambda is pointing at them - so new layer deploys won't break existing functions - but you do need to re-deploy any stack that is using the layers

[nija-at]

Thanks for sharing @shaftoe and @ranguard