Aws-cdk: Deploying new version of lambda function

[duarten]

:question: General Issue

The Question

I'm attempting to setup a CodeDeploy deployment group for a Lambda function. The CDK documentation for the Version class states:

If you want to deploy through CloudFormation and use aliases, you need to
add a new version (with a new name) to your Lambda every time you want
to deploy an update. An alias can then refer to the newly created Version.

This suggests that if I want to make a new CodeDeploy deployment, I should change the version name. For this end, I'm naming the versions using the sha1 of the latest Git commit that affects the Lambda's code or configuration. However, if I commit code that makes cosmetic changes to the configuration (i.e., to CDK code pertaining to the lambda), that will produce no differences in the CloudFormation template, and I will get the error A version for this Lambda function exists ( 1 ). Modify the function to create a new version.

This suggests that for a new version to be deployed I need to change the code or make semantic changes to the configuration. If this is the case, why require the version name to be unique between versions?

Code:

const lambdaName = basename(__dirname)

async function getRevision(dir: string): Promise<string> {
    const root = basename((await run("git rev-parse --show-toplevel")).line())
    const p = dir.substring(dir.lastIndexOf(root) + root.length + 1, dir.length)
    return (await run(`git rev-list --abbrev-commit -1 HEAD -- ${p}`)).line()
}

export async function myLambda(stack: Stack): Promise<Alias> {
    const f = new Function(stack, lambdaName, {
        runtime: Runtime.GO_1_X,
        handler: "main",
        code: Code.asset(join(__dirname, "lambda.zip")),
    })

    const alias = new Alias(stack, lambdaName + "-alias", {
        aliasName: "live",
        version: f.addVersion(await getRevision(__dirname)),
    })

    new LambdaDeploymentGroup(stack, lambdaName + "-deployment", {
        alias: alias,
        deploymentConfig: LambdaDeploymentConfig.LINEAR_10PERCENT_EVERY_10MINUTES,
    })

    return alias
}

Environment

• **CDK CLI Version: 1.18.0
• **Module Version: 1.18.0
• **OS: macOS Catalina
• **Language: Typescript

Other information

[skinny85]

Hello @duarten ,

the error you're getting:

"A version for this Lambda function exists ( 1 ). Modify the function to create a new version.".

I assume is coming up during deployment, in CloudFormation?

Thanks,
Adam

[duarten]

Hi @skinny85,

Yes, that's correct.

[skinny85]

Right. So the way we usually deal with that in the CDK is to have a new version for every synthesis - this way, the new version will always be created. Take a look at this example in our docs.

[duarten]

Correct me if I'm wrong, but in that case won't a synthesis always change from the previous one, even though the Lambda's code and configuration didn't? It also doesn't seem to solve my original problem, since I may deploy a stack which contains a lambda function whose code and configuration didn't change, but the CDK version did, thus leading to the "A version for this Lambda function exists ( 1 ). Modify the function to create a new version." error.

[skinny85]

Correct me if I'm wrong, but in that case won't a synthesis always change from the previous one, even though the Lambda's code and configuration didn't?

Yes. It's a tradeoff: either you remember to change it, or you accept the fact that you'll get a new version every deployment. I don't think it's too big of an issue to be honest.

It also doesn't seem to solve my original problem, since I may deploy a stack which contains a lambda function whose code and configuration didn't change, but the CDK version did, thus leading to the "A version for this Lambda function exists ( 1 ). Modify the function to create a new version." error.

No. It doesn't matter that the code/configuration did not change, there will be a new version, so you won't get that error.

[duarten]

No. It doesn't matter that the code/configuration did not change, there will be a new version, so you won't get that error.

Sorry, I don't follow. The issue is that I specified a new version, and I got that error. If I don't change the version, then the CloudFormation template is the same and CDK doesn't attempt a deployment. If I change the version, CDK will attempt a deployment, but will fail with that error. (I guess that's some other layer complaining that the Lambda's code and configuration have not changed.)

Example:

 15/24 | 12:23:34 AM | CREATE_IN_PROGRESS   | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version86f7530 (cognitopostconfirmationnewuserVersion86f753088FD94FA)
 15/24 | 12:23:35 AM | CREATE_IN_PROGRESS   | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version86f7530 (cognitopostconfirmationnewuserVersion86f753088FD94FA) Resource creation Initiated
 16/24 | 12:23:36 AM | CREATE_COMPLETE      | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version86f7530 (cognitopostconfirmationnewuserVersion86f753088FD94FA)

Note that version is 86f7530. Changing that to 7dad47a:

 0/4 | 12:29:33 AM | CREATE_IN_PROGRESS   | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version7dad47a (cognitopostconfirmationnewuserVersion7dad47aE2212F63)
 1/4 | 12:29:33 AM | CREATE_FAILED        | AWS::Lambda::Version             | cognito-post-confirmation/new-user/Version7dad47a (cognitopostconfirmationnewuserVersion7dad47aE2212F63) A version for this Lambda function exists ( 1 ). Modify the function to create a new version.
    new Version (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/npm/node_modules/@aws-cdk/aws-lambda/lib/lambda-version.js:28:25)
    \_ Function.addVersion (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/npm/node_modules/@aws-cdk/aws-lambda/lib/function.js:259:16)
    \_ UmaniLambda.<anonymous> (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/umani/constructs/umani-lambda.js:51:37)
    \_ Generator.next (<anonymous>)
    \_ fulfilled (/Users/duarten/code/umani/bazel-bin/deploy/app.sh.runfiles/umani/constructs/umani-lambda.js:4:58)
    \_ processTicksAndRejections (internal/process/task_queues.js:93:5)

[skinny85]

Hmm, you are correct. Apologies. I wonder whether Lambda added this validation recently...? I swear this used to work.

[duarten]

No worries :)

[skinny85]

So here's my research on the topic.

Currently, we recommend customers to do the following:

    const version = func.addVersion(new Date().toISOString()); // <==
    const alias = new lambda.Alias(this, 'LambdaAlias', {
      aliasName: 'Prod',
      version,
    });

    new codedeploy.LambdaDeploymentGroup(this, 'DeploymentGroup', {
      alias,
      deploymentConfig: codedeploy.LambdaDeploymentConfig.LINEAR_10PERCENT_EVERY_1MINUTE,
    });

However, it seems that this no longer works - if the Lambda itself is unchanged from the previous version, the new Version will fail creation.

This needs some pretty serious changes in our API:

• We should create a new API that substitutes Function.addVersion(), like Function.addHashedVersion() (name TBD of course). It should base the name of the version on the hash of the code property of the Function (either the asset, or the inline string) - if the code does not change, a new Version will not be created.
• We should probably deprecate the old addVersion() method, because it seems like it's very easy to shoot yourself in the foot using it.

[skinny85]

Actually, I figured out a workaround :) changing the description of the Function is enough to make the Version creation succeed, so this works:

``ts const func = new lambda.Function(this, 'lambdaName', { // whatever properties you need... description:Generated on: ${new Date().toISOString()}`,
});

    const version = func.addVersion(new Date().toISOString());

    const alias = new lambda.Alias(this, 'lambdaName-alias', {
        aliasName: 'live',
        version: version,
    });

    new codedeploy.LambdaDeploymentGroup(this, 'lambdaName-deployment', {
        alias: alias,
        deploymentConfig: codedeploy.LambdaDeploymentConfig.LINEAR_10PERCENT_EVERY_1MINUTE,
    });

[duarten]

Thanks for the workaround :)

[cbertozzi]

thanks for the working workaround ;) I'm facing the same issue and I tried many actions before. I think that a better way to solve this problem is to wait until the SAM CDK module is finally stable and released and then use the autoPublishAlias property that takes care for all the dynamic of recognize a new codebase and create (or not) a new version

[skinny85]

I'm glad it worked @cbertozzi :) But I think the addHashedVersion() method I talked about above is the way to go here (determine the name of the function's version based on the hash of its code property), not the SAM package.

[eladb]

+1 on addHashedVersion. I would just make name an optional argument for addVersion and default to the asset source hash. @nija-at definitely worth getting into our planning.

[hoegertn]

Additionally, I would love a feature to set the version to be retained on update. Sometimes I want the new version to be deployed but old versions to be retained for callers outside of my Stack. (e.g. Alexa Skills)

[kjpgit]

I'm just trying to get AWS Lambda provisioned concurrency and autoscaling working, and ran into this issue. I think this is ridiculous - I don't care about versions or aliases, but I have to use them to get provisioned concurrency, but versions and aliases are broken in the CDK. Great. I suggest you guys think of an easier way to "deploy a lambda with provisioned autoscaling" with CDK.

The workaround of "random description for lambda fn" seems to work for me, but man are things slow. A single function alias update takes 2 1/2 minutes with provisioned concurrency:

2/5 | 10:54:05 PM | UPDATE_IN_PROGRESS | AWS::Lambda::Alias | webhook-alias (webhookalias09ADCD64)
2/5 Currently in progress: webhookalias09ADCD64
3/5 | 10:56:37 PM | UPDATE_COMPLETE | AWS::Lambda::Alias | webhook-alias (webhookalias09ADCD64)

[aripalo]

There's also the case of Lambda@Edge, which I think relates to this issue:

CloudFront requires that a specific version of Lambda function is associated with the distribution.

Currently my options are:

1. Manually control the Lambda versions and only create a new version when I know the Lambda code has changes. This is somewhat annoying as one tends to forget to do that.

… or

1. Always create a new version of the Lambda function (even if there are no changes to the actual code). This in turn will always trigger a CloudFront update which takes 20 mimutes and this is even more annoying 😅

I'd like to see exactly that kind of code hash based solution @skinny85 suggested in https://github.com/aws/aws-cdk/issues/5334#issuecomment-562979282 as it should resolve this challenge with Lambda@Edge.

[stefanolczak]

+1 on addHashedVersion. I would just make name an optional argument for addVersion and default to the asset source hash. @nija-at definitely worth getting into our planning.

What about other information that belongs to specified lambda's version? The function version includes also information like lambda runtime and all of the function settings, including the environment variables. Changing the settings without modifying code is possible so I think using just asset source hash as a version name is not enough. Instead we should somehow use hash of whole "AWS::Lambda::Function" resource.

[aripalo]

Yeah it would be nice if one could do something in the manner of:

const fn = new lambda.VersionedFunction(this, "MyFunc", { /*...*/});

… which would calculate a combined hash of the source code and of the AWS::Lambda::Function.

Then it would also create a new version if the hash has changed.

Not sure if this is doable, but just thinking out loud.

[iph]

+1 on addHashedVersion. I would just make name an optional argument for addVersion and default to the asset source hash. @nija-at definitely worth getting into our planning.

What about other information that belongs to specified lambda's version? The function version includes also information like lambda runtime and all of the function settings, including the environment variables. Changing the settings without modifying code is possible so I think using just asset source hash as a version name is not enough. Instead we should somehow use hash of whole "AWS::Lambda::Function" resource.

[From Lambda, just throwing opinion here]

Ran a few experiments, and this is partially dangerous and can lead to weird edge cases, though I do agree with the opinion that we should do this.

As an example, let's say you have the following environment variables in your function:

      Environment:
        Variables:
          hi: there
          hi2: there

and you "update" to:

      Environment:
        Variables:
          hi2: there
          hi: there

Lambda does not see this as an update to your function. It's an idempotent "update", because it's a map which has no explicit ordering. If you were to try to publish-version, and you were at version 1, Lambda would idempotently "publish" version 1 again.

Cloudformation does not interact kindly when you go from 1 -> 1. It will fail updates with Modify the function to create a new version. (this isn't Lambda). My guess is they are deeply tied in with the function version arn. Why? ¯\_(ツ)_/¯. I'll pop an issue in their queue, since that's a bit strange. Unfortunately, that could take some time to fix, so let's focus within the constraints of the problem presented.

If we were to naively hash the text within the function, Cloudformation will fail this update if you move from version{OLDHASH} to version{NEWHASH} in cloudformation logical id. So we will need to take extra steps to figure out what is idempotent in Lambda's eyes and produce a consistent hash with Lambda's expectations (e.g. sort a map and then produce a consistent hash off that sorted map, sort all function resource names before hashing). Tags and VPC configs are commutative, so would probably need to be sorted too. I think layer order is not commutative, but worth testing.

Sorry that it's frustratingly hard to get this right. We (Lambda+Cfn) need to do a bit better with these interactions and I'll start opening up communications to see what we can do --but due to backwards compatibility constraints of the past 5 years since this was originally shipped, the ship of "changing this" may have sailed.

[iph]

Examples of what I'm talking about above here: https://github.com/iph/lambda-experiments/tree/master/versioning-updates-cfn

Specifically experiment 3 in the README.