Chalice: Lambda was unable to decrypt the environment variables because KMS access was denied

[danil-smirnov]

I've just run into the very same issue as guys in serverless framework community: 279.

After chalice deploy I've found two of my Lambdas (one connected to SNS and another to S3 events) dysfunctional with the error

The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.

The error appeared in AWS console only with complete silence in logs. Hence the failure isn't easy to discover as it just silently refuse to process SNS/S3 events.

Despite the issue could be easily fixed by application deletion and re-deployment, it might be quite painful in production environment.

[jamesls]

IIRC the deployment happens fine, but fails when you try to actually invoke the Lambda function right? I'm open to ideas on how to fix/detect this, but it sounds like the workaround is to delete/recreate the app. Not sure what we'd be able to do.

[john-osullivan]

I'm running into this issue and haven't been able to reliably solve it by recreating the app. Is this an actual bug with how Lambda loads the default Lambda KMS key?

If it helps with repro, I'm deploying via Terraform.

[mateusfccp]

I'm also deploying via Terraform and getting this error. Tried to delete the resource and recreate it, but nothing changed.

[reindeer77]

I had a similar issue, I was using the deploy with the --profile option since I have to manage multiple accounts at once.

I solved by promoting the account used with the --profile option to the default one (so getting rid of the --profile option itself).
Hope this helps.

[amarjandu]

Had this issue, was not getting invocation in CloudWatch Metrics, but saw the ErrorCount increase, the issue was found in cloudtrail where the role given to the lambda function did not have permission to decrypt, this was resolved by pointing the lambda IAM role to something random, then changing it back to the original role.

strange bug.

[czieglerAC]

Having the same issue where nothing was logged in cloudwatch, but errorcount would increase. We only found out the cause by running a test event and seeing the permission error.

This happened after we made a change to the iam role that our lambda was pointing to (we did not have chalice managing our iam roles). We changed it from LambdaBasicExecutionRole to AWSLambdaVPCAccessExecutionRole to deploy our lambda to a VPC. We use Terraform to manage our IAM roles

We did this two times with two different lambda functions. The first time there was no problem. The second time we encountered this bug.

Hope this information helps.

Edit:

On further inspection, this appears to be an AWS security feature. The solution is to not change the IAM roles your lambda is pointing to in-place. Instead generate a new role with a new name, and give that new role to the lambda.