Serverless: Environment variables not updating in new lambda function version

[aliatsis]

This is a Bug Report

Description

For bug reports:

• What went wrong?
  New environment variables are not deploying to new function version for AWS Lambda.
  I've verified that the new variables are properly set inside the CloudFormation stack template.
  I've also ran every variation of deploy (with -p, with -f, with --force, with --update-config, and with code changes). My base use case is to sls package first, then deploy with -p

• What did you expect should have happened?
  Changing or adding environment variables in serverless.yml should update environment variables in lambda console for the corresponding function version when deployed.

• What was the config you used?

service:
  name: foo-service
  awsKmsKeyArn: ${ssm:/infra/lambda-env-kms-key-id}

frameworkVersion: "^1.25.0"

functions:
  foo-service:
    name: ${self:custom.stageNames.${self:provider.stage}, self:custom.defaultName}
    handler: dist/lambda.handler

provider:
  name: aws
  stage: ${opt:stage, self:custom.defaultStage}
  region: us-east-1
  profile: serverless
  deploymentBucket:
    name: ${ssm:/infra/serverless-deployment-bucket}
    serverSideEncryption: aws:kms
    sseKMSKeyId: ${ssm:/infra/serverless-deployment-bucket-kms-key-id}

  # service function settings
  runtime: nodejs6.10
  memorySize: 1024
  timeout: 30
  environment:
    SERVERLESS_STAGE: ${self:provider.stage}
    SERVERLESS_ALIAS: ${self:provider.alias}
    NODE_ENV: ${ssm:/infra/NODE_ENV}
    MY_NEW_ENV_VAR: ${ssm:/infra/foobar}
  vpc:
    subnetIds:
      - ${ssm:/infra/lambda-subnet}
    securityGroupIds:
      - ${ssm:/infra/lambda-sg}

custom:
  defaultName: ${self:service}__${self:provider.stage}
  defaultStage: staging
  defaultRegion: us-east-1
  stageNames:
    staging: ${self:service}
    prod: ${self:service}

package:
  include:
    - dist/**
    - node_modules/**
    - '!**/*.d.ts'
    - '!**/*.test.js'
    - '!**/*.spec.js'
    - '!**/__tests__/**'
  exclude:
    - '**'

plugins:
  - serverless-pseudo-parameters
  - serverless-aws-alias

• What stacktrace or error message from your provider did you see?
  No error message. Package and Deploy were successful.

Similar or dependent issues:

• https://github.com/HyperBrain/serverless-aws-alias/issues/91

Additional Data

• Serverless Framework Version you're using:
  1.25.0
• Operating System:
  Mac High Sierra locally and Ubuntu in CircleCI
• Stack Trace:
  None
• Provider Error messages:
  None

[BDQ]

I've can confirm this issues with 1.25.0 - if the lambda's code doesn't change the function deploy is skipped so the new env var isn't setup. Changing the code, the env var will be added.

UPDATE: When I add the --force option the variable DID update for me, without function code changes.

[Sly777]

I tried with --force but It didn't work. @BDQ @horike37

serverless 1.26
macOS

[nullflux]

It appears that the impact of this issue is not limited only to environment variables, but other lambda configuration options like timeout and VPC.

Confirmed with 1.26.0, Mac OS High Sierra locally.

[hassankhan]

Had a quick look through the code, the place where this is likely occurring is here.

[zimny]

+1

[jake-lambert-volusion]

When updating timeout or memorySize, the updated values are reflected in the Cloudformation stack template and reflected on the $latest version in the AWS Lambda console, but if I navigate to the most recent version directly, they are not.

[avi-beetul]

+1

[PouriaSterling]

+1

[thomasbiddle]

+!

[hassankhan]

This issue seems to occur for me when running sls build and then sls deploy -p. If I directly use sls deploy then it seems to work properly.

[jonasof]

When I tried to deploy a single function with --function, with some environment variable as integer (example: VARIABLE: 123), I got:

* InvalidParameterType: Expected 
params.Environment.Variables['VARIABLE'] to be a string.

And the deploy failed. After changing the integer variable to string '123', the sls deploy has updated the environment variables correctly.

(serverless version: 1.26.1)

[simlu]

Similar issue. This is pretty severe for us. Anyone figured out what is going on here?

[MacMcIrish]

I've noticed that when updating environment variables in the serverless config:

environment:
  VAR: ${env:VAR}

WILL update the lambda function environment variables, but references like

environment:
  VAR: { "Ref": MyResource }

will NOT update the lambda function environment variables.

[ctala]

This happened to me today. Is there any workaround on how to solve it ?

[danepowell]

I encountered this too. In my case, it seems that if you modify a variable via the Lambda console and then re-run a serverless deploy, it will _not_ overwrite the value you set in the console. But if you then update any variable at all in the serverless config and do another deploy, it will overwrite all values in Lambda.

[saumilsdk]

+1

[sverraest]

Encountered the same issue today but even changing the code will not add / update the parameter I added in environment.

[markddavidoff]

+1

[cub33]

+1

[chrisyorkston]

Any update on this?

[mmuallem]

+1.

Happened to me today updating a Lambda. The function logic was updated, but not the environment variables.

MacOS Mojave 10.14.1. Serverless v. 1.33.2

[dschep]

Hmm. Are people only having issues with Ref based env vars as @MacMcIrish mentioned or with normal variahbles too? @MacMcIrish, I'm not surprised Ref causes an issue since the code compares the CloudFormation template to decide to redeploy or not. you can always use sls deploy -f to deploy regardless of the template not being changed. That being said, it absolutely shouldn't be necessary and we'll look into fixing this. Should be doable by having the cfn template normalization code (currently it serves to remove references to s3 buckets since those change every deploy/package) to also replace refs with a random value so the comparison finds a change in the deployment.

[brafdlog]

@dschep we are having the same issue with ssm variables

[mmuallem]

@dschep, our variables are loaded through a file, from a config YAML file

[dschep]

Hey folks, I've created https://github.com/dschep-bug-repos/sls-4638 to try to reproduce this bug, but to no avail. Do the steps in the readme cause the bug for any of you? If not, what differences does your setup have that you could make to cause my repo to reproduce the bug? Let me know here or make a PR to that repo with a change that causes the bug to occur.

[DCazier]

For what it's worth, we were also having this problem in our organization. In our serverless.yml file I decided to change the 'timeout' property of one our lambdas from 10s to 15s. Deployed that changed. All env variable changes after that began to deploy as normal.

Seems to have solved our issue. Not sure what caused this to begin with, but hopefully this helps in the debugging or to get people unblocked.

[usamayousuf13]

having similar issue. Environment variables are showing null in lambda console when i try to pass the cloudformation parameters to my lambda environment variables. I am attaching a chunk of my CF template just to elaborate.

"Parameters": {
"userIdParameter": {
"Description": "some userId here",
"Type": "String"
}
}
"createLambdaFunction": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"S3Bucket": "some-s3-bucket",
"S3Key": "deployment-package.zip"
},
"Environment": {
"Variables":{
"userId": {
"Ref": "userIdParameter"
}
}
},
"Handler": "index.handler",
"Role": "roleArn here",
"Runtime": "nodejs8.10",
"Timeout": 30
}
}

[hsz]

I met the same issue when deploying whole package with sls deploy and even --force didn't update the environment variables.
When deploying specific function with --force - it works, but that's not a perfect solution.

Anyway - as a workaround I've figured out, that setting another extra env variable called i.e. BUILD_TIME with a timestamp that changes every time - all variables are set properly.

[michael-misshore]

+1. What's weird in our particular setup is that we deploy the same stack across three separate AWS accounts. At first, only the first account would update, and the other two would not. We found that using the $(file) syntax in the ENV section was causing the second account to not update. Once we removed that var, the second account began to update as expected. We're still having issues with the 3rd account, and I'm struggling to figure out why.

As per the OP, the CF template has the changes, and a --force and --update-config does not update the ENV vars.

[veromarchand]

I am having a similar problem (also using the $file syntax, under the custom section, which is then used to generate the ENV variables), my env variables won't take in AWS. Works fine when running serverless locally.

Updating the code "works", as in, I can see the new code being executed, but environment variables are always empty.

Any suggestions?

[ahmed-abdulmoniem]

Still happening

[humbleengineer]

I thought we were suffering this bug, in particular with variables set using Ref:, but it turned out that CloudFormation had somehow lost it's grip on the stack we were referencing. CloudFormation was returning an Id from a previous incarnation of the same stack. That is at least consistent with what CloudFormation Resources tab shows - but they're both wrong! I'm raising a separate bug with AWS about this.

This was a non-production CF stack that had been deleted and recreated a few times as we got to grips with the deployment process, and sometimes it got stuck in the dreaded UPDATE_ROLLBACK_FAILED state and we deleted it rather than waiting for it to recover.

Posting in case it helps others.

[trmaphi]

I got the same problem when deploy a single function using sls deploy -s dev -r ap-southeast-1 -f api. The environment variables in serverless.yml under provider.environment doesn't update in aws.

This behavior is not expected in our development when we have numerous of lambda functions.

[cyberfox1]

Just happened to me to.

[medikoo]

This would most likely be fixed with solution proposed here: https://github.com/serverless/serverless/issues/6243#issuecomment-501668041 (just a note, to make both issues connected)

[boyvanduuren]

@medikoo I'm actually not too sure whether that's the case. Myself, and a few other people that reported in this thread before me actually don't even see the environment variable(s) update when forcing a deployment. I can force using --force, or even by doing a code change to trigger deployment, but the environment variables are not changed.
The cloudformation template did change though, so this might not even be a bug with serverless itself?

[medikoo]

@boyvanduuren yes, it's probably the case, that it requires full stack update (and not just a function update), and that's best if signaled by the framework.

[pmuens]

Hey everyone, I too looked into this today and tried to reproduce this.

(Un)fortunately I couldn't reproduce this problem on my end. Can someone post a step-by-step guide which outlines how this bug can be surfaced? Please make sure to use the most recent Serverless version as we've fixed quite some problems after the initial report in 2018.

[boyvanduuren]

@pmuens I've not been able to get a stack in this state purposefully. We've always encountered this on stacks that were running for a while and were updated at least a few times. Spent a good few hours on trying various things to get a stack into this state, but didn't succeed.

[pmuens]

Thanks for following-up @boyvanduuren :+1:

In that case I'll close this issue for now since it seems to be non-deterministic and hard to reproduce. @aliatsis feel free to re-open it if you see this bug again and can shed some light into the reproduction.

[duy9403]

I was commenting that I could reproduced the problem, but actually I misspelt enviroment, I deleted the original comment (by mistake). I apologize for the notification.

[divyenduz]

I can reproduce this in the following scenario. If I change the environment variable from the AWS Console UI, then sls deploy doesn't change it anymore. Was the experience same for any of you?

[rooleek]

Guys,
this is stil reproducable. Any workarounds?

[humbleengineer]

I can reproduce this in the following scenario. If I change the environment variable from the AWS Console UI, then sls deploy doesn't change it anymore. Was the experience same for any of you?

@divyenduz my understanding is that serverless works via CloudFormation, so if you work directly via the AWS Console neither CloudFormation nor serverless will find anything to do, although CloudFormation drift detection _might_ show up something. I don't think what you describe is related to this issue.

[divyenduz]

@markbreal Thanks for the answer, yes, I agree with that. I also noticed that --force still overrides the env vars, which is the expected behavior.

[xaun]

Yep still reproducible. Since editing an environment variable through the AWS console, serverless.yml updates to that variable are no longer being received.

[medikoo]

If you think there's a bug in existing version of a framework, please open an bug report respecting all its template remarks