Amplify-cli: amplify mock: lambda runtime environment variables

[cyrfer]

Describe the bug
When I run amplify mock my lambda logs the runtime environment variables, and I do not see printed what I expect.

Amplify CLI Version
4.21.0

To Reproduce

1. amplify init
2. amplify api add, make a Query field use a lambda
3. amplify function add, make lambda console.log('AWS_ACCESS_KEY_ID ', process.env.AWS_ACCESS_KEY_ID)
4. amplify mock
5. make a graphql request

Expected behavior
I expect AWS_ACCESS_KEY_ID, and other lambda runtime env vars, to exist in the printed log.

Desktop (please complete the following information):

• OS: Mac High Sierra
• Node Version: v12.16.2

Additional context
This issue prevents my code to make signed calls to an API GW via execute-api:Invoke.

[akshbhu]

Hi @cyrfer

This issue is already as enhancement and you can follow up here for updates on this.
Closing this issue as duplicate of #4234

[cyrfer]

@akshbhu I don't see any feedback from AWS about that ticket. Is the problem acknowledged?

However, that issue sounds more like a problem with _user-provided_ environment variables rather than _system-provided_ env vars.

[akshbhu]

@cyrfer

@akshbhu I don't see any feedback from AWS about that ticket. Is the problem acknowledged?

Yes, This ticket is marked as enhancement and we are working towards it to include this functionality.

[cyrfer]

I think this should be classified as a bug. If the lambda environment is not the same, it is very difficult to do local development.

[renebrandel]

@cyrfer - you are correct that the other isn't related. Let me reopen this issue.

[renebrandel]

in terms of the reserved environment variables, I see you're specifically looking into AWS_ACCESS_KEY_ID as an example. I'm curious to learn about your scenario. Some context: with a better understanding of your scenario, we might be able to address other issues holistically. In general, we are rethinking how env vars should work in Amplify. Your input would be helpful!

[cyrfer]

@renebrandel Developers depend on the Lambda Runtime for a variety of reasons. I don't see any of the system-provided environment variables when I execute amplify mock with a function locally.

In one scenario, I need the following env vars defined so I can sign requests to an API GW (or managed ElasticSearch cluster). They are currently absent from process.env in my Nodejs12.x function.

• AWS_REGION
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_SESSION_TOKEN

Since the Amplify CLI knows the AWS profile configured for my current environment, it should make environment variables available with the values listed above for the configured profile of the active amplify environment.

When I use AWS SAM, these variables use a local profile of my choosing, depending on the value of my ~./aws/credentials file, and I think I can also specify the AWS_PROFILE env var (not 100% sure).

Not to muddy the waters too much, but I think user-specified env vars should be supportable through the same mechanism as deployment. Right now terrible values are leaking into my function's environment locally.

For example, if my cloudformation template specifies the environment variable like this (pseudo code):

Parameters:
  MongoSecretName:
    Type: String

Resources:
  MyFunction:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          MONGO_SECRET: !Ref MongoSecretName

then the following lambda code will print an undesired result

exports.handler = async (event, ctx) => {
  console.log('mongo secret name', process.env.MONGO_SECRET) // prints "MongoSecretName" ?!?
}

I'm using the team-provider-info.json file to supply the value for the template parameter for 3 different environments using 2 different aws profiles locally. The expected value does not appear in the logs. I think that tells me the code used to setup environment variables did not honor template parameters.

There is also a mysterious function-parameters.json file (not parameters.json) created when using the CLI to generate a function. I don't know what to do with it. I think it should be removed to avoid confusion.

[cyrfer]

@renebrandel I get the impression that docker is NOT being used for mocking functions locally - am I correct? If not using Docker, I think that is a lost opportunity to have the local runtime closely match cloud runtime. If Docker is used, I think emulating the local aws profile might simplified. Something like this is what I guess SAM does:

docker build -t MOCK_FUNCTION_IMAGE .
ALL_ENVVARS_FROM_DEPLOY=$(get from amplify push/cf template context) # -e AWS_ACCESS_KEY_ID=etc
docker run -v ~/.aws:/root/.aws $ALL_ENVVARS_FROM_DEPLOY MOCK_FUNCTION_IMAGE

The bit about -v ~/.aws:/root/.aws is so the AWS SDK will pickup the profile info correctly, but it might mean the env var AWS_PROFILE might need to be assigned when running a container, which is not ideal because it is not part of the cloud runtime. Ive seen some parts of the aws-sdk-nodejs misuse non-default profiles, so the sdk might have bugs preventing this ideal way of developing, beyond what amplify can control.

Also, if Docker is used, is there a build step for functions I can hook into? I'm interested in using Webpack so my Nodejs development feels more like my React development. It would be nice if a NPM script like "build" could be customized in functions.

Also, using the docker cli is not really expected. I used it for demo purposes. The docker runtime would be a more helpful interface for amplify's development. To coordinate multiple containers or emulate a gateway proxy to a lambda, docker compose seems useful.

[cyrfer]

@renebrandel I still cannot use amplify mock with a lambda (bound to my graphql querries/mutations) and do simple things like call AWS SecretsManager GetSecretValue for lack of a reasonable runtime environment.

I printed the environment for my lambda function when amplify mock is running and I'm guessing the AWS SDK doesn't like it when things like this show up:

"REGION": "us-east-1-fake",

The generated CF template assigns this to the lambda:

"Environment": {
  "Variables": {
    "ENV": {
      "Ref": "env"
    },
    "REGION": {
      "Ref": "AWS::Region"
    },

If I hard-code the desired region in the template, my lambda can print a reasonable value.

"REGION": "us-west-2",

This kind of hard coding allows me to run locally, but goes against any kind of usability for infrastructure-as-code.

I'm certain you guys know this is happening. Why don't you load the ~REGION~ AWS::Region from the profile for the configured Amplify Environment?

I don't think Amplify can claim being able to run the API locally when these kinds of problems have persisted for so long.

[cyrfer]

What is going on with this project?
https://github.com/aws-amplify/amplify-cli/search?q=fake

[renebrandel]

Hi @cyrfer - we're not using Docker under the hood for mocks. We did evaluate it during our initial design. Giving that we have a significant mobile dev user base though, Docker has had issues in the past clashing with Android emulators. We're open to revisit this decision if this changes in the future.

On your other point though, we agree and think it makes sense to introduce these variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN

@jhockett will look into this over the next couple of weeks.

[cyrfer]

Hi @renebrandel Great! If amplify mock can also use the template parameters specified in team-provider-info.json for the checked out environment, then I could actually develop locally before deploying.

[deanbarrow]

Hi @cyrfer this might help you out for mocking.

I also use Secrets Manager to populate process.env but it wouldn't be difficult to read the contents of team-provider-info.json to set the environment variables.

[cyrfer]

@deanbarrow Hi! I prefer to keep only sensitive values in SecretsManager. My team's security guidelines prevent me from exposing those values as env variables when deployed, so I don't read sensitive values from env vars locally either.

[deanbarrow]

@cyrfer I think that first link should still help to get things working locally without having to deploy each time you want to make a change though.
In production you'll be able to run GetSecretValue as you've already authenticated by the attached role but to replicate this locally you would have to set REGION, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY by your .env file. You should then be able to run GetSecretValue locally without having to hardcode anything in CF templates etc.
Hope that helps.