Amplify-cli: Add env vars into lambda functions

[mrcoles]

Is your feature request related to a problem? Please describe.

I have a lambda function in my amplify project that needs access to environment variables, e.g., Stripe keys. I thought I could set them in the env vars in the AWS Amplify Console or somewhere else, but it turns out that those don’t get passed through to lambda functions.

Describe the solution you'd like

I would love to be able to expose environment variables that I set in the AWS Amplify console (or some other generalizable and secure way that avoids having them encrypted and not stored directly in my source code) to my Amplify project’s lambda functions.

Describe alternatives you've considered

For now it appears that I have to go into the AWS lambda console and set these environment variables by hand.

[troygoode]

@mrcoles you can set environment variables for the Lambda functions within the CloudFormation template:

https://github.com/aws-amplify/amplify-cli/issues/678#issuecomment-452451525

[mrcoles]

@troygoode thx for the link! IMO that’s not a good solution, because it would require putting secret keys into my git repo, which is a common anti-pattern for security. For now, I think I’m going to just manually enter the environment variables into the lambda functions via the AWS lambda console and keep that step in a launch checklist that I have to manually review. If I could expose them from the centralized AWS Amplify console, then that would be the best, as there’s a single source of truth for these environment variables and it’s separated from my codebase.

[troygoode]

@mrcoles Ah yes, I agree RE: secret keys. We're using AWS Secret Manager for those (and giving the Lambda access to the correct secret):

const AWS = require('aws-sdk')

module.exports = async () => {
  const secretsManager = new AWS.SecretsManager()
  const secret = await secretsManager.getSecretValue({ SecretId: 'YOUR_KEY' }).promise()
  if (!secret) {
    throw new Error('Secret not found')
  }
  return JSON.parse(secret.SecretString)
}

Add to the PolicyDocument in your CloudFormation template:

{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:GetSecretValue"
  ],
  "Resource": {
    "Fn::Sub": [
      "arn:aws:secretsmanager:${region}:${account}:secret:YOUR_KEY",
      {
        "region": {
          "Ref": "AWS::Region"
        },
        "account": {
          "Ref": "AWS::AccountId"
        }
      }
    ]
  }
}

[mrcoles]

@troygoode that’s helpful, thx! The snippet and policy are great! However, for the AWS Console/CLI I still have the feature request for one place to set these (that way misconfigurations are way less likely).

[aashwin-gaur]

Is there any progress on this front, I would like to be able to create environment variables in my lambda cloudformation template by passing them as parameters to the template. (ideally during an 'amplify push')

[kaustavghosh06]

Today we released an updated flow as a part of the functions category to pass resource identifiers like the cognito userpool ID (managed and genrated by the Amplify CLI) to a lambda function as environment variables and also populate the corresponding lambda execution role to access these resources. You can install the latest version of the CLI and go through the amplify update function flow to update your existing functions to access your Amplify generated resources in the project.

[mrcoles]

@kaustavghosh06 are there docs on this?

[janhesters]

@kaustavghosh06 How can you access the DynamoDB table generated using amplify add api? I it seems like you can only access tables that are generated by amplify add storage.

PS: I tried reverse engineering how Amplify adds environment variables. I created a Lambda function using the rest template and one using the crud template. I then looked at their respective {name}-cloudformation-template files to see how they get which env variable. I tried manually adding my AppSync's DynamoDB's ARN, but I couldn't figure out how Amplify adds env names. If you could provide a name to manually add your DynamoDB's ARN and table name we could use that as a workaround until amplify update function (or add) recognizes the DynamoDB created by amplify add api.

[byronsorrells]

I also started using AWS Secrets Manager. The ENV environment variable is already provided in each lambda, so I did the following to use environment specific secrets:

const secret = await secretsManager.getSecretValue({
  SecretId: `${process.env.ENV}/appName/scopeName`
}).promise();

That allows you to create secrets with the following patterns:

• dev/app/scope
• staging/app/scope
• prod/app/scope

[LucasAndrad]

@kaustavghosh06 why this issue was closed man? _"Today we released an updated flow as a part of the functions category to pass resource identifiers like the cognito userpool ID ..."_ where are the docs? I saw this same comment from you many times here in many issues, sorry dude but we're not clairvoyant, we just want to use amplify. Thanks.

[kaustavghosh06]

@LucasAndrad Sorry for not attaching a link to the docs when closing the issue. But here's the doc reference - https://docs.amplify.aws/cli/function#function-templates You can jump to - You can update the Lambda execution role policies for your function to access other resources generated and maintained by the CLI, using the CLI

[jonmifsud]

@kaustavghosh06 I think what @LucasAndrad meant is that your permission related solution is a workaround which requires custom AWS Resources, and does not allow one to add new ENV vars to a Lambda from the CLI.

I think the ticket was meant as a way to be done through the CLI, which would be nice to have.

[funkfinger]

Yes, the issue I am running into is getting non AWS related environment variables from Amplify to Lambda without having to access the console and keeping the configuration mostly in code. If there was a way either via the CLI or the Cloudformation stack (AS A REF - so secrets are not in source repo), things would be a lot more convenient.

[ventinus]

@mrcoles Ah yes, I agree RE: secret keys. We're using AWS Secret Manager for those (and giving the Lambda access to the correct secret):

const AWS = require('aws-sdk')

module.exports = async () => {
  const secretsManager = new AWS.SecretsManager()
  const secret = await secretsManager.getSecretValue({ SecretId: 'YOUR_KEY' }).promise()
  if (!secret) {
    throw new Error('Secret not found')
  }
  return JSON.parse(secret.SecretString)
}

Add to the PolicyDocument in your CloudFormation template:

{
  "Effect": "Allow",
  "Action": [
    "secretsmanager:GetSecretValue"
  ],
  "Resource": {
    "Fn::Sub": [
      "arn:aws:secretsmanager:${region}:${account}:secret:YOUR_KEY",
      {
        "region": {
          "Ref": "AWS::Region"
        },
        "account": {
          "Ref": "AWS::AccountId"
        }
      }
    ]
  }
}

I'm very close to getting this working but getting an error saying the lambda does not have access to call getSecretValue on the secret arn. I can see that the policy document is being inlined as I expect in the lambda execution role. The only thing I can think of is that the error says it's an assumed role trying to call getSecretValue rather than the lambda execution role. Am I adding the policy in the wrong place? I tried updating the resource policy of the Secret to include my root account but that didn't change anything.

[ryanhollander]

@ventinus It's not clear what you are or are not doing so I'm going to run down the gamut. First, make sure your lambda and secret share regions. Otherwise in your lambda you may need to switch regions before querying secrets manager. Now, when I set this up I setup the permission as * (I should probably adjust that!), so it may be that you need another permission like ListSecret or DescribeSecret, honestly I'm not sure. I'd try adding read permissions if the error is unclear. If that's not the issue, you might not have put the cloudformation in the right place. It should go in the cloudformation json in the root of the function you created in amplify. While you can setup your own role, I usually just piggy-back on the lambdaexecutionpolicy like so (notice the secrets manager entries and how they fit):
"lambdaexecutionpolicy": { "DependsOn": [ "LambdaExecutionRole" ], "Type": "AWS::IAM::Policy", "Properties": { "PolicyName": "lambda-execution-policy", "Roles": [ { "Ref": "LambdaExecutionRole" } ], "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Resource": { "Fn::Sub": [ "arn:aws:logs:${region}:${account}:log-group:/aws/lambda/${lambda}:log-stream:*", { "region": { "Ref": "AWS::Region" }, "account": { "Ref": "AWS::AccountId" }, "lambda": { "Ref": "LambdaFunction" } } ] } }, { "Effect": "Allow", "Action": [ "secretsmanager:*" ], "Resource": "*" }, ] } } },

I hope this helps.

[ventinus]

@ryanhollander thank you for your response, it worked using "*" for resource. I experimented a bit more with this and I am using the parameters.json file to pass the value of the secret id to the CFN template and I see it being logged correctly from within the lambda but when the function errors calling getSecretValue, the arn of the requested secret resource included some kind of hashing after the secret id, e.g. arn:aws:secretsmanager:us-east-1:XXXXXXXXX:secret:CloudFrontSigningKeys-DsCEwV whereas my logging of the secret id was only CloudFrontSigningKeys. I adjusted the resource in the template to add a * after my injected secret id and that worked.

I'm not sure where the hashing is coming from though I'd be surprised if it was coming from amplify. Again, thanks for your help!

[ryanhollander]

@ventinus you're welcome!

I'm not sure I understand this part: I am using the parameters.json file to pass the value of the secret id to the CFN template

I may not completely understand what you are doing, but if it helps, all I do is add the permissions and call secretsmanager inside my lambda with the secretid (not the ARN), what is returned is a JSON object that has several bits of data in it, like this (this is actual in production code I use, but I changed the actual secret id):

const secrets = await secretsManager.getSecretValue({SecretId: "MySecretId"}).promise((data, err) => {
            if (err) console.log(`Error getting secret: ${err}`)
        });
        if (secrets.hasOwnProperty("SecretString")) {
            let secretString = secrets.SecretString;
            let secretJson = JSON.parse(secretString);
            let clientId = encodeURIComponent(secretJson['ClientId']);
            let clientSecret = encodeURIComponent(secretJson['Secret']);
...

[ventinus]

Apologies for the lack of clarity, I'm defining the secret id in parameters.json so it lives in an easy to find place to update and I inject that parameter value as an environment variable to the lambda as well as in IAM policy.

[ryanhollander]

@ventinus oh I see now, good idea. Thanks for clarifying.

[mansdahlstrom1]

I've got stuck with this to and found a solution that i think will work for me at least. Leveraging dynamic variables in CloudFormation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html#dynamic-references-secretsmanager.

1. Add your secrets to AWS secrets manager, (Either via console or create a script that loads a local .env file and the update secrets manager)

2. Add the following cloudformation to let lambda use secret manager to resolve env vars

"Environment": {
  "Variables": {
    "ENV": {
      "Ref": "env"
    },
    "REGION": {
      "Ref": "AWS::Region"
    },
    "KEY1": "{{resolve:secretsmanager:YOUR_SECRET_ID:SecretString:key1}}",
    "KEY2": "{{resolve:secretsmanager:YOUR_SECRET_ID:SecretString:key2}}",
    "KEY3": "{{resolve:secretsmanager:YOUR_SECRET_ID:SecretString:key3}}"
  }
}

If you are like me and hate being dependant of using the AWS console to have a working app i've included a node script here for updating the secret from .env file you have locally.

require('dotenv').config();
const SecretManager = require('aws-sdk/clients/secretsmanager');

const { KEY1, KEY2, KEY3 } = process.env;

const sm = new SecretManager({ apiVersion: '2017-10-17', region: 'eu-west-1' });

const secretId = 'YOUR_SECRET_ID';
const secretString = JSON.stringify({
  KEY1,
  KEY2,
  KEY3,
});

const updateSecret = async () => {
  const params = {
    SecretId: secretId,
    SecretString: secretString,
  };

  try {
    const data = await sm.putSecretValue(params).promise();
    return data;
  } catch (err) {
    console.error(err);
    throw err;
  }
};

const createSecret = async () => {
  const params = {
    Name: secretId,
    SecretString: secretString,
  };

  try {
    const data = await sm.createSecret(params).promise();
    return data;
  } catch (err) {
    console.log('Failed to create secret');
    console.error(err);
    process.exit(-1);
  }
};

(async () => {
  console.log(KEY1, KEY2, KEY3);
  try {
    await sm
      .describeSecret({
        SecretId: secretId,
      })
      .promise();

    await updateSecret();
  } catch (err) {
    if (err.code === 'ResourceNotFoundException') {
      console.log(
        `Secret "${secretId}" does not exists in your current env, creating secret...`
      );
      await createSecret();
    }
  }
})();