Aws-sam-cli: sam local invoke (golang) unable to access secretsmanager with "The security token included in the request is invalid" error

[pts-jchin]

Description

sam local invoke lambda with secretsmanager GetSecretValue access respond with "The security token included in the request is invalid" error.

It seems that the request to secretsmanager has X-Amz-Security-Token header set to (null) when aws_session_token is not set in the environment variables.

This seems to only happen when docker image amazon/aws-sam-cli-emulation-image-go1.x:rapid-1.0.0 is used.

It is working when docker image amazon/aws-sam-cli-emulation-image-go1.x:debug-1.0.0 debugger mode

Steps to reproduce

SAM template based on the Helloworld template from sam init

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app

  Sample SAM Template for sam-app

Globals:
  Function:
    Timeout: 5

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: hello-world/
      Handler: hello-world
      Runtime: go1.x
# set AWS_SESSION_TOKEN to empty string as a workaround
#      Environment: 
#       Variables:
#          AWS_SESSION_TOKEN: "" 

Outputs:
  HelloWorldFunction:
    Description: "First Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

main.go

package main

import (
    "fmt"
    "os"

    "github.com/aws/aws-sdk-go/service/secretsmanager"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
)

func handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {

    awsConfig := &aws.Config{
        Region: aws.String(os.Getenv("AWS_REGION")),
    }
    // debug with HTTP body
    sess := session.Must(session.NewSession(awsConfig.WithLogLevel(aws.LogDebugWithHTTPBody)))

    svc := secretsmanager.New(sess)

    input := secretsmanager.GetSecretValueInput{
        SecretId:  aws.String("secretID"),
        VersionId: aws.String("00000000-0000-0000-0000-000000000000"),
    }

    resp, err := svc.GetSecretValue(&input)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to get secret: %s", err)
        return events.APIGatewayProxyResponse{}, err
    }

    return events.APIGatewayProxyResponse{
        Body:       fmt.Sprintf("Secret string: %s \n", *resp.SecretString),
        StatusCode: 200,
    }, nil
}

func main() {
    lambda.Start(handler)
}

Observed result

X-Amz-Security-Token header is set with (null) in the request, which results in HTTP 400 error with message UnrecognizedClientException: The security token included in the request is invalid

$ sam local invoke --debug
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
local invoke command is called
No Parameters detected in the template
1 resources found in the template
Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
Found one Lambda function with name 'HelloWorldFunction'
Invoking hello-world (go1.x)
No environment variables found for function 'HelloWorldFunction'
Environment variables overrides data is standard format
Loading AWS credentials from session with profile 'None'
Resolving code path. Cwd=###SCRUBBED###/sam-app/.aws-sam/build, CodeUri=HelloWorldFunction
Resolved absolute path to code is ###SCRUBBED###/sam-app/.aws-sam/build/HelloWorldFunction
Code ###SCRUBBED###/sam-app/.aws-sam/build/HelloWorldFunction is not a zip/jar file
Failed to download image with name amazon/aws-sam-cli-emulation-image-go1.x:rapid-1.0.0
Failed to download a new amazon/aws-sam-cli-emulation-image-go1.x:rapid-1.0.0 image. Invoking with the already downloaded image.
Mounting ###SCRUBBED###/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
Starting a timer for 5 seconds for function 'HelloWorldFunction'
START RequestId: cf90b33d-7301-1159-dc53-8eb0409b7c2a Version: $LATEST
2020/07/27 22:37:25 DEBUG: Request secretsmanager/GetSecretValue Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1

Host: secretsmanager.us-east-1.amazonaws.com

User-Agent: aws-sdk-go/1.33.12 (go1.14.4; linux; amd64)

Content-Length: 74

Authorization: AWS4-HMAC-SHA256 Credential=###SCRUBBED###/20200727/us-east-1/secretsmanager/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=###SCRUBBED###

Content-Type: application/x-amz-json-1.1

X-Amz-Date: 20200727T223725Z

X-Amz-Security-Token: (null)

X-Amz-Target: secretsmanager.GetSecretValue

Accept-Encoding: gzip



{"SecretId":"secretID","VersionId":"00000000-0000-0000-0000-000000000000"}
-----------------------------------------------------
2020/07/27 22:37:25 DEBUG: Response secretsmanager/GetSecretValue Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 400 Bad Request

Content-Length: 106

Content-Type: application/x-amz-json-1.1

Date: Mon, 27 Jul 2020 22:37:26 GMT

X-Amzn-Requestid: 6b885ccd-40c1-424b-a125-0a2f162afba5




-----------------------------------------------------
2020/07/27 22:37:25 {"__type":"UnrecognizedClientException","message":"The security token included in the request is invalid"}
Unable to get secret: UnrecognizedClientException: The security token included in the request is invalid
        status code: 400, request id: 6b885ccd-40c1-424b-a125-0a2f162afba5UnrecognizedClientException: The security token included in the request is invalid
        status code: 400, request id: 6b885ccd-40c1-424b-a125-0a2f162afba5: requestError
null
END RequestId: cf90b33d-7301-1159-dc53-8eb0409b7c2a
REPORT RequestId: cf90b33d-7301-1159-dc53-8eb0409b7c2a  Init Duration: 172.09 ms        Duration: 214.96 ms     Billed Duration: 300 ms Memory Size: 128 MB     Max Memory Used: 33 MB
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'duration': 1968, 'exitReason': 'success', 'exitCode': 0, 'requestId': '54946743-b91a-4ea5-a3e0-e8ef0189fd6d', 'installationId': 'dd7cb5e5-1fb6-440e-a461-37c9850e3756', 'sessionId': '0e62a418-c765-4e44-825f-0d0bac6ef8ef', 'executionEnvironment': 'CLI', 'pyversion': '3.7.8', 'samcliVersion': '1.0.0'}}]}
Telemetry response: 200

{"errorType":"requestError","errorMessage":"UnrecognizedClientException: The security token included in the request is invalid\n\tstatus code: 400, request id: 6b885ccd-40c1-424b-a125-0a2f162afba5"}

Expected result

Expected error ResourceNotFoundException: Secrets Manager can't find the specified with workaround by setting function environment variable AWS_SESSION_TOKEN with empty string in the template.

X-Amz-Security-Token header is expected to be not set.

$ sam local invoke --debug
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
local invoke command is called
No Parameters detected in the template
1 resources found in the template
Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
Found one Lambda function with name 'HelloWorldFunction'
Invoking hello-world (go1.x)
Environment variables overrides data is standard format
Loading AWS credentials from session with profile 'None'
Resolving code path. Cwd=###SCRUBBED###/sam-app/.aws-sam/build, CodeUri=HelloWorldFunction
Resolved absolute path to code is ###SCRUBBED###/sam-app/.aws-sam/build/HelloWorldFunction
Code ###SCRUBBED###/sam-app/.aws-sam/build/HelloWorldFunction is not a zip/jar file
Failed to download image with name amazon/aws-sam-cli-emulation-image-go1.x:rapid-1.0.0
Failed to download a new amazon/aws-sam-cli-emulation-image-go1.x:rapid-1.0.0 image. Invoking with the already downloaded image.
Mounting ###SCRUBBED###/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
Starting a timer for 5 seconds for function 'HelloWorldFunction'
START RequestId: fea79a5a-2df5-18ac-f320-bd1652094434 Version: $LATEST
2020/07/27 22:32:25 DEBUG: Request secretsmanager/GetSecretValue Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1

Host: secretsmanager.us-east-1.amazonaws.com

User-Agent: aws-sdk-go/1.33.12 (go1.14.4; linux; amd64)

Content-Length: 74

Authorization: AWS4-HMAC-SHA256 Credential=###SCRUBBED###/20200727/us-east-1/secretsmanager/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=###SCRUBBED###

Content-Type: application/x-amz-json-1.1

X-Amz-Date: 20200727T223225Z

X-Amz-Target: secretsmanager.GetSecretValue

Accept-Encoding: gzip



{"SecretId":"secretID","VersionId":"00000000-0000-0000-0000-000000000000"}
-----------------------------------------------------
2020/07/27 22:32:25 DEBUG: Response secretsmanager/GetSecretValue Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 400 Bad Request

Content-Length: 99

Content-Type: application/x-amz-json-1.1

Date: Mon, 27 Jul 2020 22:32:26 GMT

X-Amzn-Requestid: 000c661a-6e55-47e6-83ea-0b9d9e22132a




-----------------------------------------------------
2020/07/27 22:32:25 {"__type":"ResourceNotFoundException","Message":"Secrets Manager can't find the specified secret."}
Unable to get secret: ResourceNotFoundException: Secrets Manager can't find the specified secret.ResourceNotFoundException: Secrets Manager can't find the specified secret.: ResourceNotFoundException
null
END RequestId: fea79a5a-2df5-18ac-f320-bd1652094434
REPORT RequestId: fea79a5a-2df5-18ac-f320-bd1652094434  Init Duration: 546.62 ms        Duration: 471.29 ms     Billed Duration: 500 ms Memory Size: 128 MB     Max Memory Used: 33 MB
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'duration': 4684, 'exitReason': 'success', 'exitCode': 0, 'requestId': '29c79c9c-e82b-48f6-98df-0b0fb304068d', 'installationId': 'dd7cb5e5-1fb6-440e-a461-37c9850e3756', 'sessionId': '5c66f293-ee02-4f7b-b8ba-e272828b7857', 'executionEnvironment': 'CLI', 'pyversion': '3.7.8', 'samcliVersion': '1.0.0'}}]}
Telemetry response: 200

{"errorType":"ResourceNotFoundException","errorMessage":"ResourceNotFoundException: Secrets Manager can't find the specified secret."}

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Mac, Amazon Linux
2. sam --version: 1.0.0
3. aws-lambda-go v1.13.3
4. aws-sdk-go v1.33.12
5. go --version: 1.14.4

sam local invoke with debugger mode did not have the same issue and have the expected result

$ sam local invoke -d 5986 --debugger-path ./bin/ --debug-args "-delveAPI=2" --debug
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
local invoke command is called
No Parameters detected in the template
1 resources found in the template
Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
Found one Lambda function with name 'HelloWorldFunction'
Invoking hello-world (go1.x)
No environment variables found for function 'HelloWorldFunction'
Environment variables overrides data is standard format
Loading AWS credentials from session with profile 'None'
Resolving code path. Cwd=###SCRUBBED###/sam-app/.aws-sam/build, CodeUri=HelloWorldFunction
Resolved absolute path to code is ###SCRUBBED###/sam-app/.aws-sam/build/HelloWorldFunction
Code ###SCRUBBED###/sam-app/.aws-sam/build/HelloWorldFunction is not a zip/jar file
Failed to download image with name amazon/aws-sam-cli-emulation-image-go1.x:debug-1.0.0
Failed to download a new amazon/aws-sam-cli-emulation-image-go1.x:debug-1.0.0 image. Invoking with the already downloaded image.
Mounting ###SCRUBBED###/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
Setting up SIGTERM interrupt handler
API server listening at: [::]:5986
2020-07-27T22:45:29Z info layer=debugger launching process with args: [/var/task/hello-world]
2020-07-27T22:45:32Z debug layer=debugger continuing
START RequestId: 905abe41-e80c-1a4d-9137-a44f92932bbd Version: $LATEST
2020/07/27 22:45:32 DEBUG: Request secretsmanager/GetSecretValue Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: secretsmanager.us-east-1.amazonaws.com
User-Agent: aws-sdk-go/1.15.81 (go1.14.4; linux; amd64)
Content-Length: 74
Authorization: AWS4-HMAC-SHA256 Credential=###SCRUBBED###/20200727/us-east-1/secretsmanager/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=###SCRUBBED###
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20200727T224532Z
X-Amz-Target: secretsmanager.GetSecretValue
Accept-Encoding: gzip

{"SecretId":"secretID","VersionId":"00000000-0000-0000-0000-000000000000"}
-----------------------------------------------------
2020/07/27 22:45:33 DEBUG: Response secretsmanager/GetSecretValue Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 400 Bad Request
Content-Length: 99
Content-Type: application/x-amz-json-1.1
Date: Mon, 27 Jul 2020 22:45:33 GMT
X-Amzn-Requestid: c4f99e40-d332-45ca-b7cc-c1aa433704f5


-----------------------------------------------------
2020/07/27 22:45:33 {"__type":"ResourceNotFoundException","Message":"Secrets Manager can't find the specified secret."}
Unable to get secret: ResourceNotFoundException: Secrets Manager can't find the specified secret.
        status code: 400, request id: c4f99e40-d332-45ca-b7cc-c1aa433704f5END RequestId: 905abe41-e80c-1a4d-9137-a44f92932bbd
REPORT RequestId: 905abe41-e80c-1a4d-9137-a44f92932bbd  Init Duration: 7321.46 ms       Duration: 326.65 ms     Billed Duration: 400 ms Memory Size: 128 MB     Max Memory Used: 73 MB

{"errorType":"requestError","errorMessage":"ResourceNotFoundException: Secrets Manager can't find the specified secret.\n\tstatus code: 400, request id: c4f99e40-d332-45ca-b7cc-c1aa433704f5"}
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'duration': 5826, 'exitReason': 'success', 'exitCode': 0, 'requestId': '50df1a2a-a810-4198-939b-d9b66705ec32', 'installationId': 'dd7cb5e5-1fb6-440e-a461-37c9850e3756', 'sessionId': '3f6959ae-25ec-4530-9047-4946369851d2', 'executionEnvironment': 'CLI', 'pyversion': '3.7.8', 'samcliVersion': '1.0.0'}}]}
Telemetry response: 200

[agrounds]

I'm experiencing a similar problem in a Java-based environment. I'm using sam local invoke to run a lambda written in Kotlin. If I use SAM v1.0.0, the AWS_SESSION_TOKEN environment variable gets set to (null), causing an authentication error.

[crwgregory]

Does anyone have a good workaround besides manually commenting the env var in and out? I tend to forget to comment it out before running sam deploy and then 5 min later cloudformation tells me I am uploading a reserved keyword to the lambda function, then I have to go back and comment it out and start the process all over again.

I wanted to use something like the local env file that sam lets you supply, but it seems like the variable has to be defined in the template before sam will allow you to inject a different value.

It also doesn't look like you can get access to the underlying docker image to inject a value that way.

---

Also really well detailed report jchin!

[briggySmalls]

I've been having this issue also, but it was using AWS IoT, not the secretsmanager.

The workaround also worked for me: adding AWS_SESSION_TOKEN="" to the template.yaml.

This is a regression, right? I didn't have this issue until recently.

[cmsImagine]

@briggySmalls workaround worked for me running locally.

When deploying to AWS I got this error:

Lambda was unable to configure your environment
variables because the environment variables you
have provided contains reserved keys that are
currently not supported for modification.
Reserved keys used in this request:
AWS_SESSION_TOKEN (Service: AWSLambdaInternal;
Status Code: 400; Error Code:
InvalidParameterValueException; Request ID:
8bbcb8c3-d2b3-454b-9ae3-1fdc1abb44ee; Proxy:
null)

So I just comment it out when I'm done doing local testing.

[sriram-mv]

This should be fixed with latest images have also been updated. I would suggest pruning those images that end with :rapid-1.0.0 and re-trying, and update to the latest version of SAM CLI.

Please re-open this issue if you are still facing this. Closing.