Aws-sam-cli: Invalid API Gateway Response Keys: {'base64Encoded'} in {'statusCode': 200, 'headers': {'Content-Type': 'text/plain'}, 'body': 'Hello World', 'base64Encoded': False}
Description
We are using aws-serverless-java-container-core to set up our API Gateway proxy lambdas. Each lambda returns an AwsProxyResponse. When run in AWS these lambdas work fine through the configured API Gateways; however, when testing locally with the aws-sam-cli (sam local start-api) we are only able to receive 502 Bad Gateway responses from the local gateway.
It appears that release 0.16.0 of aws-sam-cli added validation to the parameters returned from lambdas (https://github.com/awslabs/aws-sam-cli/pull/1154). When we check the logs for our local lambdas we find that the property for base64encoding is not respecting the declared JsonProperty annotation and is being serialized with a value different that the one that is expected. The serialized value is consistent between the local and AWS environments.
This is a similar issue to the one described in issue 830, but it is unrelated. We are not using the multiValueHeaders field in any of our lambda responses, and the attached error log does not contain this field.
The issue we are facing is that the "isBase64Encoded" field of the AwsProxyResponse object is being serialized with a name of "base64Encoded" rather than the expected name of "isBase64Encoded". This is resulting in all requests to the local gateway returning 502 regardless of the actual status of the lambda. We are using the AwsProxyResponse object provided by aws-serverless-java-container-core defined here: https://github.com/awslabs/aws-serverless-java-container/blob/master/aws-serverless-java-container-core/src/main/java/com/amazonaws/serverless/proxy/model/AwsProxyResponse.java
Downgrading the aws-sam-cli to version 0.15.0 does prevent the issue from occurring as there is no validation logic present in that version
Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
aws-sam-cli version: 0.16.1
aws-serverless-java-container-core version: 1.3
Add --debug flag to command you are running
Error Log:
2019-05-22 14:56:37 Invalid API Gateway Response Keys: {'base64Encoded'} in {'statusCode': 200, 'headers': {'Content-Type': 'text/plain'}, 'body': 'Hello World', 'base64Encoded': False}
2019-05-22 14:56:37 Function returned an invalid response (must include one of: body, headers or statusCode in the response object). Response received: {"statusCode":200,"headers":{"Content-Type":"text/plain"},"body":"Hello World","base64Encoded":false}
2019-05-22 14:56:37 127.0.0.1 - - [22/May/2019 14:56:37] "GET /echoStatusCode/200 HTTP/1.1" 502 -
psaws
All 12 comments
@psaws This is a duplicate of #830. See my comment here: https://github.com/awslabs/aws-sam-cli/issues/1190#issuecomment-494469325
Closing as duplicate.
jfuss
on 23 May 2019
I miss read the issue, my fault.
jfuss
on 23 May 2019
@psaws SAM CLI does not serialize the data. The expected key is "isBase64Encoded" from https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-output-format. You should cut an issue against the aws-serverless-java-container-core project.
jfuss
on 23 May 2019
It seems like this issue was fixed on aws-serverless-java-container-core side in 1.3.2, but I am still getting the same error.
Please, take a look at this
related issue
vladamx
on 6 Jul 2019
Would be great to see this fixed for those working with custom event types.
testphreak
on 25 Jul 2019
I am still experiencing this issue locally using the 1.3.2 version.
gakinson
on 27 Sep 2019
Hello, here are my notes on how I could get it working by duplicating your class and playing around with annotations.
Failed attempts
- Move the @JsonProperty annotation to the field and removing it from the "is" getter
- Tried add additional letters tot he prefix
Successful attempts
- Deleting the "isBase64Encoded()" and putting JsonProperty("isBase64Encoded") to the field isBase64Encoded
- Issue with this is that there is no getter now.
- Rename the "isBase64Encoded()" function to "getIsBase64Encoded()"
Still trying some others that are backwards compatible, but please address this in some way.
gakinson
on 27 Sep 2019
I am still getting the issue.
SAM version is 0.39.0
I am trying to follow this tutorial: https://aws.amazon.com/blogs/opensource/java-apis-aws-lambda/
and get the following error:
Invalid API Gateway Response Keys: {'base64Encoded'} in {'statusCode': 200, 'headers': {'Content-Type': 'application/json'}, 'body': '{"pong":"Hello, World!"}', 'base64Encoded': False}
Function returned an invalid response (must include one of: body, headers, multiValueHeaders or statusCode in the response object). Response received: {"statusCode":200,"headers":{"Content-Type":"application/json"},"body":"{\"pong\":\"Hello, World!\"}","base64Encoded":false}
It is Jan 2020, should this not be fixed by now?
afayes
on 7 Jan 2020
This is not related to SAM CLI and is a combination of AWS Lambda and they dependency you use: https://github.com/awslabs/aws-serverless-java-container/issues/262#issuecomment-536061559
SAM CLI only serialize the output of the container into json, therefore if you are running into this issue, something is wrong in your serialization within your code.
Closing as it is not something SAM CLI controls.
@afayes You are getting that error because the keys are wrong in your output (base64Encoded should be isBase64Encoded). See my earlier comment above: https://github.com/awslabs/aws-sam-cli/issues/1193#issuecomment-495004163
jfuss
on 7 Jan 2020
For Kotlin: Use @JvmName
@get:JvmName("getIsBase64Encoded") val base64Encoded: Boolean = false
pranjitk
on 15 Apr 2020
Here's an example trace running local (1.3.2)
› sam local start-api -t template.yaml
Mounting mnE2AB52C4 at http://127.0.0.1:3000/{proxy+} [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
Mounting None at http://127.0.0.1:3000/ [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2020-09-27 14:26:26 * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
Invoking com.example.ProxyHandler (java11)
Decompressing /Users/kirk/s/local/cdk-mn/backend/build/libs/demo-all.jar
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-java11:rapid-1.3.2.
Mounting /private/var/folders/66/_95_86950lb9drnn0s2dp5h40000gn/T/tmpyedmand9 as /var/task:ro,delegated inside runtime container
21:26:55.376 [main] INFO i.m.f.a.p.AbstractLambdaContainerHandler - Starting Lambda Container Handler
21:26:55.571 [main] INFO i.m.context.env.DefaultEnvironment - Established active environments: [ec2, cloud, function, lambda]
21:26:59.045 [main] DEBUG com.example.ProxyHandler - My handler bean is here.
START RequestId: de41ebe6-e81a-1f1d-d7c8-e9bedd8090ac Version: $LATEST
21:26:59.138 [main] DEBUG com.example.ProxyHandler - POST
21:26:59.138 [main] DEBUG com.example.ProxyHandler - /book
21:26:59.138 [main] DEBUG com.example.ProxyHandler - {"name": "foo"}
21:26:59.138 [main] DEBUG com.example.ProxyHandler - lambdainternal.api.LambdaContext@73a1e9a9
21:27:01.485 [main] DEBUG com.example.BookController - Request body is Book(name=foo)
21:27:01.596 [main] DEBUG com.example.ProxyHandler - Actual AwsProxyResponse is of type com.amazonaws.serverless.proxy.model.AwsProxyResponse
21:27:01.630 [main] DEBUG com.example.ProxyHandler - {"statusCode":200,"multiValueHeaders":{"Content-Type":["application/json"]},"body":"{\"name\":\"foo\",\"isbn\":\"f3634054-18be-4493-874a-ce0693724031\"}","isBase64Encoded":false}
END RequestId: de41ebe6-e81a-1f1d-d7c8-e9bedd8090ac
REPORT RequestId: de41ebe6-e81a-1f1d-d7c8-e9bedd8090ac Init Duration: 4678.88 ms Duration: 2510.43 ms Billed Duration: 2600 ms Memory Size: 512 MB Max Memory Used: 110 MB
Invalid lambda response received: Invalid API Gateway Response Keys: {'base64Encoded'} in {'statusCode': 200, 'multiValueHeaders': {'Content-Type': ['application/json']}, 'body': '{"name":"foo","isbn":"f3634054-18be-4493-874a-ce0693724031"}', 'base64Encoded': False}
2020-09-27 14:27:02 127.0.0.1 - - [27/Sep/2020 14:27:02] "POST /book HTTP/1.1" 502 -
Please notice the application is returning a correct proxy object (the json you see is from me serializing the AwsProxyResponse to JSON using jackson.
{"statusCode":200,"multiValueHeaders":{"Content-Type":["application/json"]},"body":"{\"name\":\"foo\",\"isbn\":\"f3634054-18be-4493-874a-ce0693724031\"}","isBase64Encoded":false}
But gateway is seeing something else
{'statusCode': 200, 'multiValueHeaders': {'Content-Type': ['application/json']}, 'body': '{"name":"foo","isbn":"f3634054-18be-4493-874a-ce0693724031"}', 'base64Encoded': False}
This leads to an HTTP response of BAD GATEWAY.
HTTP/1.0 502 BAD GATEWAY
Content-Length: 36
Content-Type: application/json
Date: Sun, 27 Sep 2020 21:27:02 GMT
Server: Werkzeug/1.0.1 Python/3.7.9
{
"message": "Internal server error"
}
Now this exact same code is deployed to api-gateway, and the response is correct
› http POST https://uu5dplmknj.execute-api.us-west-2.amazonaws.com/prod/book name=foo
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 60
Content-Type: application/json
Date: Sun, 27 Sep 2020 21:34:41 GMT
Via: 1.1 a22f4ead7e43f2bbb5f34d2cc611ecb4.cloudfront.net (CloudFront)
X-Amz-Cf-Id: XkUxvYLw9Lfez13Bjh3x_R9wOw3mqJbs_v-z_y2SWFZMzk8Aww44cw==
X-Amz-Cf-Pop: SFO5-C3
X-Amzn-Trace-Id: Root=1-5f710569-c68d1b5822258f0873d65848;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: Ti3IbF29vHcFg0Q=
x-amzn-RequestId: 7510b491-4dcb-4699-8af9-694745b49129
{
"isbn": "94f82841-f63e-4e18-9f29-2b459e0369f1",
"name": "foo"
}
Cloud watch logs don't show what lambda sends to gateway, but here's the comparable log output that goes with the above sam local log.
| timestamp | message |
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1601240544169 | START RequestId: d6139ad4-8c11-43ed-b677-b7265ef847ed Version: $LATEST |
| 1601240544307 | INFO i.m.f.a.p.AbstractLambdaContainerHandler- Starting Lambda Container Handler |
| 1601240546414 | INFO i.m.context.env.DefaultEnvironment- Established active environments: [ec2, cloud, function, lambda] |
| 1601240546414 | DEBUGcom.example.ProxyHandler- My handler bean is here. |
| 1601240546503 | DEBUGcom.example.ProxyHandler- POST |
| 1601240546503 | DEBUGcom.example.ProxyHandler- /book |
| 1601240546503 | DEBUGcom.example.ProxyHandler- {"name": "foo"} |
| 1601240546503 | DEBUGcom.example.ProxyHandler - lambdainternal.api.LambdaContext@4c402120 |
| 1601240551265 | DEBUGcom.example.BookController- Request body is Book(name=foo) |
| 1601240551524 | DEBUGcom.example.ProxyHandler- Actual AwsProxyResponse is of type com.amazonaws.serverless.proxy.model.AwsProxyResponse |
| 1601240551623 | DEBUGcom.example.ProxyHandler- {"statusCode":200,"multiValueHeaders":{"Content-Type":["application/json"]},"body":"{\"name\":\"foo\",\"isbn\":\"43094531-6441-4f0a-811c-76a4e3dcfd3a\"}","isBase64Encoded":false} |
| 1601240551643 | END RequestId: d6139ad4-8c11-43ed-b677-b7265ef847ed |
| 1601240551643 | REPORT RequestId: d6139ad4-8c11-43ed-b677-b7265ef847ed Duration: 5148.61 ms Billed Duration: 5200 ms Memory Size: 512 MB Max Memory Used: 179 MB Init Duration: 3092.41 ms |
| 1601240727148 | START RequestId: 4bb2d2d9-2fce-4d06-a5a6-5075c6c7b51b Version: $LATEST |
| 1601240727153 | DEBUGcom.example.ProxyHandler- GET |
| 1601240727153 | DEBUGcom.example.ProxyHandler- /book |
| 1601240727153 | DEBUGcom.example.ProxyHandler- null |
| 1601240727153 | DEBUGcom.example.ProxyHandler- lambdainternal.api.LambdaContext@7aaca91a |
| 1601240727158 | DEBUGcom.example.ProxyHandler- Actual AwsProxyResponse is of type com.amazonaws.serverless.proxy.model.AwsProxyResponse |
| 1601240727158 | DEBUGcom.example.ProxyHandler- {"statusCode":405,"multiValueHeaders":{},"isBase64Encoded":false} |
| 1601240727164 | END RequestId: 4bb2d2d9-2fce-4d06-a5a6-5075c6c7b51b |
| 1601240727164 | REPORT RequestId: 4bb2d2d9-2fce-4d06-a5a6-5075c6c7b51b Duration: 12.65 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 179 MB |
| 1601240994775 | START RequestId: cf2ed0f9-8290-46c2-980d-35ba96638140 Version: $LATEST |
| 1601240994780 | DEBUGcom.example.ProxyHandler- HEAD |
| 1601240994780 | DEBUGcom.example.ProxyHandler- /book |
| 1601240994780 | DEBUGcom.example.ProxyHandler- null |
| 1601240994780 | DEBUGcom.example.ProxyHandler- lambdainternal.api.LambdaContext@20435c40 |
| 1601240994781 | DEBUGcom.example.ProxyHandler- Actual AwsProxyResponse is of type com.amazonaws.serverless.proxy.model.AwsProxyResponse |
| 1601240994781 | DEBUGcom.example.ProxyHandler- {"statusCode":405,"multiValueHeaders":{},"isBase64Encoded":false} |
| 1601240994781 | END RequestId: cf2ed0f9-8290-46c2-980d-35ba96638140 |
| 1601240994782 | REPORT RequestId: cf2ed0f9-8290-46c2-980d-35ba96638140 Duration: 3.43 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 179 MB |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
kastork
on 27 Sep 2020
@jfuss
Looking at the object returned by the serverless java container (AwsProxyResponse), the properties are correct.
@JsonProperty("isBase64Encoded")
public boolean isBase64Encoded() {
return isBase64Encoded;
}
public void setBase64Encoded(boolean base64Encoded) {
isBase64Encoded = base64Encoded;
}
So I think upstream is not responsible for this.
In my comments above I serialized this object with Jackson, and as you can see, the serialization also results in isBase64Encoded.
In your comment above where you say "SAM CLI only serialize the output of the container into json", I believe you are saying that CLI serializes the AwsProxyResponse, if this is true, then it would appear that CLI's serialization of it is the cause of this.
There is a ticket on this problem over at serverless java container... https://github.com/awslabs/aws-serverless-java-container/issues/262
kastork
on 30 Sep 2020
Related issues
rhlsthrm
·
4Comments
zhangzhx
·
3Comments
joekiller
·
4Comments
jpbarto
·
4Comments
bjea
·
3Comments
Most helpful comment
It seems like this issue was fixed on aws-serverless-java-container-core side in 1.3.2, but I am still getting the same error.
Please, take a look at this
related issue