Aws-sam-cli: Feature Request: Support large event payload input to Lambda

[GeoffreyBooth]

I’m trying to build an API endpoint lambda function that accepts an uploaded binary file.

Try this:

git clone [email protected]:awslabs/aws-sam-local.git
cd aws-sam-local/samples/api-event-source
sam local start-api

and then in a new shell, from the aws-sam-local folder:

curl --request POST --url 'http://localhost:3000/' --data-binary '@./sam-banner.png'

The response I get to curl is { "message": "Internal server error" }.

Back in the first shell, I see this output:

2017/11/01 23:05:55 Connected to Docker 1.32
2017/11/01 23:05:55 Fetching lambci/lambda:nodejs6.10 image for nodejs6.10 runtime...
nodejs6.10: Pulling from lambci/lambda
Digest: sha256:7eb4ced6a15ae3c30effc4ec0cd3aabb2bd57c9a8330b37920c3d5d722d81083
Status: Image is up to date for lambci/lambda:nodejs6.10

Mounting index.handler (nodejs6.10) at http://127.0.0.1:3000/ [OPTIONS GET HEAD POST PUT DELETE TRACE CONNECT]
Mounting static files from public at /

You can now browse to the above endpoints to invoke your functions.
You do not need to restart/reload SAM CLI while working on your functions,
changes will be reflected instantly/automatically. You only need to restart
SAM CLI if you update your AWS SAM template.

2017/11/01 23:06:19 Invoking index.handler (nodejs6.10)
standard_init_linux.go:185: exec user process caused "argument list too long"
2017/11/01 23:06:25 Function index.handler timed out after 3 seconds
2017/11/01 23:06:25 Error invoking nodejs6.10 runtime: io: read/write on closed pipe

If I remove the --data-binary, to send a POST request without a body, I get the expected OK response.

If I change --data-binary '@./sam-banner.png' to --data-binary '@./README.md', replacing the 129 KB file with a 21 KB one, I also get the OK response. So I gather that there must be some (fairly low) limit on request size? Is there some way I can increase it?

Also, I feel like aws-sam-local should print an informative error message, rather than (I presume) passing invalid parameters onward to Docker for it to throw an exception on.

[kaarthikeyansp]

+1

[neilgirardi]

+1

[neilgirardi]

According to this page: http://docs.aws.amazon.com/lambda/latest/dg/limits.html, when invoking an Event, there is a 128K size limit on the payload.

[PaulMaddox]

Interesting bug - thanks @GeoffreyBooth for the easily reproducible test case.

We pass the event data (the contents of sam-banner.png in your example) over stdin to the Lambda runtime Docker container, which seems to be triggering a bug in github.com/opencontainers/runc.

I need to investigate this some more.

[shinichy]

+1

[mhart]

docker-lambda supports the AWS_LAMBDA_EVENT_BODY env variable if the event body is not passed over the command line – should work in all supported runtimes, but be sure to test

[fakataha]

+1

[mhart]

Just added support to docker-lambda to pass the event body via stdin

[frob]

I am a bit confused by this. Should I be setting DOCKER_LAMBDA_USE_STDIN to something?

[alecl]

Is the alternate input to docker lambda going to be integrated into SAM?

[ngmiller]

@frob Did you figure out how to properly set and use that environment variable? I'm running into this issue as well and setting DOCKER_LAMBDA_USE_STDIN to either 1 or true in my host shell is not working. I suspect it needs to be passed directly to the underlying docker instance somehow, but the sam local start-api options do not provide any means of passing env vars to that instance.

[frob]

I have not figured it out. For larger api tests I have been publishing to aws for the test.

[mster]

@frob have been forced to do the same.

[pcariel]

something new ?

[joain946]

I'm also interested if there is a plan to fix this. As I understand it this is not an AWS limit since for synchronous calls the payload limit is 6 MB. But instead this is related to something with the Docker integration

This problem will be triggered by binary files larger than around 95 kB since those will be Base64 encoded behind the scene which increases the file size with factor 1.33 which will then trigger the bug with the 128 kB payload limit.

[jrkienle]

Is there a PR out / plan to fix this? This issue has been open since 2017

[Nistrul]

+1

[Nop0x]

+1

[Dhwanil-Patel]

Is there any update? Or some intermediate solution?

[hdgone]

uploading files through sam cli seems to be a dead option. Way too much pain and nothing helps. Also no support given

[DKurilo]

I wrote some solution to make it working.
There are no answer in my PR, I don't know if it will be merged or at least commented, but you can pull my branch, build it with make init and to use it.
I did it to be able to debug app to work with images. As some test project I used this one:
https://github.com/DKurilo/hello-sam
You can try how it works with sam and with samdev that is built from my branch.

[boompig]

@PaulMaddox is there any update on this? I also ran into this issue. It would be nice at the very least to get a more informative message from SAM than

standard_init_linux.go:211: exec user process caused "argument list too long"

Perhaps for now SAM can just look at payload size and determine if it's too large to send over the Docker interface, then error with an informative message like "we're sorry, due to limitations with Docker we do not allow payloads larger than XX KB". This may not be a perfect solution, but it would at least be helpful in this case.

[PaulMaddox]

Sorry @boompig, I am no longer actively involved in the development of SAM CLI, and haven't been for a few years (since it was rewritten in Python).

Hopefully one of the SAM CLI team can give an update.

[DKurilo]

@boompig , looks like now we need to ask @jfuss about it.

[pragmasoft-ua]

According to this page: http://docs.aws.amazon.com/lambda/latest/dg/limits.html, when invoking an Event, there is a 128K size limit on the payload.

Invocation payload (request and response) | 6 MB (synchronous)

[BenVosper]

Do any contributors have thoughts on this? It would be a shame if this issue got buried. @DKurilo 's PR (https://github.com/aws/aws-sam-cli/pull/2105) seems to be a solution

[mosasiru]

I'm also suffer from this issue severely. I cannot post a 107KB file.

[anubhavmalik]

Having the same issue. Does anyone know a workaround?

[rynoV]

Also having this issue.