Aws-sam-cli: Testing Api Gateway endpoint locally returning `No response from invoke container` and `Invalid lambda response received` in Dockerised sam app

[alliesground]

Description:

I'm dockerising sam app. When accessing api endpoint for HelloWorld locally, it responds with No response from invoke container for HelloWorldFunction and Invalid lambda response received: Lambda response must be valid json.

It works fine whithout dockerisation.

Steps to reproduce:

Dockerfile
FROM ruby:2.7.0-alpine

RUN apk add --update --no-cache \
build-base \
postgresql-dev \
postgresql-client \
python3 \
py3-pip \
util-linux \
python3-dev

RUN pip3 install aws-sam-cli

ARG USER
ARG HOME
ARG UID

RUN apk add --update \
sudo

RUN echo "Welcome home: $USER => $UID"

RUN adduser -S -D -G users -u $UID $USER
RUN addgroup -S sudo
RUN echo "%sudo ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/sudo
RUN adduser $USER sudo

RUN echo "Welcome home: $USER"

WORKDIR ${HOME}

EXPOSE 3001

ENTRYPOINT ["sh", "./entrypoint.sh"]

docker-compose.yml
version: '3.8'
services:
sam_app:
build:
context: ./sam-app
args:
- HOME
- USER
- UID
user: "${UID}:100"
command: ["$PWD"]
ports:
- "3001:3001"
volumes:
- ./sam-app:$HOME
- /var/run/docker.sock:/var/run/docker.sock

Entrypoint.sh
BASEDIR="$1"

echo "Basedir => ${BASEDIR}"

sudo sam local start-api \
--template ./template.yaml \
--host 0.0.0.0 \
--port 3001 \
--docker-volume-basedir "${BASEDIR}/sam-app/" \
--docker-network drink_default \
--debug

hello_world/app.rb
def lambda_handler(event:, context:)
{
statusCode: 200,
body: {
message: "Hello World!",
# location: response.body
}.to_json
}
end

Observed result:

am_app_1 | 2020-12-21 01:31:22,493 | Constructed String representation of Event to invoke Lambda. Event: {"body": null, "headers": {"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9", "Accept-Encoding": "gzip, deflate, br", "Accept-Language": "en-GB,en;q=0.9", "Connection": "keep-alive", "Host": "localhost:3001", "Sec-Fetch-Dest": "document", "Sec-Fetch-Mode": "navigate", "Sec-Fetch-Site": "none", "Sec-Fetch-User": "?1", "Upgrade-Insecure-Requests": "1", "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36", "X-Forwarded-Port": "3001", "X-Forwarded-Proto": "http"}, "httpMethod": "GET", "isBase64Encoded": false, "multiValueHeaders": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9"], "Accept-Encoding": ["gzip, deflate, br"], "Accept-Language": ["en-GB,en;q=0.9"], "Connection": ["keep-alive"], "Host": ["localhost:3001"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Upgrade-Insecure-Requests": ["1"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36"], "X-Forwarded-Port": ["3001"], "X-Forwarded-Proto": ["http"]}, "multiValueQueryStringParameters": null, "path": "/hello", "pathParameters": null, "queryStringParameters": null, "requestContext": {"accountId": "123456789012", "apiId": "1234567890", "domainName": "localhost:3001", "extendedRequestId": null, "httpMethod": "GET", "identity": {"accountId": null, "apiKey": null, "caller": null, "cognitoAuthenticationProvider": null, "cognitoAuthenticationType": null, "cognitoIdentityPoolId": null, "sourceIp": "172.22.0.1", "user": null, "userAgent": "Custom User Agent String", "userArn": null}, "path": "/hello", "protocol": "HTTP/1.1", "requestId": "dd99d1bb-cc40-42f8-a2f2-eeefcc22d111", "requestTime": "21/Dec/2020:01:30:38 +0000", "requestTimeEpoch": 1608514238, "resourceId": "123456", "resourcePath": "/hello", "stage": "Prod"}, "resource": "/hello", "stageVariables": null, "version": "1.0"}
sam_app_1 | 2020-12-21 01:31:22,493 | Found one Lambda function with name 'HelloWorldFunction'
sam_app_1 | 2020-12-21 01:31:22,494 | Invoking app.lambda_handler (ruby2.7)
sam_app_1 | 2020-12-21 01:31:22,494 | No environment variables found for function 'HelloWorldFunction'
sam_app_1 | 2020-12-21 01:31:22,494 | Environment variables overrides data is standard format
sam_app_1 | 2020-12-21 01:31:22,494 | Loading AWS credentials from session with profile 'None'
sam_app_1 | 2020-12-21 01:31:24,549 | Resolving code path. Cwd=/home/sameer/projects/drink/sam-app/, CodeUri=hello_world/
sam_app_1 | 2020-12-21 01:31:24,549 | Resolved absolute path to code is /home/sameer/projects/drink/sam-app/hello_world
sam_app_1 | 2020-12-21 01:31:24,550 | Code /home/sameer/projects/drink/sam-app/hello_world is not a zip/jar file
sam_app_1 | 2020-12-21 01:31:24,574 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-ruby2.7:rapid-1.15.0.
sam_app_1 |
sam_app_1 | 2020-12-21 01:31:24,574 | Mounting /home/sameer/projects/drink/sam-app/hello_world as /var/task:ro,delegated inside runtime container
sam_app_1 | 2020-12-21 01:31:25,386 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
sam_app_1 | 2020-12-21 01:31:26,707 | Cleaning all decompressed code dirs
sam_app_1 | 2020-12-21 01:31:26,707 | No response from invoke container for HelloWorldFunction
sam_app_1 | 2020-12-21 01:31:26,707 | Invalid lambda response received: Lambda response must be valid json
sam_app_1 | 2020-12-21 01:31:26 172.22.0.1 - - [21/Dec/2020 01:31:26] "GET /hello HTTP/1.1" 502 -
sam_app_1 | 2020-12-21 01:31:26 172.22.0.1 - - [21/Dec/2020 01:31:26] "GET /favicon.ico HTTP/1.1" 403 -

Expected result:

Json output on browser {message: "hello world"}

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: ubuntu VERSION="20.04.1 LTS (Focal Fossa)"
2. sam --version: 1.15.0
3. Homebrew: VERSION="2.6.2"
4. docker: VERSION="19.03.8"

Add --debug flag to command you are running

[jfuss]

@alliesground SAM CLI is meant to be run from your machine and interact with docker, not run SAM CLI within Docker. My suggestion here is to do that instead of Dockerize.

What's the use case on wanting to do this? Why do you need to be running docker within docker?

[rbliss]

I also get this exact error. I will chime in on my use case which may or may not be similar:

In my case, I'm using VSCode's Dev Containers, to create an isolated dev environment SAM is running in. My setup is nearly identical, using a volume to expose the host's docker socket, and SAM's --docker-volume-basedir flag to direct SAM to the appropriate location. Hitting the endpoint gives me the same No response from invoke container for HelloWorldLambda

Unfortunately, SAM's --debug flag doesn't go into much detail about what went on with the lambda container it spins up to serve the request. I started tracing through the CLI source to see what happened, but haven't had time to finish my search.

[alliesground]

@jfuss my use case is similar to that of @rbliss i.e to create an isolated dev environment, which includes react-app and db, all composed with docker-compose.

[alayor]

I'm also having the same problem. My use case is exactly the same as @alliesground.

I followed the instructions on this post. It makes me think this used to work before.
https://medium.com/monsoon-engineering/running-aws-sam-in-a-docker-container-2491596672c2

[gpomykala]

I receive same error when trying to hit a lambda spun with 'sam local start-lambda --docker-network host'. It used to work with older version of SAM CLI. I am not doing anything extraordinary, just a basic integration test of a lambda client hitting lambda hosted in SAM CLI

[jfuss]

@gpomykala Your issue is logged here: https://github.com/aws/aws-sam-cli/issues/2436#issuecomment-742242804 and workarounds here: https://github.com/aws/aws-sam-cli/issues/2436#issuecomment-743472832

@rbliss @alayor @alliesground
We have seen this happen when SAM CLI cannot communicate to the container. Instead of exec-ing within the container, we updated to move to publishing a port and then calling into the container of http. This could be the cause here as well. SAM CLI will communicate to the container over localhost only, so there is probably some docker network trickery going on. I highly suggest to not use SAM CLI with the container like this. It's not really a use case we directly support or recommend.

[alayor]

@jfuss Thank you for your response and links.
I wonder if there could be a way to change the "communicate to the container over localhost" setting.
I just wanted to test if changing "localhost" to "docker.for.mac.localhost" would make it work.
I know this is not recommended, but I figured I could ask anyway.

[jfuss]

@alayor I am open to ways to make this work if we can find a solution that works across OS. I am not sure if moving to the docker.for.mac.localhost (or host.docker.internal which I think might work across OS).

Just mapping this out:

• SAM runs in Container A on a bridge docker network (by default).
• Executing a sam local command will start up a container and publish ports to localhost (not sure if this is localhost of the machine or on the bridge docker network).
• SAM attempts to call localhost: to communicate to the container and fails to get a response.

If in step two, this does publish a port on the machines localhost, then I would assume what you are suggesting could work. Super hacky and seems much easier to just run the CLI on your machine and not have to care about docker at all. Could be a personal preference though.

[alliesground]

@jfuss thanks for your response. It was just an itch of mine to try to dockerize sam-cli especially after reading some articles on the possibilities. Anyways I will move forward with your advice and run CLI on the host machine and maybe try to connect it with other tech stacks running in the container.

[ilyasotkov]

I was able to get SAM local invoke to work inside Dev Container by replacing hardcoded localhost here

https://github.com/aws/aws-sam-cli/blob/develop/samcli/local/docker/container.py#L41

with host.docker.internal. I used this guide for attaching the VS Code debugger: https://github.com/aws/aws-toolkit-vscode/blob/a380685696e39247d6ec0f8f1e9928ba2562f550/docs/debugging-python-lambda-functions.md, also replacing localhost with host.docker.internal in launch.json.

@jfuss I'm running this on macOS, AFAIK this won't work on linux out of the box due to https://github.com/docker/for-linux/issues/264, but a SAM CLI option to specify host would be nice.