Aws-sam-cli: Cannot connect to sam local from another docker container

[ferencbeutel4711]

Hi there,

I have an application which runs in a docker container on my local machine.
Sam local is running on the docker host machine and successfully reachable by the host. However, I am unable to reach sam local from within the mentioned docker container. If I start an express server on the host machine for example, I am able to reach it from within my docker container.
I think this might be an issue with the way sam local is running. If I understand correctly, sam local is running in a docker container aswell, so the routing in my mentioned example would be:

my docker -> host -> sam local docker -> sam local server

I have the assumption that the following is happening: my docker is sending the request against the host machine, which is sending it to the sam local docker. however, the sam local docker does not accept the request for some reason. Maybe it is a problem with the way incoming requests into sam local's docker container are handled?
I would be happy if you could provide some insights there and maybe assist with solving the issue. The testing setup should be easily setup: simple start a docker, ssh into it and try to telnet the docker host on the port sam local is running on.

Thanks!

[BeneStem]

Hi, a coworker of @ferencbeutel here.
Just wanted to add the information that this setup is running on a Mac with "docker for mac".
So the "my docker" is not running directly on the host "Mac" but on another virtual machine from "docker for mac" which is running on the host.
So the setup is more likely looking like this:

my docker -> docker for mac vm -> host -> docker for mac vm -> sam local docker -> sam local server

Just saying...

If sam-local is somehow restricting access to only be called from localhost this setup won't work, because you can't call localhost in "my docker" because localhost will be the "docker for mac vm" and not the real "host"... So the request to sam local is not coming from a localhost address but from the ip of the "docker for mac vm"...

[purefan]

the docker containers for each lambda are turned off until the lambda is needed: when an http request comes in the docker container for the lambda that handles it "wakes up", processes the request and goes back to sleep.

[ferencbeutel4711]

Hi @purefan, thanks for your reply. Sadly, that info does not help me with the issue.
To further clarify the problem:

With the native docker solution on ubuntu for example, everything works as expected. The problem does only occur with docker for mac
I think the root of the problem lies within sam local relying on localhost:

Lets assume that my public ip address under which I am available on the internet is xxx.xxx.xxx.xxx
If I run a simple server, a tomcat on port 9000 for example, I have no trouble doing the following:

telnet xxx.xxx.xxx.xxx 9000 => connected
telnet localhost 9000 => connected

If I run a sample docker container doing nothing but listening on port 9091, i have no trouble too:

telnet xxx.xxx.xxx.xxx 9091 => connected
telnet localhost 9091 => connected

however, if I run sam local on port 3000 for example, the following happens:

telnet xxx.xxx.xxx.xxx 3000 => connection refused
telnet localhost 3000 => connected

This is a problem since I am not able to access my local mashines "localhost" from inside another docker container for example! However, I am able to access my public ip from inside a docker container. So the issue is that sam local seems to not be bound to my public ip address but only on localhost.

[purefan]

Oh ok, yeah I understood something completely different. So just to be sure can you confirm if this is correct:

1) On Ubuntu:
1.1) You have a yml similar to this:

Resources:
  MockFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: nodejs6.10
      Handler: index.handler
      CodeUri: .
      Events:
        GetResource:
          Type: Api
          Properties:
            Path: /{proxy+}
            Method: get

1.2) if you start api you get something like this:

Mounting index.handler (nodejs6.10) at http://127.0.0.1:3000/{proxy+}

You can now browse to the above endpoints to invoke your functions.
You do not need to restart/reload SAM CLI while working on your functions,
changes will be reflected instantly/automatically. You only need to restart
SAM CLI if you update your AWS SAM template.

1.3) If you browse to http://127.0.0.1:3000/ your lambda is executed

2) On mac
2.1) You have the same yml file from 1.1
2.2) You get the same output from 1.2 when starting your api
2.3) Browsing to http://127.0.0.1:3000/ does not execute your lambda, the browser cannot find the route to your api or there is a network error before hitting your lambda

Is that correct? Im just trying to fully understand what the issue is

[ferencbeutel4711]

That is partially correct. The part about ubuntu is correct.
Sam Local is accessable for me under 127.0.0.1:3000 and works perfectly fine. However, I expected it to be accessable from the world wide web too, via my public ip address, just like any other docker container running on my machine, which is sadly not the case.

[ferencbeutel4711]

@purefan any update on this? I would like to help digging into this issue, however, i have zero knowledge about the startup process of sam local and would be glad if you could point me into the right direction there. I am particularly interested in the way the Docker instance is started, so the cli arguments and the Dockerfile used would be a great start.

What I find very strange is the fact that docker ps -a does not show your started docker instance.

[purefan]

There are a couple of things that pop to mind which might help you figure this out, one is that I would not assume that a docker container binds to a publicly accessible network, 127.0.0.1 is a loopback interface so anything on localhost can reach it, but I wouldn't assume thats open to an outside network. If I want to make sure that its open I would use something like ngrok during development

The docker containers are stopped until a request comes in, then the container is started, the request is processed, theres an http response and the container is stopped again. If you manually type docker ps you will hardly see the container, but you can test this by having 3 consoles open:

In console 1

Start your sam local

In console 2

watch docker ps

In console 3

while [ true ]; do curl http://127.0.0.1:3000/any-url-your-api-implements; done

As for the docker part I would begin with this line

[ferencbeutel4711]

Thanks for your answer. In the meantime, I discovered a new docker for mac feature which I wasnt aware of yet: docker.for.mac.localhost
This is a DNS name that docker for mac resolves to the ip address of the host. This is exactly what I needed in order to be able to reach sam local from inside another docker container, since I am able to reach sam local via docker.for.mac.localhost:{samLocalPort}
I am closing this issue. Thanks again for your help!

@BeneStem This feature is great! You should have a look aswell.

[BeneStem]

@ferencbeutel Bäm. Thank you. Great feature!

Greetings
/Ben

[tms1337]

Edit: I am still testing this approach, it does not really work since I have a Docker inside of a Docker so hostnames do not resolve correctly :(

If not using Mac, try starting sam local inside Docker container (we need to go deeper)

Take a look at this for more details:
https://github.com/cnadiminti/docker-aws-sam-local

This is the example usage in docker-compose:

sam_local_service:
    image: 'cnadiminti/aws-sam-local'
    volumes:
      - ${LAMBDA_CODE_WORKDIR}:/var/opt
      - /var/run/docker.sock:/var/run/docker.sock
    command: ["local", "start-api"]

This way you just link sam_local_service and put it into the same network and you can access sam local api via sam_local_service:3000