Aws-sam-cli: Container cannot be connected to host network

[uhef]

Description:

Trying to call service running on host is not possible since sam local container cannot connect to host network.

Steps to reproduce the issue:

1. Run a service on the host (in my case a PostgreSQL instance running on port 9999 on the host).
2. Do docker network ls and see that there is host network.
3. Run sam local and connect container to host network like this: sam local start-api --docker-network host
4. Make a DB query to localhost:9999 in the lambda code running in sam

Observed result:

APIError: 500 Server Error: Internal Server Error ("container cannot be disconnected from host network or connected to host network")

Expected result:

Client running in sam local container should be able to connect to DB running on host.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Ubuntu 16.04
Docker version 17.05.0-ce, build 89658be

Output of sam --version:

SAM CLI, version 0.6.0

Optional Debug logs:

+ sam local start-api --docker-network host --debug
2018-09-17 11:36:54 local start-api command is called
2018-09-17 11:36:54 2 resources found in the template
2018-09-17 11:36:54 Found Serverless function with name='ExampleFunction' and CodeUri='.'
2018-09-17 11:36:54 Trying paths: ['/home/xyz/.docker/config.json', '/home/xyz/.dockercfg']
2018-09-17 11:36:54 Found file at path: /home/xyz/.docker/config.json
2018-09-17 11:36:54 Found 'auths' section
2018-09-17 11:36:54 Found entry (registry=u'https://index.docker.io/v1/', username=u'xyz')
2018-09-17 11:36:54 Found entry (registry=u'https://12345.dkr.ecr.eu-west-1.amazonaws.com', username=u'AWS')
2018-09-17 11:36:54 Found entry (registry=u'6789.dkr.ecr.eu-west-1.amazonaws.com', username=u'AWS')
2018-09-17 11:36:54 Found entry (registry=u'12345.dkr.ecr.eu-west-1.amazonaws.com', username=u'AWS')
2018-09-17 11:36:54 http://localhost:None "GET /v1.35/_ping HTTP/1.1" 200 2
2018-09-17 11:36:54 2 resources found in the template
2018-09-17 11:36:54 Found '1' API Events in Serverless function with name 'ExampleFunction'
2018-09-17 11:36:54 Detected Inline Swagger definition
2018-09-17 11:36:54 Lambda function integration not found in Swagger document at path='/venues' method='get'
2018-09-17 11:36:54 Found '0' APIs in resource 'ServerlessRestApi'
2018-09-17 11:36:54 Removed duplicates from '0' Explicit APIs and '1' Implicit APIs to produce '1' APIs
2018-09-17 11:36:54 1 APIs found in the template
2018-09-17 11:36:54 Trying paths: ['/home/xyz/.docker/config.json', '/home/xyz/.dockercfg']
2018-09-17 11:36:54 Found file at path: /home/xyz/.docker/config.json
2018-09-17 11:36:54 Found 'auths' section
2018-09-17 11:36:54 Found entry (registry=u'https://index.docker.io/v1/', username=u'xyz')
2018-09-17 11:36:54 Found entry (registry=u'https://12345.dkr.ecr.eu-west-1.amazonaws.com', username=u'AWS')
2018-09-17 11:36:54 Found entry (registry=u'6789.dkr.ecr.eu-west-1.amazonaws.com', username=u'AWS')
2018-09-17 11:36:54 Found entry (registry=u'12345.dkr.ecr.eu-west-1.amazonaws.com', username=u'AWS')
2018-09-17 11:36:54 Mounting ExampleFunction at http://127.0.0.1:3000/venues [GET]
2018-09-17 11:36:54 You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2018-09-17 11:36:54 Localhost server is starting up. Multi-threading = True
2018-09-17 11:36:54  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
2018-09-17 11:37:19 Constructed String representation of Event to invoke Lambda. Event: {"body": null, "httpMethod": "GET", "resource": "/venues", "queryStringParameters": null, "requestContext": {"httpMethod": "GET", "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", "path": "/venues", "extendedRequestId": null, "resourceId": "123456", "apiId": "1234567890", "stage": "prod", "resourcePath": "/venues", "identity": {"accountId": null, "apiKey": null, "userArn": null, "cognitoAuthenticationProvider": null, "cognitoIdentityPoolId": null, "userAgent": "Custom User Agent String", "caller": null, "cognitoAuthenticationType": null, "sourceIp": "127.0.0.1", "user": null}, "accountId": "123456789012"}, "headers": {"Accept-Encoding": "gzip, deflate", "X-Forwarded-Port": "3000", "Connection": "keep-alive", "Accept": "*/*", "User-Agent": "PostmanRuntime/7.2.0", "Host": "127.0.0.1:3000", "X-Forwarded-Proto": "http", "Cache-Control": "no-cache", "Postman-Token": "82dbfff5-7f82-400c-9cc1-70c19755a98d"}, "stageVariables": null, "path": "/venues", "pathParameters": null, "isBase64Encoded": false}
2018-09-17 11:37:19 Found one Lambda function with name 'ExampleFunction'
2018-09-17 11:37:19 Invoking test-index.run (nodejs6.10)
2018-09-17 11:37:19 Environment variables overrides data is standard format
2018-09-17 11:37:19 Loading AWS credentials from session with profile 'None'
2018-09-17 11:37:19 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2018-09-17 11:37:19 Changing event name from before-call.apigateway to before-call.api-gateway
2018-09-17 11:37:19 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2018-09-17 11:37:19 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2018-09-17 11:37:19 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2018-09-17 11:37:19 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2018-09-17 11:37:19 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2018-09-17 11:37:19 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2018-09-17 11:37:19 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2018-09-17 11:37:19 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2018-09-17 11:37:19 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2018-09-17 11:37:19 Loading variable profile from defaults.
2018-09-17 11:37:19 Loading variable config_file from defaults.
2018-09-17 11:37:19 Loading variable credentials_file from defaults.
2018-09-17 11:37:19 Loading variable data_path from defaults.
2018-09-17 11:37:19 Loading variable profile from defaults.
2018-09-17 11:37:19 Loading variable credentials_file from defaults.
2018-09-17 11:37:19 Loading variable config_file from defaults.
2018-09-17 11:37:19 Loading variable profile from defaults.
2018-09-17 11:37:19 Loading variable metadata_service_timeout from defaults.
2018-09-17 11:37:19 Loading variable profile from defaults.
2018-09-17 11:37:19 Loading variable metadata_service_num_attempts from defaults.
2018-09-17 11:37:19 Loading variable profile from defaults.
2018-09-17 11:37:19 Looking for credentials via: env
2018-09-17 11:37:19 Looking for credentials via: assume-role
2018-09-17 11:37:19 Looking for credentials via: shared-credentials-file
2018-09-17 11:37:19 Found credentials in shared credentials file: ~/.aws/credentials
2018-09-17 11:37:19 Loading variable profile from defaults.
2018-09-17 11:37:19 Loading variable profile from defaults.
2018-09-17 11:37:19 Loading variable region from config file with value 'us-west-2'.
2018-09-17 11:37:19 Loading variable profile from defaults.
2018-09-17 11:37:19 Loading variable profile from defaults.
2018-09-17 11:37:19 Loading variable region from config file with value 'us-west-2'.
2018-09-17 11:37:19 Loading variable profile from defaults.
2018-09-17 11:37:19 Loading variable profile from defaults.
2018-09-17 11:37:19 Loading variable region from config file with value 'us-west-2'.
2018-09-17 11:37:19 Resolving code path. Cwd=/home/xyz/workspace/foo/bar/local_build, CodeUri=.
2018-09-17 11:37:19 Resolved absolute path to code is /home/xyz/workspace/foo/bar/local_build
2018-09-17 11:37:19 Code /home/xyz/workspace/foo/bar/local_build is not a zip/jar file
2018-09-17 11:37:19 Trying paths: ['/home/xyz/.docker/config.json', '/home/xyz/.dockercfg']
2018-09-17 11:37:19 Found file at path: /home/xyz/.docker/config.json
2018-09-17 11:37:19 Found 'auths' section
2018-09-17 11:37:19 Found entry (registry=u'https://index.docker.io/v1/', username=u'xyz')
2018-09-17 11:37:19 Found entry (registry=u'https://12345.dkr.ecr.eu-west-1.amazonaws.com', username=u'AWS')
2018-09-17 11:37:19 Found entry (registry=u'6789.dkr.ecr.eu-west-1.amazonaws.com', username=u'AWS')
2018-09-17 11:37:19 Found entry (registry=u'12345.dkr.ecr.eu-west-1.amazonaws.com', username=u'AWS')
2018-09-17 11:37:19 http://localhost:None "GET /v1.35/images/lambci/lambda:nodejs6.10/json HTTP/1.1" 200 None
2018-09-17 11:37:19 Looking for auth config
2018-09-17 11:37:19 Looking for auth entry for 'docker.io'
2018-09-17 11:37:19 Found u'https://index.docker.io/v1/'
2018-09-17 11:37:19 Found auth config
2018-09-17 11:37:21 http://localhost:None "POST /v1.35/images/create?tag=nodejs6.10&fromImage=lambci%2Flambda HTTP/1.1" 200 None

Fetching lambci/lambda:nodejs6.10 Docker container image......
2018-09-17 11:37:21 Mounting /home/xyz/workspace/foo/bar/local_build as /var/task:ro inside runtime container
2018-09-17 11:37:21 http://localhost:None "POST /v1.35/containers/create HTTP/1.1" 201 201
2018-09-17 11:37:21 http://localhost:None "GET /v1.35/containers/3f82b1088218100d717f95b2f4888406a87e7ea18a469677dc43769163f0623c/json HTTP/1.1" 200 None
2018-09-17 11:37:21 http://localhost:None "GET /v1.35/networks/host HTTP/1.1" 200 536
2018-09-17 11:37:21 http://localhost:None "POST /v1.35/networks/bcef0cb45fb4bd0cbabd58f232af039d4fc54af1d41c9a645ea3391cb47398a4/connect HTTP/1.1" 500 94
2018-09-17 11:37:21 http://localhost:None "GET /v1.35/containers/3f82b1088218100d717f95b2f4888406a87e7ea18a469677dc43769163f0623c/json HTTP/1.1" 200 None
2018-09-17 11:37:21 http://localhost:None "DELETE /v1.35/containers/3f82b1088218100d717f95b2f4888406a87e7ea18a469677dc43769163f0623c?force=True&link=False&v=False HTTP/1.1" 204 0
2018-09-17 11:37:21 Exception on /venues [GET]
Traceback (most recent call last):
  File "/home/xyz/.local/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/xyz/.local/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/xyz/.local/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/xyz/.local/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/xyz/.local/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/xyz/.local/lib/python2.7/site-packages/samcli/local/apigw/local_apigw_service.py", line 140, in _request_handler
    self.lambda_runner.invoke(route.function_name, event, stdout=stdout_stream, stderr=self.stderr)
  File "/home/xyz/.local/lib/python2.7/site-packages/samcli/commands/local/lib/local_lambda.py", line 80, in invoke
    self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
  File "/home/xyz/.local/lib/python2.7/site-packages/samcli/local/lambdafn/runtime.py", line 79, in invoke
    self._container_manager.run(container)
  File "/home/xyz/.local/lib/python2.7/site-packages/samcli/local/docker/manager.py", line 61, in run
    container.create()
  File "/home/xyz/.local/lib/python2.7/site-packages/samcli/local/docker/container.py", line 125, in create
    network.connect(self.id)
  File "/home/xyz/.local/lib/python2.7/site-packages/docker/models/networks.py", line 57, in connect
    container, self.id, *args, **kwargs
  File "/home/xyz/.local/lib/python2.7/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/xyz/.local/lib/python2.7/site-packages/docker/api/network.py", line 248, in connect_container_to_network
    self._raise_for_status(res)
  File "/home/xyz/.local/lib/python2.7/site-packages/docker/api/client.py", line 231, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/xyz/.local/lib/python2.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
APIError: 500 Server Error: Internal Server Error ("container cannot be disconnected from host network or connected to host network")
2018-09-17 11:37:21 127.0.0.1 - - [17/Sep/2018 11:37:21] "GET /venues HTTP/1.1" 502 -

[uhef]

I should add that at the same time I'm able to access the service running on host using just a plain busybox container and telnet:

 ⚙  ~  docker run -itd --network host --name net-test busybox
9e12af0c0951174acf8b345994dd832985ac11226e6c9c1b4a7295ddb828ca03
 ⚙  ~  docker exec -it net-test telnet localhost 9999
exit
Connection closed by foreign host

[stuartjennings]

For anyone running on a Mac, Docker for Mac allows you to connect to localhost from within a container by making requests to http://host.docker.internal.

[miguelut]

I'm having a similar issue. I ran into this issue in conjunction with #508. We have a business case where we need to have one lambda call several others in succession. We'd like to be able to test this locally. I feel like #508 gets us halfway there, but it still seems like I need to connect to the host network so I can hit the endpoints stood up by sam local start-lambda -- I keep getting connection refused [Edit: when I don't use the --docker-network host flag -- when I do use it I get the error above]. I think it's this issue. Any help out there available for this?

[jfuss]

@uhef Thanks for submitting the issue. It appears to me that docker requires a different way to attach the container to the host network, specifically though using the network_mode instead of connecting.

If my understanding is correct, what we are currently doing is incorrect in two ways:

1. We cannot support connecting to the host network
2. After we create the container, we attach the container to the network that is passed into the command. This causes the container to be attached to both bridge (the default network) and the one specified.

The first one is what you reported and we need to address. As for the second one I listed: I need to double check with some coworkers to see what the intended behavior is for this.

[jfuss]

@miguelut Yes #508 does solve that use-case. If you are running in windows or mac you should use @stuartjennings suggestion above. Docker for Linux does not have host.docker.internal yet, but you can find suggestions on connecting on their GH Issue.

[jfuss]

Looks like attaching to both bridge (the default) and the one specified by the --docker-network is expected. This is the help text for --docker-network: "Specifies the name or id of an existing docker network to lambda docker containers should connect to, along with the default bridge network. If not specified, the Lambda containers will only connect to the default bridge docker network."

[jfuss]

This was released with v0.6.2.

Closing

[ngarg-panw]

docker.for.mac.localhost worked for me.