Moby: Document how to get real remote client ip for service running in container

Created on 28 Jul 2015  Â·  93Comments  Â·  Source: moby/moby

Hello,

I have an hard time figuring out how to get the real remote ip of a client connecting to a (web) service inside a container which port is exposed to a port on the host.

So far the solution seems to be either one of:

  • use --net=host
  • disable the userland proxy
  • configure nginx on the host to forward the real ip address?

Here are related issues:

https://github.com/docker/docker/issues/7540
https://github.com/jwilder/nginx-proxy/issues/130
https://github.com/jwilder/nginx-proxy/issues/133

None of the current solutions seems to be "right" and they all seem temporary. Can you clarify/explain how things _should_ be, even if that means that the current docker 1.7.1 is bugged in that regard and that we have to use hack X or Y until things are fixed in 1.8.0.

aredocs arenetworking

Most helpful comment

Solved the problem by abandoning docker for production, 2 years ago. :whistling:

All 93 comments

Hi!

Please read this important information about creating issues.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

This is an automated, informational response.

Thank you.

For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues


BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:
uname -a:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

ENEEDMOREINFO

Description of problem:

I have an hard time figuring out how to get the real remote ip of a client connecting to a (web) service inside a container which port is exposed to a port on the host.

So far the solution seems to be either one of:

  • use --net=host
  • disable the userland proxy
  • configure nginx on the host to forward the real ip address?

Here are related issues:

https://github.com/docker/docker/issues/7540
https://github.com/jwilder/nginx-proxy/issues/130
https://github.com/jwilder/nginx-proxy/issues/133

None of the current solutions seems to be "right" and they all seem temporary. Can you clarify/explain how things _should_ be, even if that means that the current docker 1.7.1 is bugged in that regard and that we have to use hack X or Y until things are fixed in 1.8.0.

docker version:

Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

docker info:

Containers: 1
Images: 508
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 510
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-58-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 4
Total Memory: 15.64 GiB
Name: philippe-desktop
ID: QLBB:VYAR:TKYL:NZQN:MGON:SNU4:7BLV:XFEJ:AIGR:LZY6:LCIF:GHBU

uname -a:

Linux philippe-desktop 3.13.0-58-generic #97-Ubuntu SMP Wed Jul 8 02:56:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

Physical

How reproducible:

Easily.

Steps to Reproduce:

  1. FIXME
    2.
    3.

Actual Results:

We get docker0 bridge ip.

Expected Results:

We get the remote client ip.

Additional info:

Creating issues in this format is a PITA and goes against github flow.

disable the userland proxy

This is the correct answer, and is soon to become the default action (see #14856)

_Edit: If this solution seems unsatisfactory, can you explain your thoughts on the matter?_

Alright, thanks!

@phemmer

Edit: If this solution seems unsatisfactory, can you explain your thoughts on the matter?

the problem is that if you use --iptables=false you can not disable userland proxy. Please take a look here
Will appreciate any suggestions. Thanks.

Ok, what's the latest recommended solution if --userland-proxy=false is unworkable (as described in #14856)?

  • use --net=host
  • disable the userland proxy
  • configure nginx on the host to forward the real ip address?

Also, can we re-open this issue if #14856 was reverted? Or has it been documented somewhere?

+1 for reopen, as "no userland proxy" is not viable at the moment.

+1 to reopen from me. Looking for the real IP address to set SSL via a reverse proxy

+1 this issue is really scary.

+1 we need more documentation on this subject and workable solution.

+1
receiving the wrong source IP inside the container results in problems with SPF validatio, spam scoring, access statistics etc.

+1
Need a way to handle geolocated requests.

+1 need to know which IP source connect to my app, and fail2ban if needed

Since we already have so many +1s, let's have another. And a minimal setup I used to test this & exhibit the problem: http://serverfault.com/questions/813298/

Another +1 here. Need to have true client IP address for logging and security purposes. Re-open with some documentation please :)

+1 here. Really need to get true client IP address.

+1 here. Really need to get true client IP address.

+1 here.

$ docker version
Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:44:32 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 21:44:32 2016
 OS/Arch:      linux/amd64
$ uname -r
3.13.0-100-generic

Failed using the geoip module....

+1 The NAT should only be used on outbound traffic and not on inbound traffic

+1 here

+1 here. Need the real IP address for access statistics, logging and security purposes.

+1 This is a real problem. We need the real IP for our firewall

+1. Please reopen the issue.

+1 IMO this should be flagged as critical and fixed ASAP

Looks like this is issue in IP6 module in Linux kernel and you might try to solve this by disabling ip6 in grub loader settings. I afraid that docker developers will not fix it.

@phpid using mac osx by the way :)

+1

+1

@phpid Do you have a link that further explains that?

Have not tested but the docker forum reckons you can get the real IP by using '--net=host'

+1

+1

+1

+1

+1. Need to reopen this issue ASAP...

+1
I tried starting the container with --net=host but in that case, the docker container gets a completely random ip address. The container becomes inaccessible and all the services can't be reached. This is really important.

+1

Try with a new Docker. They updated Linux kernel to 4.9.12 in v17.03.

I already have docker v17.03.0-ce. I still see the docker IP address and not the remote ip address.

+1

Given the amount of comments, I reopened this issue so docker's staff can somehow provide a final answer.

It's been a while I did not experience the problem, and I seem to recall that for me the problem only happened for requests that came from the same host (e.g curl http://localhost:8080), so it kinda made sense.

That said, given the testcases are out there and that this issue would be pretty dramatic if it happened all the time, maybe what needs to be done is find out _why_ it does not happen all the time. I know that it does not happen with straightforward NGinx containers on docker version 17.03.1-ce (default installation, default network, no fiddling with iptables, etc).

+1

Just to make sure everyone understands the issue, this "bug" only happen for requests to localhost:

philippe@pv-desktop:~$ ip a s eth0 | grep 10.10
    inet 10.10.10.248/24 brd 10.10.10.255 scope global dynamic eth0

philippe@pv-desktop:~$ docker run -it --rm -p 80:80 nginx
172.17.0.1 - - [20/Jun/2017:07:17:16 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
10.10.10.218 - - [20/Jun/2017:07:17:50 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
10.10.10.248 - - [20/Jun/2017:07:23:45 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"

The 1st request was curl localhost (or curl 127.0.0.1) on the same machine.
The 2nd request was curl 10.10.10.248 done on another computer.
The 3rd request was curl 10.10.10.248 done on the same machine.

We see that for localhost requests, the IP is the docker ip. For requests done on the remote interface, the IP is the correct one.

So the only bug we are talking about is that 127.0.0.1 ends up as the docker IP, which does no seem outrageous to me.

EDIT: guys, stop writing +1. Just upvote the comment that makes sense to you instead.

+1

I experienced this bug with a nginx based WAF. I could not get the clients ip real ip.

@Silex:
When using Docker for Mac and probably Docker for Windows as well, you will still only receive the IP of the Gateway, even though you're accessing the service from the internet.

docker run -it --rm -p 80:80 nginx

Request made from the internet (Internet -> Router (Port forward) -> Docker for Mac Host -> Service (cmd above):

172.17.0.1 - - [13/Jul/2017:15:54:50 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" "-"

Network config:

$ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "7465456abe9ea80d6072a250346e8e1404935186a1bead466349c884e97f7282",
        "Created": "2017-07-13T12:51:26.104444429Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

@dersimn: ah right, that kinda makes sense because docker runs in a VM in this case.

Can you try with --net=host ?

@Silex cannot use network_mode: host with networks: on docker-compose

--net=host is not working at all in _Docker for Mac_. See issue#68

This still exists even on 17.06.0-ce, build 02c1d87

+1 Would be nice to be able to specify that for a given port we want the packets arriving at the container to have originating IP/PORT preserved.

ports:
  - 1234:1234/udp+origin

Just stumbled upon this one, quite annoying... Any workaround? Seems to be only affecting MAC and Windows users?

I follow this issue quite some time now.

Do you guys have the _wrong_ ip when connecting to the server with IPv6 and/or with IPv4?
In my case only when connecting with v6 the _wrong_ ip shows up, obviously. Docker needs to translate v6 to v4 and vice versa.

I had similar issue combined with https://github.com/moby/moby/issues/28589. My situation (IPs are changed):

I have a host with public IP 1.2.3.11 with nginx listening on 0.0.0.0:80. On this host I have a container (with local IP 172.17.0.3) with web server which needs to listen on port 80 too. So I added another public IP 1.2.3.50 to the host (so now, the host has two public IPs on one interface) and then I wanted to forward the port 1.2.3.50:80 into the container.

This can not be done with --userland-proxy=true, because when I run docker run with --publish "1.2.3.50:80:80" Docker wants to bind port 1.2.3.50:80, but he can not, because of the nginx on host which listens on 0.0.0.0:80. Error:

docker: Error response from daemon: driver failed programming external connectivity on endpoint mycontainer (9084a8cd64a7d4c49582ab9ea149ffc93040a3e57d94e2277d61ab8f92a1f98b): Error starting userland proxy: listen tcp 1.2.3.50:80: bind: address already in use.

To workaround this error, I had to run docker run with --publish "1.2.3.50:8080:80". And forward the port using iptables:

iptables -t nat -A PREROUTING -d 1.2.3.50 -p tcp --dport 80 -j DNAT --to-destination 1.2.3.50:8080

But then the container sees only 172.17.0.1 as remote address (that is what this issue is about).

I should be able to solve it with --userland-proxy=false and then run docker run with --publish "1.2.3.50:80:80", but Docker still wants to bind something (see the https://github.com/moby/moby/issues/28589). Error:

docker: Error response from daemon: driver failed programming external connectivity on endpoint mycontainer (f3ccc9cc8eaf14f32a2342acaf40bb5d07f5dfd1ede11cd931f649ae819892eb): listen tcp 1.2.3.50:80: bind: address already in use.

Workaround:

Run docker run with --publish "1.2.3.50:8080:80" so it’s not in conflict with other services listening. You can use any port instead of 8080. Forward the port manually using iptables:

iptables -t nat -A PREROUTING -d 1.2.3.50 -p tcp --dport 80 -j DNAT --to-destination 172.17.0.3:80

Now the container sees real remote IP.

FYI -

Disabling userland proxy did not seem to work for me - the IP is still the docker's network interface IP rather than the client IP.

Trying to network_mode: host right now - but is that seriously the only way to get the client IP address? How are people building apps that log client IP addresses??

Found the solution in my wsgi app
request.environ.get('HTTP_X_REAL_IP', request.remote_addr)

Can someone please publish the full solution? That would help the larger community...

Smile!!! :) It improves your face value...

On Oct 3, 2017, at 8:07 AM, k2xl notifications@github.com wrote:

Found the solution in my wsgi app
request.environ.get('HTTP_X_REAL_IP', request.remote_addr)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/moby/moby/issues/15086#issuecomment-333871905, or mute the thread https://github.com/notifications/unsubscribe-auth/AYsXMsdH4zbTxiBJb7hs2iSc9aomndjUks5sok28gaJpZM4FhSSK.

@k2xl not all app based on http :(

I really need this to get working to separate different flow streams from routers (they're identified by their source IPs...)

Here's a solution that works for me. It's based on classic port forwarding (which @michallohnisky already pointed out above).

Principles

  • The docker iptables option is disabled as we manage iptables rules ourselves
  • We use a user-defined docker network with a dedicated private IP address range
  • Each container gets a static IP address in that network
  • The container does not expose (--expose) or publish (--publish or -p) any ports

Setup on Ubuntu 16.04

Disable docker's iptables

sudo systemctl stop docker

Create a file /etc/systemd/system/docker.service.d/noiptables.conf with this content:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --iptables=false
sudo systemctl daemon-reload
sudo systemctl start docker

Create user-defined docker network

docker network create -d bridge --subnet 192.168.0.0/24 --gateway 192.168.0.1 dockernet

Note: The --gateway option makes the host reachable under 192.168.0.1 from the containers.

Add custom rules to ufw (iptables)

We use ufw to manage our firewall. But it would also work with any other iptables based firewall.

First change the default forward policy to ACCEPT in /etc/default/ufw:

DEFAULT_FORWARD_POLICY="ACCEPT"

Then we add custom rules to the iptables NAT table. We use masquerading to let the containers connect to the internet and DNAT to forward ports from the host to specific containers. This is achieved by adding the following to the top of /etc/ufw/before.rules:

# nat Table rules
*nat

# Set default policies
:POSTROUTING ACCEPT [0:0]
:PREROUTING ACCEPT [0:0]

# Masquerade traffic from dockernet
-A POSTROUTING -s 192.168.0.0/24 -o eth0 -j MASQUERADE
# ... and from default docker network
-A POSTROUTING -s 172.17.0.0/16 -o eth0 -j MASQUERADE

# Port forwarding for specific services to docker container
-A PREROUTING -p tcp -i eth0 --dport 25 -j DNAT --to-destination 192.168.0.2:25
-A PREROUTING -p tcp -i eth0 --dport 80 -j DNAT --to-destination 192.168.0.2:80
# ... more DNAT rules here ...

# don't delete the 'COMMIT' line or these nat table rules won't be processed
COMMIT

Note Our public facing interface on the host is eth0. Your mileage may vary.

Now we can open the specified ports on the host and start the firewall:

sudo ufw allow 25
sudo ufw allow 80
sudo ufw enable

Configure container to use this network

Here's an example docker-compose.yml:

version: '3'
services:
    app:
        build: ./
        networks:
            dockernet:
                ipv4_address: 192.168.0.2
networks:
    dockernet:
        external: true

So far the only solution that worked for me was --net=host.
(that makes you not being able to use most of the other docker features....).
Any other strategy (without going crazy with iptables or other network "hacks"...) failed miserably.

I agree with @goetas . What I did for this was yes put the port mode to host

ports:
      - target: 8001
        published: 8001
        mode: host

But this will disable the ingress networking where you can access the port on any node of the Swarm. Basically, you will only access this/these ports if you type the actual address of that server on the browser's address bar. But the benefit of using mode host is it has lesser latency in the networking point of view. This is perfect for Nginx gateways.

+1

So I actually did go and disable the userland proxy in the Docker daemon. This didn't actually seem to fix getting the remote IP when navigating via localhost. In addition, since Chrome wants to resolve localhost to IPv6, requests won't actually get to the container without going through all the problems of getting IPv6 working within Docker. Firefox and CURL can be used to force an IPv4 localhost request though.

Setting the container to use host networking does work, but is suboptimal. I would be absolutely ok with disabling userland proxy if it fixes anything.

My current setup:

  • Kubuntu 17.10
  • Linux kernel 4.13.0-32
  • Docker 17.12.0-ce

I have tried disabling the userland proxy which renders my setup unusable as only the server can see access the port mapped containers.
All calls made externally to the server failed.
I have also tried to disable IPv6 as it was suggested in https://github.com/hardware/mailserver/issues/43 by doing the following:

/etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

However, it seemed to have no effect on my system.

@Steiniche When I ran into that, I found that if I added another entry in my /etc/hosts file with 127.0.0.1 localhost2, and I navigated to localhost2:<exposed_port>, I could actually get a request to the Docker instance. No clue as to why this is, but I guess the localhost hostname is super special. :/

Hello, got similar problem running rsyslog inside container. Workaround for me is to install conntrack tools on host and flush table after starting container (with conntrack -F).

In our case, we have connections coming in via IPv6, but Docker is replacing those with the IPv4 address of the docker0 gateway.

This should be the same question, right?

Correctly express "any address" to iptables.

3 years and counting.... +1

Well, the issue is still there..

'sup, Docker?

I think everyone would appreciate to see that feature. Because it is a big problem if all clients in your webapp have the "same IP adress"

3 years and counting.... +1

3 years and counting.... +1

Solved the problem by abandoning docker for production, 2 years ago. :whistling:

While it is interesting to note that people have been trying to get it working for several years now, I learnt yesterday (RTFM) that the host networking is only supported on Linux. I have been trying to get this working on my Windows box... When I tried this on my Linux machine it worked without a problem.

Though that raises a quesion: Why only on Linux and not on Windows. Is there a roadmap when this will be supported on Windows?

+1 , almost 4 yrs,

+1, disappointingly that a fundamental NAT networking behavior is broken and to date no proper documentation (I have scour thru' docs.docker.com for days) of the issue nor workaround.

Even more disheartening to chance upon this thread to know that this issue has been around for 4 years!

PS. Have tried disabling userland-proxy and it doesn't work.

3 years and counting.... +1

Though that raises a quesion: Why only on Linux and not on Windows. Is there a roadmap when this will be supported on Windows?

There's another issue related to this, but to be precise... it's the Linux CLI that works but not the Windows CLI when I did it. I had a system where Docker was running inside VirtualBox running linux guest in windows host before.

https://github.com/moby/moby/issues/25526#issuecomment-383227920

The bug I opened was this https://github.com/moby/moby/issues/38447

Don't understand the principle, but it works
It is not feasible to use similar configuration in your own project. Who will explain
https://github.com/cnzjh/ipinfo.tw

@cnzjh the principle is "simple": you need to add a "reverse-proxy" on top of your network.
The proxy will handle them and will reroute all the request internally without loosing the "correct" IP.

Problem is you can handle easily HTTP protocols (the system just add the Real IP in the HTTP Headers); instead classic TCP connections are another story (far from being simple)

For HTTP protocols here an example to build an autonomous reverse-proxy with traefik:

services:
  cloud-reverse-proxy:
    command:
      - '--logLevel=ERROR'
      - '--defaultentrypoints=https,http'
      - '--entryPoints=Name:http Address::80'
      - '--entryPoints=Name:https Address::443 TLS'
      - '--retry'
      - '--docker.endpoint=unix:///var/run/docker.sock'
      - '--docker.watch=true'
      - '--docker.exposedbydefault=false'
      - '--acme=true'
      - '--acme.entryPoint=https'
      - '--acme.httpchallenge'
      - '--acme.httpchallenge.entrypoint=http'
      - '--acme.domains=foo.mydomain.test'
      - '[email protected]'
      - '--acme.onHostRule=true'
      - '--acme.storage=acme.json'
    image: 'traefik:1.7.14-alpine'
    network_mode: host
    ports:
      - published: 80
        target: 80
      - published: 443
        target: 443
      - published: 4443
        target: 4443
    restart: always
    volumes:
      - 'cloud-reverse-proxy:/acme:rw'
      - '/var/run/docker.sock:/var/run/docker.sock:rw'
version: '3.7'
volumes:
  cloud-reverse-proxy:
    driver: local
    driver_opts:
      device: /whatever/dir/you/want
      o: bind
      type: none

And then, if you start any microservice by using the traefik labels, the reverse proxy will correctly send you the Real IP in the HTTP Headers:

networks:
  foo.network:
    name: foo.network
services:
  my-microservice:
    image: 'my-image:latest'
    labels:
      traefik.enable: 'true'
      traefik.frontend.passHostHeader: 'true'
      traefik.httpapi.frontend.rule: 'PathPrefix:/api'
      traefik.httpapi.port: '8080'
      traefik.mqttapi.frontend.rule: 'PathPrefix:/MQTT'
      traefik.mqttapi.port: '1884'
    networks:
      foo.network: {}
    restart: always
version: '3.7'

4+ years.. this is a critical issue, hope fix is on it's way!

Hello in 2020,

are there any updates on the topic?

I got it working fine using traefik v2 as reverse proxy, now I get the real ip of visitors.

I bet this won't be fixed by the end of this decade!

Can't this be fixed with some TPROXY tricks in iptables?

Using iptables -t nat -A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE was advised on many forums about this issue but it doesn't seem to work for me.

https://stackoverflow.com/questions/30383845/what-is-the-best-practice-of-docker-ufw-under-ubuntu

This sounds like a similar issue to #25526, which I found a workaround for.

Detailed here, in case it helps anybody: https://github.com/moby/moby/issues/25526#issuecomment-688323134

Never noticed this before until I tried running a process that limits the number of connections per IP address and everything broke :/

Internet -> Router (Port forward) -> Docker for Windows Host -> Service

All the service sees are connections from IP 172.17.0.1...

Was this page helpful?
0 / 5 - 0 ratings