Emqx: listener.ws.external.proxy_address_header not work.

[PMExtra]

Environment

• Docker emqx/emqx:v3.2.3

Description

I used nginx as wss reverse proxy of emqx. And I configured them follow the official document and https://github.com/emqx/emqx/issues/1335#issuecomment-361044491 .

But the emqx still not get the real ip.

cat /opt/emqx/etc/emqx.conf | grep listener.ws.external.proxy:

listener.ws.external.proxy_address_header = X-Forwarded-For
listener.ws.external.proxy_port_header = X-Forwarded-Port
## listener.ws.external.proxy_protocol = on
## listener.ws.external.proxy_protocol_timeout = 3s
## See: listener.ws.external.proxy_address_header
## See: listener.ws.external.proxy_port_header

nginx configuration:

server {
    listen       8084 ssl http2;
    server_name  mqtt.mydomain.tld;

    ssl_certificate     /etc/nginx/cert.d/mycert.cer;
    ssl_certificate_key /etc/nginx/cert.d/mycert.key;

    location /mqtt {
        proxy_pass http://emqx:8083/mqtt;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

mydomain.tld is just for protecting the privacy, the actual config is set to the correct domain.

Now I can connect it from the client successfully and everything is awesome except the client IPAddr and Port.

Please help me.

[PMExtra]

I know I can use the proxy protocol v2, but I still want to know why the HTTP Header Solution doesn't work.

[tigercl]

Hi, @PMExtra.
I think you should configure listener.wss rather than listener.ws.

[PMExtra]

@tigercl Really? The SSL connection are terminated by Load Balance (reverse proxy). And it proxy pass to the backend(emqx) with http (ws) protocol. Are you sure I should configure listener.wss ?

[tigercl]

@PMExtra I'm sorry I misunderstood. Can you capture network packages to confirm that HTTP request header received in EMQ X contain X-Forwarded-For?

[Jonkimi]

same problem, package captured:

[PMExtra]

This is really a bug.

[sh3d2]

Hi,

is this ever going to get fixed?

as of version 4.0.7 i get

emqx: 2020-06-01T13:33:00.533657+02:00 [error] You've tried to set listener.ws.external.proxy_address_header, but there is no setting with that name.
emqx: 2020-06-01T13:33:00.533929+02:00 [error] Did you mean one of these?
emqx: 2020-06-01T13:33:00.661287+02:00 [error] listener.ws.$name.verify_protocol_header
emqx: 2020-06-01T13:33:00.661484+02:00 [error] listener.ws.$name.proxy_protocol
emqx: 2020-06-01T13:33:00.661644+02:00 [error] listener.wss.$name.verify_protocol_header
emqx: 2020-06-01T13:33:00.729780+02:00 [error] Error generating configuration in phase transform_datatypes
emqx: 2020-06-01T13:33:00.730014+02:00 [error] Conf file attempted to set unknown variable: listener.ws.external.proxy_address_header

[HJianBo]

Hi guys:
The ws/wss proxy header configuration options only work well on v3.x and v2.x. E.g:

listener.ws.external.proxy_address_header = X-Forwarded-For
listener.ws.external.proxy_port_header = X-Forwarded-Port

And after v4.0 these contents have changed. These mentioned options are not works between v4.0 and v4.0.5

We have fixed it at 4.1-alpha.2 and v4.0.6: https://github.com/emqx/emqx/pull/3372.

In the current version, we have removed the proxy_address_header and proxy_port_header. These options are not necessary. Theproxy_protocol` is enough and it keeps accordance for other listeners

e.g:

In the emqx.conf:

listener.ws.external.proxy_protocol = on
listener.ws.external.proxy_protocol_timeout = 3s

and on the LB side, just enable the proxy-protocol option

[PMExtra]

@HJianBo We want to terminate SSL/TLS at the LB, because the network that between emqx adn LB is trusted, that make the lower cost on emqx backend.
So I think support layer 7 proxy with X-Forwarded-For header is indeed necessary.

[HJianBo]

That's ok. The proxy-protocol is layer 4 proxy. It is a more efficient solution that X-Forwarded-*.

see: https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/

[PMExtra]

@HJianBo That's what I said. We need a layer 7 proxy.
Could you terminate SSL/TLS at the LB while using the layer 4 proxy? I don't think so.
So we need a layer 7 proxy that support a way to tell the backend what is the real remote ip.

[sh3d2]

@PMExtra @HJianBo actually i managed to get it to work with haproxy and proxy protocol turned on. i can see real ips in the emqx dashboard.
but, with

listener.ws.external.proxy_protocol = on
listener.ws.external.proxy_protocol_timeout = 3s

configured, i cannot connect to emqx directly in any way (via websocket).

[HJianBo]

@PMExtra

Could you terminate SSL/TLS at the LB while using the layer 4 proxy

Of course. Just like this deployment:

       wss                   ws (with proxy-protocol)
Client ------>  HAProxy/NGINX  ----------> EMQ X

@sh3d2 Is it v4.1.0 ?

[sh3d2]

@HJianBo 4.0.7. will try with 4.1.0 on monday

[PMExtra]

@HJianBo All right, Thank you for answering my question. I'll read the docs more carefully.

[sh3d2]

@HJianBo unfortunately same thing with 4.1.0

Jun  6 15:45:10.449 dev-push-emqx0 <info> emqx: 2020-06-06 15:45:10.448 [error] Ranch listener 'mqtt:ws:9001' had connection process started with cowboy_clear:start_link/4 at <0.5444.0> exit with reason: {{badmatch,{error,protocol_error,'The PROXY protocol header signature was not recognized. (PP 2.1, PP 2.2)'}},[{cowboy_clear,connection_process,4,[{file,"/emqx-rel/_build/emqx-pkg/lib/cowboy/src/cowboy_clear.erl"},{line,38}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}

with proxy protocol on its not possible to connect to emqx via websocket when proxy protocol header is not available (ex bypassing haproxy/nginx)

[nightwolfz]

Having the same issue with websockets. Can we bring proxy_address_header and port back?

Now I'm stuck on old version forever...