Addons: NGINX HA SSL proxy - websocket forwarding?

[dedepene]

I'm running into an issue with the ESPHome addon, and probably others too. What happens is that the addon loads its webui fine, but all subsequent requests are denied (i.e. "show logs", "validate" etc.). A lot of entries like this in the esphome addon log:

WARNING 403 GET /ace (0.0.0.0) 1.93ms
WARNING 403 GET /logs (0.0.0.0) 1.90ms
WARNING 403 GET /ace (0.0.0.0) 1.61ms
WARNING 403 GET /ace (0.0.0.0) 1.80ms
WARNING 403 GET /ace (0.0.0.0) 1.94ms
In the meantime, if I load the local instance (http://192...) - it works fine. So I googled a bit and it seems to be a websocket forwarding issue, but I'm lost on this one. Any idea how to configure nginx to do this please?

Thanks!

[pvizeli]

You speak about the addon?

[dedepene]

You speak about the addon?

I'm asking for a little help configuring the NGINX HA SSL addon so it doesn't break the other addons - e.g. ESPHOME

[pvizeli]

The addon work in default configuration with ingress. Can you post the config they you did?

[dedepene]

Sure - the settings and some errors from the nginx addon log below. Nothing fancy in the config - just added domain and changed the SSL port.

https://imagebin.ca/v/5B960bbKBSsx

2020/02/04 08:50:24 [error] 33#33: *122818 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 192.168.1.1, server: domain.duckdns.org, request: "GET /api/hassio_ingress/CL8o3ozNBPdXQ-NmyQnYZlWXbf9wqTPqWYvxgVPsSz8/ace HTTP/1.1", upstream: "http://172.30.32.1:8123/api/hassio_ingress/CL8o3ozNBPdXQ-NmyQnYZlWXbf9wqTPqWYvxgVPsSz8/ace", host: "domain.duckdns.org:8124"
2020/02/04 08:50:29 [error] 33#33: *122827 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 192.168.1.1, server: domain.duckdns.org, request: "GET /api/hassio_ingress/CL8o3ozNBPdXQ-NmyQnYZlWXbf9wqTPqWYvxgVPsSz8/ace HTTP/1.1", upstream: "http://172.30.32.1:8123/api/hassio_ingress/CL8o3ozNBPdXQ-NmyQnYZlWXbf9wqTPqWYvxgVPsSz8/ace", host: "domain.duckdns.org:8124"
2020/02/04 08:50:30 [error] 33#33: *122829 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 192.168.1.1, server: domain.duckdns.org, request: "GET /api/hassio_ingress/CL8o3ozNBPdXQ-NmyQnYZlWXbf9wqTPqWYvxgVPsSz8/ace HTTP/1.1", upstream: "http://172.30.32.1:8123/api/hassio_ingress/CL8o3ozNBPdXQ-NmyQnYZlWXbf9wqTPqWYvxgVPsSz8/ace", host: "domain.duckdns.org:8124"

Also, not sure if it's related but I get a lot of these in hassio log itself:

2020-02-04 11:39:50 WARNING (MainThread) [aiohttp.websocket] websocket connection is closing.
2020-02-04 11:40:36 WARNING (MainThread) [aiohttp.websocket] websocket connection is closing.

[dedepene]

Now all of a sudden, having changed my WAN interface on the router to another port, I'm no longer able to access HA via SSL and external IP/DNS!!! The LAN access is still there. A bunch of these appeared:

20-02-04 13:24:08 INFO (MainThread) [hassio.api.proxy] Home Assistant WebSocket API request initialize
20-02-04 13:24:08 INFO (MainThread) [hassio.api.proxy] WebSocket access from a0d7b954_nodered
20-02-04 13:24:09 ERROR (MainThread) [hassio.api.proxy] Client error on WebSocket API Cannot connect to host 172.30.32.1:8123 ssl:False [Connection refused].
20-02-04 13:24:14 INFO (MainThread) [hassio.api.proxy] Home Assistant WebSocket API request initialize
20-02-04 13:24:14 INFO (MainThread) [hassio.api.proxy] WebSocket access from a0d7b954_nodered
20-02-04 13:24:14 ERROR (MainThread) [hassio.api.proxy] Client error on WebSocket API Cannot connect to host 172.30.32.1:8123 ssl:False [Connection refused].

[dedepene]

I had to revert back to the original port configuration (why?!) in order to have HA accessible via SSL from the outside. Again, ESPHome loads the web UI but whatever i sclick from there I get "terminated process" and nothing happens. Getting these in the meantime:

2020-02-04 16:10:28 WARNING (MainThread) [aiohttp.websocket] websocket connection is closing.
2020-02-04 16:10:57 WARNING (MainThread) [aiohttp.websocket] websocket connection is closing.
2020-02-04 16:11:03 WARNING (MainThread) [aiohttp.websocket] websocket connection is closing.

20-02-04 14:00:07 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:09 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:13 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:15 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:19 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:21 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:25 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:27 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:31 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:33 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:37 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:39 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:43 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:45 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:49 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:51 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:55 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:00:57 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:01:01 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:01:03 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace
20-02-04 14:01:07 ERROR (MainThread) [hassio.api.ingress] Ingress error: 403, message='Invalid response status', url='http://172.30.32.1:62520/ace 

from the ESPHOME addon log:

WARNING 403 GET /ace (0.0.0.0) 3.11ms
WARNING 403 GET /ace (0.0.0.0) 2.01ms
WARNING 403 GET /ace (0.0.0.0) 2.61ms
WARNING 403 GET /ace (0.0.0.0) 1.50ms
WARNING 403 GET /ace (0.0.0.0) 1.81ms
WARNING 403 GET /ace (0.0.0.0) 2.45ms

[dedepene]

seriously? i'm the only one with this problem???

here's more from the nginx proxy addon log when trying to interact with esphome over ssl:

2020/02/10 20:35:09 [error] 34#34: *122547 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 192.168.1.1, server: domain.duckdns.org, request: "GET /api/hassio_ingress/CL8o3ozNBPdXQ-NmyQnYZlWXbf9wqTPqWYvxgVPsSz8/ace HTTP/1.1", upstream: "http://172.30.32.1:8123/api/hassio_ingress/CL8o3ozNBPdXQ-NmyQnYZlWXbf9wqTPqWYvxgVPsSz8/ace", host: "domain.duckdns.org:8124"
2020/02/10 20:35:14 [error] 34#34: *122553 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 192.168.1.1, server: domain.duckdns.org, request: "GET /api/hassio_ingress/CL8o3ozNBPdXQ-NmyQnYZlWXbf9wqTPqWYvxgVPsSz8/ace HTTP/1.1", upstream: "http://172.30.32.1:8123/api/hassio_ingress/CL8o3ozNBPdXQ-NmyQnYZlWXbf9wqTPqWYvxgVPsSz8/ace", host: "domain.duckdns.org:8124"

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dedepene]

Still having the same issues as above. Now also getting this in ESPhome logs:

WARNING Choked at offset 52 while unpacking b'\x00\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x02\x0bbathroom_sw\x0b_esphomelib\x04_tcp\x05local\x00\x00\xff\x80\x01\xc04\x00\x0c\x00\x01\x00\x00\x11\x94\x00\x02\xc0V\xc0V\x00!\x00\x01\x00\x00\x11\x94\x00\x08\x00\x00\x00\x00\x17\xa5\xc0\x8c\xc0V\x00\x10\x00\x01\x00\x00\x11\x94\x00)\x19address=bathroom_sw.local\x0eversion=1.14.2\xc0\x8c\x00\x01\x00\x01\x00\x00\x00x\x00\x04\xc0\xa8\x01\xeb'
WARNING Exception occurred:
Traceback (most recent call last):
  File "/opt/esphome/esphome/zeroconf.py", line 203, in __init__
    self.read_others()
  File "/opt/esphome/esphome/zeroconf.py", line 252, in read_others
    domain = self.read_name()
  File "/opt/esphome/esphome/zeroconf.py", line 308, in read_name
    "Bad domain name (circular) at %s" % (off,))
esphome.zeroconf.IncomingDecodeError: Bad domain name (circular) at 52

[Lanjelin]

Haven't tried the addon, but runnin nginx on another server, and configured this to forward to my HA rPi. To get the websockets working, I had to add the following to my *.conf

location /api/websocket {
    proxy_pass http://192.168.1.xxx:8123/api/websocket;
    proxy_set_header Host $host;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

[dedepene]

@pvizeli any idea how to configure the nginx proxy addon, being in the official repo and all....?

[dedepene]

Haven't tried the addon, but runnin nginx on another server, and configured this to forward to my HA rPi. To get the websockets working, I had to add the following to my *.conf

location /api/websocket {
    proxy_pass http://192.168.1.xxx:8123/api/websocket;
    proxy_set_header Host $host;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

Unfortunately, this didn't work in the docker environment the addon operates in :( I am amazed no one else has this problem and the people maintaining this project deny this is an issue :(

With your config I get:

2020/04/01 10:00:13 [error] 41#41: *124 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 192.168.1.1, server: XXXXX.duckdns.org, request: "GET /api/hassio_ingress/ntoRmZ_pKvfRi3teEkwTtfpH5pnmtoWtJ1BDzAkfbCs/ace HTTP/1.1", upstream: "http://172.30.32.1:8123/api/hassio_ingress/ntoRmZ_pKvfRi3teEkwTtfpH5pnmtoWtJ1BDzAkfbCs/ace", host: "XXXXX.duckdns.org:8124"

[netweaver1970]

same issue here, I'm running a letsencrypt docker image, using the build-in nginx to rev. proxy to my node-red, HomeAssistant, portainer, grafana web servers in a subdomain way.

All these are set up user Docker-compose. Next to that I have hass.io running on the same machine, with few add-ons, incl. esphome.

But I don't manage to get the ESPHOME add-on websocket interface to be reachable from outside. Same errors as above.
This is my current full HomeAssistant nginx config (as used by the letsencrypt docker image):

What can I do to help debug this issue?
Thanks for thinking along.

`server {
listen 443 ssl;
server_name ha.*;
include /config/nginx/ssl.conf;
client_max_body_size 0;

location / {
    include /config/nginx/proxy.conf;
    resolver 127.0.0.11 valid=30s;
    set $upstream_homeassistant homeassistant;
    proxy_pass http://192.168.0.10:8123;
}

location /api/websocket {
    resolver 127.0.0.11 valid=30s;
    proxy_pass http://192.168.0.10:8123/api/websocket;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    #proxy_redirect off;
    #proxy_ssl_verify off;
}

}
`

[dedepene]

OK, I had it sorted out on my end, but my setup is way simpler - just hassio with the nginx reverse proxy add-on. Thing is, I had the addon configured to use 8124 locally for SSL....so as soon as I forwarded 443 WAN to 8124 on the hassio machine and it's all good now - ingress for esphome works fine.

[ludeeus]

@dedepene Nice!
@netweaver1970 You should try the forum or the discord server 👍
https://github.com/home-assistant/hassio-addons/tree/master/nginx_proxy#support

[cajuncoding]

Same/similar issues here... my system is fully configured for external access and everything works well except for some HassIO plugins that attempt to establish web socket connections to /api/hassio_ingress.... which fail unless I access directly via IP address on my internal network...

I'm using LetsEncrypt Docker (linuxserver/letsencrypt) on an Unraid server for all reverse-proxy routing, and had gotten it all working well, using the sample configuration for HomeAssistant provided in the docker (i.e. .../nginx/proxy-conf/homeassistant.subdomain.conf.sample).

However, that sample was setup for a standard core HomeAssistant install and not HassIO, which I've finally figured out requires additional proxy configuration for websockets to the api/hassio_ingress route.

I finally got my add-ons working by adding the following location configuration to my Nginx proxy configuration (in addition to the other one for the default HomeAssistant websockets api route: api/websockets):

# Duplicate websocket configuration specifically for HassIO add-ons (e.g. /api/hassio_ingress)
# Details of how this config works can be read here:
# https://www.serverlab.ca/tutorials/linux/web-servers-linux/how-to-proxy-wss-websockets-with-nginx/
location /api/hassio_ingress {
        resolver 127.0.0.11 valid=30s;
        set $upstream_app 192.168.1.???;
        set $upstream_port 8123;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

        proxy_set_header Host $host;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
}

I hope this helps some others out since this thread seems to be easy to find via Google for this issue!

_Note: For security & best practices (narrow scope), I added this section to specifically only hanlde the hassio_ingress route, as opposed to just allowing all requests at my root location (wildcard for any request to my HomeAssistant) to be upgraded as websockets which was mentioned in the HomeAssistant forum._

Also, for posterity I've posted this as a blog to my wiki at github here...

[netweaver1970]

Same/similar issues here... my system is fully configured for external access and everything works well except for some HassIO plugins that attempt to establish web socket connections to /api/hassio_ingress.... which fail unless I access directly via IP address on my internal network...

I'm LetsEncrypt Docker (linuxserver/letsencrypt) on an Unraid server for all reverse-proxy routing, and had gotten it all working well, using the sample configuration for HomeAssistant provided in the docker. However, that sample was setup for a standard core HomeAssistant install and not HassIO, which I've finally figured out requires additional proxy configuration for websockets to the api/hassio_ingress route.

I finally got my add-ons working by adding the following location configuration to my Nginx proxy configuration (in addition to the other one for the default HomeAssistant websockets api route: api/websockets):

# Duplicate websocket configuration specifically for HassIO add-ons (e.g. /api/hassio_ingress)
# Details of how this config works can be read here:
# https://www.serverlab.ca/tutorials/linux/web-servers-linux/how-to-proxy-wss-websockets-with-nginx/
location /api/hassio_ingress {
        resolver 127.0.0.11 valid=30s;
        set $upstream_app 192.168.1.???;
        set $upstream_port 8123;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

        proxy_set_header Host $host;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
}

I hope this helps some others out since this thread seems to be easy to find via Google for this issue!

Thanks, it for sure helped me !