Espurna: WebUI blank when accessing espurna over OpenVPN on 1.13.5 (WS sends incomplete data)

[ddf89]

Bug description
WebUI blank when accessing espurna over OpenVPN

Steps to reproduce
Access and login to webui over OpenVPN

Expected behavior
WebUI works same as when on local network

Screenshots
Over VPN:

Locally;

Tested on SONOFF POW-R2, T1 (1,2,3 gang), BASIC and S20

Accessing WEBUI no longer works for me on 1.13.5 when I'm connected via OpenVPN, this used to work properly on 1.13.3. Inspecting the WS I can see that over the VPN, the device does not send all the data frames that it sends when connected locally. I doubt this has to do anything with the VPN or network per se, since when connected locally, all my devices are still on another VLAN, so traffic is routed (via a pfsense router).

[ddf89]

Last data frame over vpn is the relayConfig object, whilst the last data frame locally is the one with all the configuration. This explains why over overvpn the UI can't load. Does anything come to mind that could cause this?

[ColinShorts]

Maybe the packet is fragmented and somehow lost? Possibly a misconfiguration in MTU size. You might be able to test by specifying packet size in ping, start at 1500 (too big for VPN normally, but the standard for Ethernet) but you might find it works around 1432.

Just a guess though 😀

[ddf89]

That thought crossed my mind already, will have to find some time to
test it out. The funny thing is that this was working perfectly on 1.13.3,
along with all the other stuff I have. Is it possible something new was
introduced in the latest version?

On Wed, Mar 6, 2019, 7:49 PM Colin Shorts [e-mail] wrote:

Maybe the packet is fragmented and somehow lost? Possibly a
misconfiguration in MTU size. You might be able to test by specifying
packet size in ping, start at 1500 (too big for VPN normally, but the
standard for Ethernet) but you might find it works around 1432.

Just a guess though 😀

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/xoseperez/espurna/issues/1610#issuecomment-470228146,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAsGU8hnGALl0OhsxTdZu_oKS0A0-pw_ks5vUA2hgaJpZM4bhh7x
.

[ColinShorts]

I think that it's too low level to be attributed directly to espurna directly, possibly network stack related, that is unless the packet now contains a lot more data than it used to.

I might be able to test this tomorrow, but I'd likely need to flash two devices and dust off my VPN server. I normally use a dynamic socks proxy over SSH, maybe you could try that as a workaround in the mean time?

[mcspr]

The difference is that old one sends several ws data frames instead of just one in 1.13.5
Technically, it must work exactly the same, because the underlying library will space out the data.

And, funnily enough, the first thing i tried is reverse ssh tunnel + wscat tool on the remote server (~150ms for good measure), which worked perfectly.

[ddf89]

The problem is indeed with mtu, it seems the large data frame is being
dropped because it exceeds the maximum mss negotiated in the TCP handshake.
This won't be reproducible with ssh tunnels, since ssh will most probably
break down the frame to fit in it's own packet size. On the other hand,
openvpn will refuse to fragment it at layer 3 by default.

The main question now for me is, why doesn't espurna (or the underlying
lib) split the tcp packet to fit the negotiated mss?

Hope this makes sense.

On Thu, Mar 7, 2019, 10:05 AM Max Prokhorov [e-mail]
wrote:

The difference is that old one sends several ws data frames instead of
just one in 1.13.5
Technically, it must work exactly the same, because the underlying library
will space out the data.

And, funnily enough, the first thing i tried is reverse ssh tunnel + wscat
tool on the remote server (~150ms for good measure), which worked perfectly.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/xoseperez/espurna/issues/1610#issuecomment-470443492,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAsGU0MY_gKuR1ZnlmSaMV3kZpeb0TpPks5vUNZcgaJpZM4bhh7x
.

[ddf89]

[ColinShorts]

I still think this problem is caused way deeper than espurna, i.e. by way of inclusion from:
#include <ESP8266WiFi.h> in justwifi.

There is an example of a user with problems using Hyperion and a large string of WS28nn LED's who runs into problems with fragmentation/dropped packets here

caveat emptor, I do use a lot of espurna devices, and on the odd occasion OpenVPN (just not together). I'm not a developer and tend to get a bit lost in the code at points. @xoseperez or @mcspr will know a lot more than me from the code/library side.

[ddf89]

I see your point. The thing is that the response for / gets split down properly according to the negotiated MSS. It's just the WS datagram that's not, I suspect this to be true for the whole WS conversation.

[mcspr]

Right. Both responses are handled by ESPAsyncWebServer:
HTTP: https://github.com/me-no-dev/ESPAsyncWebServer/blob/95dedf7a2df5a0d0ab01725baaacb4f982dedcb2/src/WebResponses.cpp#L263
WS: https://github.com/me-no-dev/ESPAsyncWebServer/blob/95dedf7a2df5a0d0ab01725baaacb4f982dedcb2/src/AsyncWebSocket.cpp#L54

Have almost the same results on my test openvpn setup, it fails even with 1.6kb of data :/
Debugging does not really makes sense, as the data is reported as sent OK. ACK never arrives from it's point of view and it times out the connection.
Using latest Core / lwip2 does "fix" this behaviour.

[ColinShorts]

It might be a moot point, but I take it you are both (also) using UDP for OpenVPN?

[mcspr]

Yes. But problem is still there with proto tcp configs.

[ReclusiveGeek]

The same thing happens if you use NAT port forwarding but not if you use an ssh tunnel. It would appear that the issue is not related to UDP or TCP as it happens with both.

[mcspr]

I think @icoma89 already had outlined the exact problem - MTU/MSS size matters.
For example, on Windows machine:

$ netsh interface ipv4 show subinterfaces

shows default MTU of 1500 - everything works

$ netsh interface ipv4 set subinterface "WLAN" mtu=1453

edited: mtu value from #1614, because mss below is 1413

Causes config ws frame to be dropped by the router. ref tcpdump:

> IP DESKTOP-I8S9V0N.lan.62839 > ESPURNA-35A259.lan.80: Flags [S], seq 3747294428, win 64998, options [mss 1413,nop,wscale 8,nop,nop,sackOK], length 0
> IP ESPURNA-35A259.lan.80 > DESKTOP-I8S9V0N.lan.62839: Flags [S.], seq 6513, ack 3747294429, win 5840, options [mss 1460], length 0
...
> IP truncated-ip - 190 bytes missing! ESPURNA-35A259.lan.80 > DESKTOP-I8S9V0N.lan.62839: Flags [P.], seq 530:2180, ack 227, win 5614, length 1650: HTTP

And the same, but properly fragmented, when using lwip2:

> IP ESPURNA-35A259.lan.80 > DESKTOP-I8S9V0N.lan.61152: Flags [P.], seq 530:1943, ack 227, win 5614, length 1413: HTTP
> IP DESKTOP-I8S9V0N.lan.61152 > ESPURNA-35A259.lan.80: Flags [.], ack 1943, win 64998, length 0
> IP ESPURNA-35A259.lan.80 > DESKTOP-I8S9V0N.lan.61152: Flags [P.], seq 1943:2208, ack 227, win 5614, length 265: HTTP
> IP DESKTOP-I8S9V0N.lan.61152 > ESPURNA-35A259.lan.80: Flags [.], ack 2208, win 64733, length 0

edit:
assuming that this configuration change is valid, because I can access a different non-esp device
and assuming that MTU mismatch can also happen somewhere on the network path to the device i.e. router sets this value (which would trigger this too?)

[mcspr]

The main question now for me is, why doesn't espurna (or the underlying
lib) split the tcp packet to fit the negotiated mss?

Related lwip bug:
http://savannah.nongnu.org/bugs/?46384
http://git.savannah.gnu.org/cgit/lwip.git/commit/src?id=8e8571da6a6771f9d2d82bbd0b5a6c27474ce0fc

Since we are still suffering with Core 2.3.0, lwip can be built from source:

$ cd ~/.platformio/packages/[email protected]/
$ # patch tools/sdk/lwip/src/core/tcp_out.c
$ make -C "tools/sdk/lwip/src" install TOOLS_PATH=~/.platformio/packages/toolchain-xtensa/bin/xtensa-lx106-elf-

Resulting library is tools/sdk/lib/liblwip_src.a. The pre-built one is at tools/sdk/lib/liblwip_gcc.a, so we can just replace the file. Or change LIBS variable of the scons builder (extra_script)

[mcspr]

@xoseperez two options about shipping lwip1 fix:

1. ide & pio option. pre-build lwip lib archive, put it somewhere in the repo (like code/lib). prepend directory path to the LIBPATH in extra_script - linker will pick our lwip variant before the framework's. that is for current releases
   using old framework with arduino ide - manually file replace of <framework-dir>/tools/sdk/lib/liblwip_{gcc,src}.a. otherwise, it might be better to advise using latest Core with lwip2, where this problem is fixed already (and many other things fixed by sdk...)
2. instead of building beforehand, integrate builder into the pio script. only problem is, it only works if the user has make installed. meaning, no Windows support. no problem for travis though.

[simonedimarzio]

Hi all,
Thanks @mcspr for debugging this issue and thanks @xoseperez for your work.
Due to this issue is not possible to get access into espurna web UI through an OpenVPN connection.

I'm a programmer but not a very expert C compiler, could you please provide some more info on how to include the fix into the project?
I'm using Windows, it would be great to have the compiled lib to link/substitute into the project.

Thanks a lot!

[mcspr]

@simonedimarzio Easiest way is to just build using latest Core versions, where lwip2 is the default.
But, you can use WSL and ignore the Windows "problem" :) Just install platformio there.
And BTW I may be wrong about incompatibility, never tested that.

I think #1723 is "good enough" solution. If user has all dependencies installed, this will automatically patch lwip1 in Core only when ESPURNA_PIO_PATCH_ISSUE_1610=something is in the environment. That includes releases here, travisN tests and nightly builder (https://github.com/mcspr/espurna-nightly-builder/releases)

[ddf89]

Thanks for this @mcspr. I can confirm issue is resolved with espurna-1.13.6-dev.nightly20190519+gitaac2e1e4-itead-sonoff-pow-r2.bin (Sonoff POW R2).

[simonedimarzio]

Thanks @mcspr, also for me the fix worked perfectly!

Steps:

1. Pulled the latest dev version from the repo
2. pio run -e <env> -t build-lwip
3. pio run -e <env>
4. Flash device