Ems-esp: (v2) Web UI sometimes becomes unresponsive and doesn't load, on an ESP8266

[proddy]

Intermittent issues with the Web again. Sometimes it fails to connect (only on an ESP8266) and requires several refreshes and in the worst case a restart. I've noticed this before when the Telnet is running but it also happens standalone.

Free heap is around 57% so its not memory related. I'll try and isolate into a reproducible test case starting with all the other services disabled (uart, mqtt, telnet, etc).

[MichaelDvP]

Sure it's not memory related? My observations:

• if not connected to ems the web loads fast
• firefox networkinspector says its hanging on the big kB scripts 2.7638.js and 0.3b6a.js .
• when trying to reload and wait the interface loads after a crah/reset before memory is allocated by ems-devices.
• with NTP enabled memory is a bit lower web only loads in a very short period after crash/reset.

Is it possible to split the big scripts to smaller packages?

[proddy]

thanks for that input. I'll do some profiling later and also add debugging to the asyncwebserver. To reduce the file sizes I can try and do some more tree-shaking on the react side. The NTP takes a lot of space for the ESP8266. With me it gets caught on different files:
``
GET http://10.10.10.140/js/0.e0d4.js net::ERR_CONTENT_DECODING_FAILED 200 (OK)
GET http://10.10.10.140/js/0.e0d4.js net::ERR_CONTENT_DECODING_FAILED 200 (OK)
​```
and sometimes on the favicon.ico which is a large file.

[MichaelDvP]

I don't have such a filename, maybe different npm/node versions?
The biggest file is 2.7638.js with compacted size of 159kB. I think the webserver trys to read the whole file in rambuffer before sending. I see a similar issue here

[proddy]

The names of the .js chunked files can change as it's based on the hash. I'm not sure our problem is the same as the one reported above as the ESP8266 never crashes/reboots. It's like it just fails to make the connection. @bbqkees reported this too and said if you wait a while (like 30mins) it loads again. So perhaps memory-related and waiting for the garbage collector to kick-in? There is maybe something we can do with the size of the web json payloads (which is always fixed to 1024) and look at playing with shrinktofit() to reduce the traffic. First thing though is to isolate the problem - EMS-ESP running on a Wemos D1 mini with nothing connected and powered by USB.

[bbqkees]

I did some quick tests yesterday with the web inspector and it loads/tries to load one or two js files and then it stops.
At this moment it loads completely and instantly so can't verify when it fails. The 30 min waiting time could have been 2 hours, not sure.
At first I thought the ESP would just crash but the uptime is more than 4 days now so that not the problem.

[MichaelDvP]

as the ESP8266 never crashes/reboots.

it reboots if you try to reload a couple of times (every 5-10 sec) while it hangs. Reset info is:
Fatal exception:28 flag:2 (Exception) epc1:0x4023cff5 epc2:0x00000000 epc3:0x00000000 excvaddr:0x0000000b depc:0x00000000
and afair exeption 28 is memory related.

something we can do with the size of the web json payloads

after the web is loaded the first time, changing the tabs is always responding fast. Loading the json payload is no problem.
Not closing the browser is a solution.

EMS-ESP running on a Wemos D1 mini with nothing connected and powered by USB

always loads fast, have tried that before.

EDIT: with a lot of reloads i have crashed this usb powered standalone with exception:29 flag:2 (Exception) epc1:0x402267c5

wemos only on usb have 55% free and heap 9.920 / 9.760 bytes (2% fragmentation),
on emsbus its 47% free, heap 8.456 / 8.264 bytes (3% fragmentation).

[proddy]

I ran some more tests and pretty sure its memory related. On an ESP8266 with no EMS devices attached free heap memory is 70% (13.5Kb). When I connect the boiler it drops down to 58% (11Kb). In both cases web is responsive. Now when I add my RC20 thermostat it drops down to 33% (6.2Kb) which is just not enough and the web hangs and sometimes causes the ESP to restart.

I also tried disabling MQTT and using only USB power and it made no difference. So that rules out the MQTT component hogging excessive memory or issues the WiFi Tx strength.

I will look into why the thermostat class eats so much heap.

[proddy]

Now when I add my RC20 thermostat it drops down to 33% (6.2Kb) which is just not enough and the web hangs and sometimes causes the ESP to restart.

The 2.0.0b12b build I just checked-in uses an optimized version of AsyncWebServer. It reduces the firmware size by 8%, the Flash RAM by 10% and when I tested I noticed the available heap went from 33% (as reported above) to 53%. So a significant improvement. It's still not fully table with all devices loaded but chances are high that the Web loads now. There is still some mem improvements to be made in the code, for example moving from DynamicJson to StaticJson for documents < 1kb.

[MichaelDvP]

Great, web loads fast on 8266, tried some 10 times without hanging. Ram is the same range as b11, but much better than the pervious b12.

[proddy]

Is it true the b12 from v2_cmd uses more heap memory than the b11 from the V2 branch? I didn't realize. The goal of v2_cmd was to reduce the memory consumption. I never benchmarked it so unsure.

[MichaelDvP]

Yes, iirc on my system (ems on, mqtt on, syslog on, ems with controller, boiler, mixer, thermostat, one hc, no telnet) is was/is:
(values from web/system-page)
~55 % with b11 (8,5k heap free)
~42 % with b12 (6,8k heap free)
~56% now b12, but free heap is 14k! (i first only look at the percentage).

Now with telnet open it is:
44%, 11k in main level
37%, 9k in system level
41%, 10k in boiler
40%, 9,8k in thermostat
With opened telnet web reload hang.

[MichaelDvP]

Here is a comparisation of versions, values always from webpage:
v2 from travis, v2_cmd compiled here, for v2dev i used the new Webserver and replaced thermostat and boiler commands to call the mqtt-function.
comparisation_heap.txt

[proddy]

interesting. I would have thought with v2_cmd the heap used would be less compared to your v2dev. I'll need to look at the cmd code again and see why it's not using the stack as memory as all I'm doing now is adding the command to a vector array and the command strings are stored in Flash memory.

Also maybe we should look also into the memory fragmentation as this doesn't recover and anything >50% is bad. https://cpp4arduino.com/2018/11/06/what-is-heap-fragmentation.html

I'm honestly getting tired of the ESP8266. The ESP32 is so much nicer to work with

[bbqkees]

Here with b11 the heap fragmentation is about 1 to 4% and jumps to 20% when Telnet is opened and closed.
Then after a few minutes of doing nothing the fragmentation returns to 1%.
Does not seem a real problem to me.

As long as the web interface is open, it keeps on working.
If you close it and reopen or you do a refresh it gets stuck when loading the first two js files.
It takes 500ms to load the second js file, maybe its too big?

On another reload:

10 seconds

And then a successful reload immediately after:

Or could it be that it doe snot load the js files in the right order?

[proddy]

@bbqkees its definitely the heap memory that is causing the errors. Chunking the web files into smaller partitions could help but right now I'm trying to optimize the code. If you run an ESP8266 without the bus attached (or just use tx_mode 0) it loads fine because the memory hog is in the EMS objects, particular the Thermostat. I have made some improvements in the v2_cmd branch and it seems a lot more responsive so you can give that a try.

@MichaelDvP the mqtt and console commands are registered by constructing an object with the string and callback function into a vector in the function Mqtt::add_command(). With the thermostat it will create 25 objects for each of the commands. The strings are stored in Flash and should be around 200 bytes (if you count up the bytes) except the ESP8266 goes and allocates 2000 bytes of Heap memory, so it's not very efficient. This is what I'm looking into now.

[proddy]

This should be fixed with #445

[proddy]

closing. will re-open if new issues are found.