Espeasy: Move internal webserver to ESPAsyncWebServer

[Fusseldieb]

Hi,
I've cloned the repository and took a look at the webserver that ESPEasy uses and almost fell to the ground in shock.
The HTML build itself slowly up, passing through zillions of functions, each appending a bit of HTML.

This has obvious disadvantages:

• It's _much_ harder to maintain, especially for who doesn't own and knows the code inside out. It's easy to loose your mind there.
• The page is almost fixed in there. No customization possible. If someone wants another looking interface, he would need to dig through tons of code snippets, replace all of them and/or change functions, which finally breaks stuff and/or introduces bugs (That can be a pain in the a.. and is extremely frustrating)
• Uses more processing power. If the ESP needs to assemble and replace the HTML on the fly, performance can be affected heavily, leaving to timeouts sometimes and even skipping some pulses from some sensor, especially if it sends a lot of pulses/data.
• No syntax highlighting on the HTML,CSS or JS part, which also means no way to tell if some tag isn't closed or similar.

My idea was to move to the ESPAsyncWebServer. It has a huge documentation, support and supports some nice functions, like serving from the SPIFFS "file system" with 3 lines of code. The advantages are obvious:

• Serving directly from the SPIFFS could serve normal (.html) files, which then includes some JS Framework (Vue?) which loads important sensor data and stuff in the background (through a small JSON API). This means that someone could write normal HTML,CSS or JS code, completely separated from the C++. This would make proper syntax hightlighting possible and developing much easier.
• Since "web-code" is separated, upgrading the entire look & feel could be easily done within hours of coding, instead of days, making everything a breeze, allowing us to use all nice functions that the editor can handle for that specific file type. New frontend eye-candys could be implemented within minutes, without breaking anything on the backend - guaranteed.
• Since the files will be served as they are without any processing whatsoever, the ESP will need a lot less processing power to do so. It just send out the entire file and done. No need to stop to fetch some data. This will be delivered separately through some small JSON API.
• 3 lines of code to serve all files. There's not much that can be made wrong.

I've made some tests with the library mentioned above and it loads a lot faster than the current implementation. I've managed to load a entire webapp onto the ESP, making it display the Vue Example page, with the Vue Router working sucessfully and _FAST_. Delivering 500KB of data _and images_ took a blink of an eye. _Everthing with less than 10 lines of code._

I am really interested in upgrading the webserver in this project, but I don't know if there's approval to do so, because it would be sad to invest days/months into it and then be rejected. So, I'm giving the idea here and asking for approval. Or, if somebody else wants to do that, that would be nice too.

[Grovkillen]

I'm currently updating the GUI which will be "completely" separated from the core loop. We will then do all updates using JSON. So please wait for us to first release a stable and then we'll go ahead with implement this new modular approach regarding GUI.

[Fusseldieb]

We will then do all updates using JSON.

That's nice. What do you think about doing this with this library also?
It's asynchronous and it just works out of the box.

The code below is everything you need to serve something with a fixed path:

server.on("/something", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send(200, "text/plain", "... Something here?");
  });

Just to show you that's not hard to implement. I don't know if you know this library yet, but I'm impressed on how well it works.

I believe it saves you a lot of hassle, instead of doing something on your own. And if you want that I implement a GUI later, it will be a lot easier if we use the same approach.

JSON is also possible out of the box: https://github.com/me-no-dev/ESPAsyncWebServer#arduinojson-basic-response

[Grovkillen]

We will look into that thanks!

[TD-er]

We are indeed working towards JSON + static files in JavaScript to make the rendering a job for the browser, not the ESP.
But for now, this was already a huge update in speed compared to what was before.
And I moved a lot of little HTML parts to functions to save on flash strings. Flash strings (using the F() macro) are not checked for duplication.

I looked at the AsyncWebServer a while ago and by then the reported issues were such that I didn't dare to move over there. Also using chunked transfers, we now have tweaked the current webserver to impressive speeds (almost all pages are rendered in under 300 msec)

Changing to AsyncWebServer is such a big step, we will not do it before the next release.
Also I have to look at how it will handle Json responses. Most JSON libraries use quite a lot of memory and our current implementation does not. Indeed changing is a lot of work, but for now I consider it to be a proof of concept until we move to the new interface, on which @Grovkillen is working very hard.

As soon as we make the move to JSON + static files only, we will have a look at the Async webserver again.

[Fusseldieb]

Great news @TD-er !
I'll wait anxiously until the next release.

I moved a lot of little HTML parts to functions to save on flash strings. Flash strings

This is nice and probably much welcomed on little microcontrollers like bare Arduinos which have very limited memory, but the ESP has SPIFFS, which makes HTML snippets in F()'s obsolete, in my opinion.

the reported issues were such that I didn't dare to move over

Hmm... Testing out this library I found nothing that doesn't work correctly. Even reloading pages like crazy didn't freeze anything, so it should be good. I really stress tested it.

almost all pages are rendered in under 300 msec

I _think_ it could beat that, especially because there's no rendering involved. I'll test more later and give feedback.

Changing to AsyncWebServer is such a big step, we will not do it before the next release.

I think moving between libraries will _later_ become easier when parts are already separated. Now it's hard because the current implementation uses _completely_ different approach (static HTML strings encapsulated in everything).

Most JSON libraries use quite a lot of memory

Depending on how the strings are assembled and if the size is dynamic or fixed. More testing would be necessary - count me in, I'll happily help.

but for now I consider it to be a proof of concept until we move to the new interface

As soon as we make the move to JSON + static files only, we will have a look at the Async webserver again.

That's fair. One thing at a time :)

[TD-er]

I think it could beat that, especially because there's no rendering involved. I'll test more later and give feedback.

If you use the inspect tool in the browser, you will see JSON messages are served in 60 - 200 msec (depending on the number of plugins active) on average and most HTML pages will take about 300 msec.
If you run the meminfodetail command and then open the sysinfo page, you will see longer load times (~800 msec), but that page is about 40k in size ;)

[Fusseldieb]

I think it could beat that, especially because there's no rendering involved. I'll test more later and give feedback.

If you use the inspect tool in the browser, you will see JSON messages are served in 60 - 200 msec (depending on the number of plugins active) on average and most HTML pages will take about 300 msec.
If you run the meminfodetail command and then open the sysinfo page, you will see longer load times (~800 msec), but that page is about 40k in size ;)

Thanks for those details. I think I might have a older version, because values are all static. I'll definitively update now and see again.

_(EDIT: Note to self: Maybe a client-based (Frontend) version check, and if a new is detected, it'll prompt to download it.)_

EDIT 2: I just saw that Websockets are possible with the ESP. Maybe it's better for live-updates than GET'ting JSONs?

_Needless to say that this project is absolutely amazing and I'm amazed to see where this is going._

[TD-er]

The library used states:

ESPAsyncTCP libary is required.

Or is it only when "Async TCP mode" is enabled?

Also I don't know how many sockets can be kept open.
Still it looks like a nice upgrade to make the page even more interactive.

[Fusseldieb]

@TD-er

From what I've read, it runs in "normal" TCP mode, but asynchronous mode can be activated by using the ESPAsyncTCP library (what, from what I saw, is a good measure).

Also I don't know how many sockets can be kept open.

It seems that it hasn't a limit (other than the ESP's processing limit - only discovered by testing I suppose), but I believe there's no need for more than 3-5.

I think the next big step in the future would be upgrading from JSON to WebSockets, as this would drastically lower the latency (connection is kept open until the page is closed). More live updates - yay.

[Grovkillen]

Have you looked at the web log viewer and the device page. Those are updated live and we're pretty happy with it :)

[Fusseldieb]

Have you looked at the web log viewer and the device page. Those are updated live and we're pretty happy with it :)

Yeah, these are neat. It looks great and it works, but what does it use? If JSON, that's nice, too, but I pointed at WebSocket because it was _made_ for such stuff. It potentially has a better performance.

[Grovkillen]

We will look at it too... since it's modular it will be easier to make that move later (modular in the new approach, not current approach). Thanks for the feed back.

[TD-er]

The log viewer sends JSON, which are gathered by the browser using GET.
It is true the log viewer would really benefit from websockets, since you then don't have to keep a log buffer anymore. Just send it and be done with it.
Also the device page could show updates immediately, instead of polling.

So I definitely see the advantages of websockets, and we will for sure have a look at them... after the release. :)

[v-a-d-e-r]

Something OT @TD-er and @psy0rz: I can not see the commits since last release in GH. mega-20181120 (77 commits to mega since this release ) was the last one that shows that... :-/ Did they change something?

[TD-er]

Where are you looking?
What application?

[v-a-d-e-r]

In this list:
https://github.com/letscontrolit/ESPEasy/releases

[TD-er]

@v-a-d-e-r Maybe you can open a new issue about it, to keep things a bit more clean.
Or else I won't know where to find stuff again.
It is already next to impossible to keep things organized with all these issues and topics.

[v-a-d-e-r]

The problem is, I can not say if it is GitHub related or an ESPeasy publishing issue!? Maybe the problem is since GH belongs to MS. Then we have the guilty one.... LOL

[TD-er]

Try opening it with another browser, to make sure it is no cache issue.

[v-a-d-e-r]

Same issue with Firefox and Opera. But psy0rz is publishing the releases. Maybe he knows what's going on?

[Grovkillen]

At this time @Psy0rz is a bot. It's a script running on his server. :P

[v-a-d-e-r]

Haha, that's what I already thought. ;-) Ok, to make my problem a bit clearer:
psy0rz released this Nov 20, 2018 · 77 commits to mega since this release => ok
psy0rz released this Nov 21, 2018 => not ok, the number of commits is missing since Nov 20, 2018
That is important to know for self-compilers like me. So I can decide what I merge to my own version and what not...

[TD-er]

Just one question left, why not start a new issue about it?
Now this issue has more replies about something off-topic and it is an issue which we will need to address in the future and hopefully this pushing of commits is not.

I really like to get a little bit of grip on the information here, since we are having over 500 open issues and Github is really lacking good tools to get some grip on the information stored in issues.
Also splitting of comments into a new issue is not possible.

so please, please, please, pretty please, try to keep things a little bit on-topic!

[v-a-d-e-r]

Ok, I opened a new issue now. But I'm sure there are a lot of issus that are already fixed and just not closed.