Domoticz: Realtime compression (gzip) of html/js files during runtime/packaging

[gizmocuz]

For speed (specially for arm systems) we probably need to continue serving compressed js files

JS libraries that are not part of Domoticz but from external sources we "could" already compress to GZip (.gz) like it is already so 90%

For libraries from Domoticz, we should have them uncompressed in our repo, so other developer can quickly work with them.

js/domoticz.js
js/domoticzblocks.js
js/domoticzblocks_messages_en.js
js/domoticzdevices.js

Also all controllers in the app/ folder

Question is, do we need to compress them ? Because are they not cached by the browser already ?
If not, we're going to need a PR that decompressed them and removes the compressed once.

If yes, we can think of compressing them during runtime or when packaging

What i think is a good idea, is to compress them in runtime.

During startup of domoticz, remove all .gz files. This means, no libraries should currently be compressed.
(we should however use the minimized versions where possible)
Then if we see that a javascript file is being requested, check if we have a .gz, if not, compress the library
serve the compressed .gz

That is once solution, and probably the easiest one.

Another is to use a (md5) hash, and after startup, and on first request, check the source and the uncompressed .gz, if there is a difference (or the compressed does not exist) (re) create the compressed and serve this

Any thoughts/ideas would be welcome

[MAkerboom]

Why on runtime? It makes more sense to me to make this part of packaging

[gizmocuz]

because a lot of (open source) users will compile/run domoticz from source
They do not use packaging so in runtime this will still work

But it was just a suggestion ... ;) Counter question, why not at runtime ?

[MAkerboom]

I'm used to nodejs and react. There you compile a production build. Which creates a main.js and .css with all the needed code for the solution. So I guess making it part of the make-script seems logical

[gizmocuz]

@MAkerboom , but those files you mentioned are not compressed ?

[MAkerboom]

It's not default, but I guess if I look into it there is someone who added a module to do .gz while creating a production build for sure.

But the point I try to make: In development we retain a logical overview and readable source (uncompressed, separate and un-minified files). During the build, we create the binary, minify and compress for optimal performance. (also the appcache is now a static file and could be created during build, in my nodejs/react example a service-worker for caching is created automatically during production build)

[emontnemery]

@gizmocuz Are you working on this? Otherwise I'll have a go at build time compression next week.

[gizmocuz]

@emontnemery , no, please go ahead, just replied to you here on anther issue with this exact idea ;) (like python pyc files)

[emontnemery]

@gizmocuz Visual Studio 2017 has support for cmake: https://blogs.msdn.microsoft.com/vcblog/2016/10/05/cmake-support-in-visual-studio/
Would it be acceptable to move the windows build from a vcxproj to using cmake?

[gizmocuz]

@emontnemery , please no ;) but VS also has build events in the project settings
But please consider runtime compression, we already have the compressor.
Like python pyc files

[emontnemery]

@gizmocuz OK, I'll stick with vcxproj for Windows :)

I'd prefer build-time compression of the web UI though..
Then as a logical next step, we can easily introduce fingerprinting of the assets and get rid of the app-cache:

There's not to much gained by letting domoticz itself do the compression, since you will anyway have to do some manual step after any modification (restart domoticz) vs calling make.
Perhaps in some "developer mode", domoticz can build the web UI automatically during startup though?

[gizmocuz]

We need the appcache
But what I meant is at first time page request, check if the gz needs to be recreated

[emontnemery]

Why is the appcache needed?

I could understand it if domoticz WebUI could be used when domoticz server is down, but the WebUI is just redirected to the offline-page when the domoticz server is down..

[gizmocuz]

This is used for at least the iOS home screen offline application
And I think for Android as well
I might build it dynamic later

[emontnemery]

OK, clear.

[emontnemery]

Cache busting by fingerprinting the assets would work well also with the app cache.
There are many tools which can do that, but with some dependencies.
Would it be acceptable to use a Python script for that, it would mean Python would be needed to build domoticz (Windows, Linux, MacOS X)?

[gizmocuz]

i vote for a realtime compression like a normal web server does
But caching the compressed files to be served again if there was no change in it's original uncompressed file
We also do not need any scripts during build time

[emontnemery]

@gizmocuz It is common practice to "build" a web app, just as you build a native application. Compression of the javascript can be considered a build step.
Of course no need for scripts to just compress some javascript files, and sure, it could be done on the fly instead of during build.

However as a next step, to reduce the very large number of user issues due to the browser cache / app cache not being invalidated as expected, we really need a reliable cache busting mechanism.

A simple and widely used approach is to "fingerprint" the assets with the hash of the asset.
For example, instead of requesting domoticz.js, browser should request domoticz-82e33bb5.js where 82e33bb5 is the hash of domoticz.js. After a change of domoticz.js, the hash will be different.
In this way, it is much less likely to have bugs/issues due to mismatch between html, js and css.
This can be done by physically renaming all the files in the file system, or by letting the webserver translate the request for domoticz-82e33bb5.js to domoticz.js.

There are a lot of tools available for such fingerprinting, so the question was more related to that.

[gizmocuz]

@emontnemery , thats correct, and compression is always done within the webserver (in this case domoticz)
As extra, do not re-compress an unchanged file, and serve the previous compressed file

[gordonb3]

I have a feeling there is some confusion here.

Compression is used to limit the needed bandwidth which should help getting a page quicker to the user on slow or congested networks. As this comes at the cost of needing more CPU power for performing the compression this may in fact be counterproductive on slow machines, especially when these do not have a bandwidth issue.

In effect, I don't think you should want to do compression within a LAN environment at all. And it should be optional when accessing from an external address.