Goaccess: On-Disk Persistance Storage compression

[ms217]

Hi!

Afaik the previous Tokyo-Cabinet db feature had support for bzip2 and gz compression.
Is there any plan to re-implement such a feature? It would be definitely useful and diskspace-wise economical to have that option available again. Or any other plans to optimize the size of the database files - except the keep-lastdays option?!

[allinurl]

I think there's a real chance that we can look into adding this. I don't think it would be that hard to compress the data upon closing goaccess. Thanks for the suggestion! I'll keep it open so we can look into it.

[ms217]

Thank your for looking into it!
I'm not sure how much work it would be to implement but it would be nice to have lbzip2 and/or zstd and/or pigz compression available. zstd seems to be the most promising compression algorithm in terms of speed and compression.

[rtista]

I agree with this feature, it would be most helpful in keeping servers with low disk requirements! ZSTD seems to be the best promising indeed, I have used it in a recent project and the results are astonishing, even when compared to brotli at any level of compression!

However not all systems have zstd libraries by default, maybe default should be GZIP or no compression at all, and the desired compression library could then be provided through configuration! ;)

[ms217]

@rtista
zstd seems to be widely available on the most common linux distributions. Have a look: https://pkgs.org/download/zstd
Same goes for lbzip2 (bzip2 multithreaded) and pigz (gzip multithreaded).

[rtista]

@rtista
zstd seems to be widely available on the most common linux distributions. Have a look: https://pkgs.org/download/zstd
Same goes for lbzip2 (bzip2 multithreaded) and pigz (gzip multithreaded).

@ms217 yes, I know it is widely available, what I said is that they are not installed by default in a freshly installed server distribution such as CentOS Minimal or Ubuntu Server. And so GoAccess should default to one that either is installed by default or to no compression at all, leaving that for the user to configure! ;)

[ms217]

@ms217 yes, I know it is widely available, what I said is that they are not installed by default in a freshly installed server distribution such as CentOS Minimal or Ubuntu Server. And so GoAccess should default to one that either is installed by default or to no compression at all, leaving that for the user to configure! ;)

Sorry, I should have been more clear. Usually one doesn't install a program from source (on binary distributions like EL/Debian etc) but rather from a package. This would imply that all dependencies will be resolved by the building process of the package itself.
Also I don't think that GoAccess will utilise the compression types by using an external command which would mean that they are "built-in / linked" at build time. So, if in the build environment was none of these compression types (provided by -devel packages) available then it can only default to a default like no compression which it should anyway. Actually we are talking about the same "like in the same fashion" as it was implemented before by the Tokyo Cabinet storage.

[rtista]

@ms217 Sorry, I see what you mean now! I totally agree! ;)