Mailcow-dockerized: Enable maildir compression

[extremeshok]

Pull Request #1045 , was reject as there needs to be a discussion regarding compression.

discussion to follow---->

Currently the plugin is loaded, but actual compression is not enabled.

I suggested compression be enabled by default, which is transparent to the admin and users.

I also suggested lz4 algorithm be used, as the current dovecot version and docker libraries support this.

lz4 is also the fastest on any modern day cpu.

The result of the pull would save storage and decrease I/O.

I will also submit a custom written script to compress all existing non-compressed mails, if this becomes enabled by default.

[catcun]

I am all in for this as it will help me a lot. But if I understand, I only need to change dovecot.conf and restart the container. Also, I would need to compress existing messages.
That said, I see no problem with it being enabled by default as it may be useful to many admins, but being so easy to setup perhaps a variable in mailcow.conf would do (the way Let's Encrypt is being handled.)

[apintocr]

I also vote for compression by default.
However I believe there must be an option to disable it.

[nightah]

Think the optional route is the best way to go, a lot of people could be using ZFS storage backends with compression enabled on their dataset, no point with double compression.

[lavdnone]

lz4 default makes sense
@nightah It would be funny if zfs translators would not figure out that files are already compressed and would try to compress them again. This would be an issue which we should file with zfs guys. Couldn't find the quick answer but I don't think thats the case. Real question might be who would do it better. And if you consciously using zfs compression you would figure out to change default setting in mailcow.config to not do compression.

[nightah]

@lavdnone, I'm sure there's likely benchmarks out there which would give a rough idea.

The point I was trying to make was that whatever implementation is done, I believe there should be a switch of some sort to enable/disable (even if the default assumption is that it is enabled).
Administrators might have many cases outside of what I presented above as to why they may not want compression enabled give them the power and the option.

[catcun]

Perhaps we don't even need the option in mailcow.conf, just document the possibility and let every admin decide.

[mritzmann]

Can i restore my maildir with compression to another email software (or thunderbird with maildir) or is compression there a problem?

[lavdnone]

when restoring dovecot will auto-detect compressed files by first bites with no issues, not sure about thunderbird

[mritzmann]

At the moment I can download any email from /var/lib/docker/volumes/mailcowdockerized_vmail-vol-1/_data/ and open it with e. g. Thunderbird.

The e-mail can also be read easily with vim. 1:1 the pure email source code. If this is no longer possible with compression, it should be optional.

[andryyy]

At least there is https://github.com/funcodeio/lz4.vim for vim.

gz would be more universal (zcat, zgrep etc.) but slower...

[HorayNarea]

I may be a bit late to the party but in response to https://github.com/mailcow/mailcow-dockerized/issues/1046#issuecomment-366395922:
When ZFS dataset-compression is set to lz4 or lzjb it detects if the blocks are not compressible and does not try to "double-compress". And ~probably~ hopefully no one has /var/lib/docker on a dataset with gzip-compression

[HorayNarea]

Also: please do not avoid progress because "I want to open my mail _right on the mailserver_ with vim".

Yes, that kinda is a valid way of reading your mails, but I dare to say: the vast majority of mailcow users (especially end-users) don't do it that way and would rather have more mailspace¹

¹lz4 has about 2:1 compress-ratio on text… users love it when you tell them they get double the space for free :D

[extremeshok]

https://github.com/mailcow/mailcow-dockerized/issues/1007#issuecomment-368166523

[extremeshok]

Pull request re-submitted:
Enable maildir compression #1090

[apintocr]

Great it was accepted as default.

How can I safely encrypt the current Gmail directory for all accounts?

[apintocr]

Sorry. I meant "compress"… :)

[extremeshok]

Just compressed a bunch of domains with the options zlib_save=lz4 and zlib_save_level=9

*domain names have been removed

----- Domain: Before: 31G After: 28G ----- Domain: Before: 1.1G After: 964M ----- Domain: Before: 16G After: 14G ----- Domain: Before: 23G After: 20G ----- Domain: Before: 18G After: 16G ----- Domain: Before: 13G After: 12G ----- Domain: Before: 5.0G After: 3.5G ----- Domain: Before: 9.0G After: 7.6G ----- Domain: Before: 2.2G After: 1.7G ----- Domain: Before: 24M After: 17M ----- Domain: Before: 3.2G After: 2.9G ----- Domain: Before: 780M After: 614M ----- Domain: Before: 954M After: 820M ----- Domain: Before: 3.6G After: 3.1G ----- Domain: Before: 5.2G After: 4.7G ----- Domain: Before: 4.6G After: 4.2G ----- Domain: Before: 9.0G After: 7.8G ----- Domain: Before: 3.8G After: 3.5G ----- Domain: Before: 28G After: 24G ----- Domain: Before: 24G After: 20G ----- Domain: Before: 87G After: 77G ----- Domain: Before: 8.4G After: 7.3G

[lavdnone]

No we need SIS pushed through as default 👍 and sdbox

[extremeshok]

@apintocr I wrote a very advanced bash script to automatically compress every existing email account.

@lavdnone Im working on a patch to enable the user to choose the storage system. Currently im busy porting to mdbox + sis, as my users accounts are way too large for sdbox (+75gb accounts)

[apintocr]

@extremeshok can you share your script?
I checked your repositories and just found a very old one.

Thank you very much.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ciroiriarte]

Before: 5.0G After: 3.5G

I would actually expect greater savings since emails are just text files. Is there something special about your usage?

[ciroiriarte]

No we need SIS pushed through as default 👍 and sdbox

I'm looking for options to scale (looking after performance & cost), all the discussions I see are regarding what should be default/standard after running the installation.

actually, supported + optional + documented is more than enough :)