Mailcow-dockerized: Improve documentation to be more thorough on RAM requirements

[luxaritas]

We set up a Mailcow instance earlier this year, among much excitement, and everything was going swimmingly... until the first time it stopped responding. Rebooted, fine... then the next time it crashed I decided to look into why: it seems that running out of memory was to blame! Looking in the issues, this has come up many, many times - to list a few (with multiple people contributing to each): #3761, #3336, #3255, #3054, #2973, #2870, #861

The response to these are generally predictable:

• Use at least 4GB of RAM
• Use swap
• Disable ClamAV
• Disable Solr
• Decrease SOGo workers/mem limit
• Set memory limits via docker-compose

However, that still leaves me without a sufficient answer to the most important question: _What exactly do I need to do to ensure that my server doesn't go down?_ That is, I imagine all of these things should be in the docs:

• The docs say 4GB RAM + swap... How much swap? Or how much RAM if I want to stick with RAM?
• How much does any given configuration adjustment change the amount of RAM I need?
• How do I make any of the specified configuration changes, and what specifically is the tradeoff/limitation of changing them?

I understand that this could change based on amount of usage - but it would be great to have the data about what amount of usage correlates to what specific resource requirements, as much as is possible.

Additionally, while needing to increase the amount of RAM/swap for a given workload to operate properly is fine, the entire system freezing up if the memory usage of some process spikes unexpectedly is not. Is there anything that can be done, either by configuration or ideally by default, to prevent this scenario? I'd like the opportunity to see and respond to resource limitations/failures/slowdowns rather than getting surprised by my server disapearing and users not getting critical mail. I can set up alarms in AWS to tell me when the server stops responding to pings and even automatically reboot it, but it seems to me that I should not have to rely on that - and for a service that's supposed to "just work", having to set up monitoring to reboot it in case it stops responding seems counter to that goal.

With all that said - thank you for the wonderful software that makes this even possible for us :)

[andryyy]

Hi,

Yes, we will need to increase the defaults.

No, we cannot tell you how much you need. It depends on the amount of users and how mailcow is used. 100 IMAP users are fine with ~8 GB RAM, 100 EAS users are fine with 32 GB and 16 GB swap (just a blind guess).

We should start to recommend 6-8 GB. I tend to 8 GB and at least _some_ swap, like 1024 MiB.

You write it run oom quite often, why did you think about restarting it automatically but not adding RAM? :)

[luxaritas]

No, we cannot tell you how much you need. It depends on the amount of users and how mailcow is used. 100 IMAP users are fine with ~8 GB RAM, 100 EAS users are fine with 32 GB and 16 GB swap (just a blind guess).

Right, I definitely understand that - what I'm hoping for is a rough suggestion for how to calculate the total RAM requirement, based on number of users in certain situations, etc. I know it won't be perfect, but anything is better than "well if it starts crashing you need more" :)

You write it run oom quite often, why did you think about restarting it automatically but not adding RAM? :)

Oh, don't get me wrong, I am considering that! I just have no idea how to figure out much I should be planning for (your answer suggests a total of 9GB is your preference, which means "4GB + swap" would mean 5GB of swap - which is not specified anywhere). Additionally, what if I get an unanticipated spike for some reason that normally it should handle? The server _will_ crash and won't come back until it gets restarted, so the only way to _ensure_ availability is by having it automatically restart (or, wake me up in the middle of the night when an alarm goes off that it's no longer able to be reached so that I can do it myself...)

[andryyy]

4 GB + 5 GB swap is not the same as 8 GB + 1 GB swap, keep that in mind.

[andryyy]

Fixed in https://github.com/mailcow/mailcow-dockerized-docs/blob/master/docs/prerequisite-system.md

[luxaritas]

This is super helpful - thank you so much!!

[gsloop]

So, can we kind of get some general figures to work with?

If 6G will run with a couple of dozen IMAP sessions + 2-3 EAS, and then
16G are needed for 15EAS and 50 IMAP....

Lets break that down.
So, it looks from the examples that you're giving about 500MB per active EAS user.

Here's how I get there;
16G for 15 EAS users and 50 IMAP. (That's 10G free above the base "personal" minimum.)
15*500M each = 7.5G and then 2.5G for the 50 IMAP connections. (It might actually be more than 500MB each - because I don't think 50 IMAP connections should consume 2.5G - but what do I know.)

I totally get that there's no comprehensive formula that solves every situation - or that it's pretty hard to calculate with a lot of different details. But would it be possible to get some kind of calc like above.
X MB per active EAS connection
Y MB per active IMAP/SMTP connection
Z MB per Sogo web-connection user.

(Probably we ought to assume that "active" IMAP/EAS connections are essentially all the devices configured to do EAS, or IMAP. Not just how many users are configured for EAS/IMAP, etc. Some "weird" site might not follow that path, but I suspect most will.)

That (some formula like that) would give us a reasonably accurate place to start when sizing a server.

---

This impacts me too, because I'm trying to spec a server that's going to meet the needs for my own setup.

I also have an in-house server that's getting imminent (Postfix/Dovecot/SA - on Ubuntu 16.04). I have to decide, should we move to MC and get a support contract, and if so, what hardware do we need to spec and what hot-swap hardware do I need on-hand, etc. (This one is "big" - a couple hundred users.)

I have a couple of other use-cases and knowing the hosting costs allows me to make some decisions.

I have a test setup right now and it's mostly reliable. But I'd hate to put more people on it and find it's falling over and all the calculations I'd made on what the hardware needed were all wrong.

[andryyy]

Well, SQL grows, Redis grows. Both potentially faster with more users. Reloading Clam on a busy server can also feel more heavy on the system.

Many users may have multiple devices. That sums up fast. I have systems with 500 users which are totally low on RAM.

To clarify: I have done hours long considerations with companies about sizing, then never heard back until a year later when they run into a problem and basically asked "do I really have to pay for support now?". That's ... frustrating.

If you plan for a bigger setup, I don't want to do sizings and hardware requirements on GitHub at 5 o'clock in the morning. :) Please join the community channels, you will be helped with sizing eventually. There are many people running different workloads.

If you decided for a sizing and send me the details of your decision, I can have a look at it and give you my two cents.

I can add more examples of machines I host in the docs.

[gsloop]

I purchased hosted mailcow, just today. This large client, we'll self-host internally, and we'll buy a support contract. So, I'm not above sending you money.

But I've got some cases where I'd like to host for a small company, or a few small companies. But in that case, it really doesn't make sense to buy a contract. [There's not a ton of money in it for anyone.] Perhaps I can make those work hosted there - but there's stuff I might like to do on my own server that's not likely to work on your hosted setup... I dunno - I'm still trying to figure out the options.

And in those cases, having some decent idea where to start for RAM is pretty important. So, the more you can do to break it down so one gets a reasonable feel the better. I think giving some base RAM and then per EAS/IMAP/SoGoWeb user add amount is likely to be the most helpful.