Mailu: ClamAV using all my disk space

[jbonnett92]

Hi,
ClamAV keep's using all my disk space, and there is no option in the .env to limit it.

It seems to be the temp files e.g. /home/core/mailu/filters/clamav-0033b19aa8a68887f57a9126f168010d.tmp

Anyone know how to limit the disk usage? Seen as though I have 1gb memory.

Thank you,
Jamie

[jlkas]

The same problem is also occuring on my system.

[kaiyou]

You could limit disk space in the compose configuration using ulimits I guess. The real issue here is that files should be cleaned up though. I'll investigate as I don't see the same issue.

[flayks]

Same issue, I have tons of .tmp files from Clamav filters

264.1 MiB [##########] /clamav-a017b6824d619633b1d46a9026b3877a.tmp
259.3 MiB [######### ] /clamav-dbcb20187f29dfad673bc40081783f00.tmp
256.2 MiB [######### ] /clamav-a28a124a19ffd73780504eeb193c3a8d.tmp
247.9 MiB [######### ] /clamav-ca1b7ee4796f5b1cf08b4b433d569106.tmp
244.6 MiB [######### ] /clamav-fe6e51fb2c79a1e7400a6d1b69777b4d.tmp
242.5 MiB [######### ] /clamav-8e7e2d1f022516ee43531bfe3f5e7050.tmp
235.0 MiB [######## ] /clamav-3ad3e0f7da4783d6d4253fdba4a700ce.tmp
229.7 MiB [######## ] /clamav-87ac01f595385864e6e47aa5fd8744b8.tmp
229.4 MiB [######## ] /clamav-76961f4640c915c928648f87ccb405c5.tmp
225.0 MiB [######## ] /clamav-c278bd97ec95053012998202f334373c.tmp
223.2 MiB [######## ] /clamav-83af7e9227387273843bd51b57eb41df.tmp
220.4 MiB [######## ] /clamav-5259ad996415d545a9c96cb28a02c79a.tmp
220.0 MiB [######## ] /clamav-d6d49de527715fcb4045f7449f6167ec.tmp
218.4 MiB [######## ] /clamav-c10101aecf10d05dcd501588a928a811.tmp
217.7 MiB [######## ] /clamav-efe7487fa72ed015d7b5cfd414478ae3.tmp
217.7 MiB [######## ] /clamav-020350a67abf62cc7a50584d5edf7bf7.tmp
216.5 MiB [######## ] /clamav-f030a414c0a78fa5ab42a04718807ab3.tmp
214.6 MiB [######## ] /clamav-a525fbdb1bb94c777cbd3d9e842df95f.tmp
211.8 MiB [######## ] /clamav-bbcd1b5d865b56b15620c6c191b86fad.tmp
207.7 MiB [####### ] /clamav-6a47d84feec6795e047fe1df10e8e9c3.tmp
206.6 MiB [####### ] /clamav-e2649eef345b7524e078834ac9ecfcb7.tmp
...

[muhlemmer]

Seeing the same. Strange thing is, on my system they are located under the /data directory. While the config says:
TemporaryDirectory /tmp

Can anyone confirm the location? I think this will break the ability of setting filesystem limits, as the files are written to a mount point outside the container & persistent.

Furthermore there is:

LeaveTemporaryFiles BOOL
Do not remove temporary files (for debug purpose).
Default: no

Which is not set in the mailu config and thus tmp files should be deleted automatically.

It seems that clamd is not really behaving as it should do.

[kaiyou]

That could be related to some other issue, for instance clamav getting oomkilled due to high memory usage while it still has some temporary files open.

[muhlemmer]

Is everyone else is seeing the pileup under /tmp or /data?

I checked the server and it still has 7 gb of unused ram with an uptime of 49 days. So I don't think out of memory occurred.

[jbonnett92]

@muhlemmer I don't think you should see much RAM used anyway. I don't know why you're expecting it too.

@flayks Looks like mine :(

@kaiyou Any update?

[muhlemmer]

@jbonnett92: I was replying to @kaiyou's suggestion of oom kills.

Further, I've over-read the line where @jbonnett92 mentioned the files are under /mailu/filters. This effectively gets mounted in the container at /data, so the symptoms are same as I'm seeing.

Troubleshooting the problem as we speak.. (might not finish today)

[muhlemmer]

So, my conclusions for the night:

My server is running mailu:1.5. The clamav image is based on alpine:edge. I've seen more strange bugs lately in images based on edge. There is a "file locking" bugfix in Alpine, which has not yet made it into the Docker images yet.

Because of the earlier found bugs, we've moved to alpine:3.8 for clamav on mailu:master. I've modified my docker-compose.yml to check if this bug is due to the same cause.

  antivirus:
    image: mailu/$ANTIVIRUS:master

After that, I ran docker-compose up -d to pull and run the new image.

Before this change, a new .tmp directory was created every 1 minute. After the change, I haven't seen a new one appear for almost 15 minutes now.

[root@sun mailu]# ls -lt /mailu/filter | head -n 15
total 391887
-rw-r--r-- 1 root root    797696 Oct 10 22:32 rspamd.rrd
-rw------- 1 root root       520 Oct 10 22:32 mirrors.dat
drwxr-xr-x 3 root root      4096 Oct 10 22:14 clamav-084e8cc824eac2445a9dd483430de98e.tmp
-rw-r--r-- 1 root root 115033600 Oct 10 22:14 safebrowsing.cld
drwxr-xr-x 3 root root      4096 Oct 10 22:13 clamav-2e6909f7b1a8c8ce4f9e560928d571e8.tmp
-rw-r--r-- 1 root root     47876 Oct 10 22:12 symbols.cache
-rw-r--r-- 1 root root       178 Oct 10 22:12 stats.ucl
drwxr-xr-x 3 root root      4096 Oct 10 22:12 clamav-77b82525c8dd78a5235d6c6d91fcb059.tmp
drwxr-xr-x 3 root root      4096 Oct 10 22:11 clamav-9c8015a2e57da40e77c7f5dce608c091.tmp
drwxr-xr-x 3 root root      4096 Oct 10 22:11 clamav-972b0b26a38e7797ec84c414e66c3533.tmp
drwxr-xr-x 3 root root      4096 Oct 10 22:09 clamav-a0aefb4d95e14e37e48f05b9a661299d.tmp
drwxr-xr-x 3 root root      4096 Oct 10 22:09 clamav-bf9200e9ff2261c0d462332a47684b8c.tmp
drwxr-xr-x 3 root root      4096 Oct 10 22:08 clamav-f00e1ce09602424f666c8e52dd61f16d.tmp
drwxr-xr-x 3 root root      4096 Oct 10 22:08 clamav-fe683f2b019ef9d834ed8a61d84f52a3.tmp

Can other people please check if this fixes the issue? If I don't see new tmp entries and it is confirmed by more users, I'll make a backport to 1.5.

[jbonnett92]

I actually removed the image back in May, @flayks if you could see and let me know please.
I will throw it back on my server if it is.

[muhlemmer]

I'm unable to give a feedback. I've started the process of deleting the *tmp entries (391887 of them, on a GLusterFS) and I found in the morning the rm operation is still running and clamav is throttling CPU through the roof. I've stopped clamav for now and wait until the rm operation is completed.

[muhlemmer]

Ok, I found out why this was never-ending. I accidentally let a test server running on the same GLusterFS, from another host. Killing the server finished the rm process. :hankey: So back to normal live and troubleshooting.

[muhlemmer]

Ok, good news.

The clamav-*.tmp files are being created by freshclam. This means it is really caused by the same cause as #625, where probably freshclam fails to do the update.

With running mailu/clamav:master I did the following to verify:

docker-compose.yml :

  antivirus:
    image: mailu/$ANTIVIRUS:master

For the following test, you need two terminals open.
On terminal 1:

docker-compose stop antivirus
rm -rf /mailu/filter/*.tmp
docker-compose pull
docker-compose run --rm  antivirus freshclam

On terminal 2 (execute just before freshclam and terminate after with ctrl+C):

while true; do
  ls /mailu/filter/*.tmp
  sleep 1
done

You should see that during the update process of freshclam, some *.tmp directories appear and disappear. In the end to the update procces, you should just see:

ls: cannot access '/mailu/filter/*.tmp': No such file or directory

And you can terminate the while loop. If freshclam tells you that the DB is up-to-date, you need to delete the *.cvd files in /mailu/filter for this test.

When everything seems okay, you can run docker-compose up -d to bring up clamav again.

Strong note: Since this error is caused by clamav not doing the updates properly, your virus databases are probably very outdated! (Security issue!). I'll work on the backport now.

[muhlemmer]

Dammit. I see now that I already moved it to Alpine:3.7 myself in august. And probably I never did the docker-compose pull on my production server, remaining it to run on Edge.

So, switched back to original docker-compose file (image: mailu/$ANTIVIRUS:$VERSION) to refer to 1.5. Ran docker-compose pull and verified above test. *.tmp folders are cleaned up after freshclam.

I think this issue can be closed. Please re-open if anyone finds dangling *.tmp folders after pulling the latest 1.5 images.

[exhuma]

I am just struggling with this as well. I noticed that my mail server no longer received any mails: Out of disk space.

After investigating I came across this issue and I see the exact same problem with clamav. I only recently installed mailu (1-2 months ago) and am on 1.6

Is the above still valid? I am currently struggling with a cold and my head is not working. I've temporarily fixed the issue by cleaning out old and unused docker images on the box. So I reclaimed some space and I should be able to catch a bit of sleep. But tomorrow I need to work on this.

Any guidance on what I should look for?

[ghost]

same prob

[jbunner]

Had the same problem, now resolved. Docker CE 18.09.7 on CentOS 7.6, Mailu 1.6.
Antivirus container continually restarted about every 2 min, max out CPU, filled up all available drive space with .tmp directories under mailu/filter

Setting up hourly cron job to rm -Rf /mailu/filter/clamav-*.tmp seemed to remove temp files, but antivirus container still continued to crash and restart.

For me the resolution was to down the containers, then remove all of the AV database files under mailu/filter *.cvd and *.cld

If you look under the antivirus start.py, it should pull the AV files if they don't exist on first run. I had previously removed the main.cvd file, but that didn't seem to fix it; removing all of the AV definitions and re-upping the containers seems to be stable now.

[exhuma]

Thanks for the tip. I will try this.

[exhuma]

@jbunner didn't help. Disk has filled up again 😢

[exhuma]

I have now resorted to a fairly drastic solution:

#!/bin/bash
if [[ "$(id -u)" != "0" ]]; then
        echo "Must be run as root!"
        exit 1
fi
cd /opt/mailu
docker-compose down
rm -rf filter/*.tmp
rm -rf filter/*.cvd
docker-compose up -d

I'm running this daily now. It means my server will have downtime, but - at least for me - that's OK. But this is still a really ugly solution.

[jbunner]

If the solution works in the meantime. My guess is bouncing the container is quick, within 10-20 seconds, so users would probably never notice. The only other thing I’d suggest trying is throwing another gig of memory towards clamd, it can consume a bit once signatures are all loaded.

Sent from my iPhone

On Aug 27, 2019, at 4:13 PM, Michel Albert [e-mail] wrote:

I have now resorted to a fairly drastic solution:

!/bin/bash

if [[ "$(id -u)" != "0" ]]; then
echo "Must be run as root!"
exit 1
fi
cd /opt/mailu
docker-compose down
rm -rf filter/.tmp
rm -rf filter/.cvd
docker-compose up -d
I'm running this daily now. It means my server will have downtime, but - at least for me - that's OK. But this is still a really ugly solution.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[kaiyou]

I am reopening this, as it will probably require some more work in the container start logic.

[muhlemmer]

At the time, I also had #625 open and gliderlabs/docker-alpine#447. It seemed to be related to a bug in musl (standard C lib in Alpine). Just adding this for reference, it might help to troubleshoot the nature of clam restarts for other users. Assuming your system does have enough memory to run this.

Another thing, in Alpine 3.10 ClamAV got upgraded from 0.100.3 to 0.101.4 8 days ago. I've checked and master has the newer version available. 1.7 is currently rebuilding and should have it available in the next hour.

1.6 and 1.5 are going to be stuck with with clamAV 0.100.3 for now.

[kaiyou]

Any recent feedback on this issue while using a recent clamav and musl?

[mborko]

still have the same problem with tons of tmp files. Currently with the 1.7 version

[schenkp]

Also have the issue with many tmp files in tmp folder. My server disc 20GB is full after half a day. using version 1.7

[AndrewSav]

Yep, same here server with 50GB of space fills up in a day.

[inspiraluna]

anybody a solution?

[AndrewSav]

yeah, read the thread for a work around

[muhlemmer]

Please see #1482. User feedback on the experimental fix is required.

Note that only feedback from users with sufficient RAM are considered. Full Mailu stack + ClamAV needs 2GB. Space for the OS needs to be considered. So typically minimal 3GB total RAM. Or 2GB with sufficient swap (will perform poorly, but prevents OOM kill).

[mborko]

I have now resorted to a fairly drastic solution:

#!/bin/bash
if [[ "$(id -u)" != "0" ]]; then
        echo "Must be run as root!"
        exit 1
fi
cd /opt/mailu
docker-compose down
rm -rf filter/*.tmp
rm -rf filter/*.cvd
docker-compose up -d

I'm running this daily now. It means my server will have downtime, but - at least for me - that's OK. But this is still a really ugly solution.

I agree, and a really nasty problem was for me only restarting the antispam and antivirus container - thought it would be less downtime, but with this action I opened the Pandora's Box, after some monitoring I realized it would be wise also to restart the front container after antispam and antivirus is up again.

So, now I have a cron.hourly script, which checks the disk usage and acts like you described, but only stopping the anti* containers.

#!/bin/bash
diskusage=$(df -h | grep sda | awk '{print $5;}' | cut -f1 -d'%')
max=85
if [ "$diskusage" -gt "$max" ]; then
    test -f /srv/mailu/docker-compose.yml || exit 0
    cd /srv/mailu/
    docker-compose stop antispam antivirus
    rm -rf /srv/mailu/filter/*.tmp
    rm -rf /srv/mailu/filter/*.cvd
    touch /srv/mailu/filter/`date +%Y%m%d-%H%M%S`.tmp
    docker-compose up -d
    docker-compose restart front
fi

[muhlemmer]

@mborko, have you tried pulling the images as suggested in #1482?

[muhlemmer]

Also, could you make your cron job to run free and safe the results to a file, just to see the memory conditions of the system and post them back here:

free -h > /tmp/free.txt
And perhaps also use something like:

docker stats --no-stream

[inspiraluna]

giving more RAM to my mailu server (8GB) solved the solution for me completely.

[te-online]

Even going from 2 to 4GB solved the issue for me.

[muhlemmer]

https://github.com/Mailu/Mailu/pull/1482#issuecomment-630996996

[TheDavidJohnson]

Bumping from 2GB to 3GB of RAM has apparently solved this issue for me. Thanks to everyone for your efforts on this!

[ay1man4]

Had the same problem, now resolved. Docker CE 18.09.7 on CentOS 7.6, Mailu 1.6.
Antivirus container continually restarted about every 2 min, max out CPU, filled up all available drive space with .tmp directories under mailu/filter

Setting up hourly cron job to rm -Rf /mailu/filter/clamav-*.tmp seemed to remove temp files, but antivirus container still continued to crash and restart.

For me the resolution was to down the containers, then remove all of the AV database files under mailu/filter *.cvd and *.cld

If you look under the antivirus start.py, it should pull the AV files if they don't exist on first run. I had previously removed the main.cvd file, but that didn't seem to fix it; removing all of the AV definitions and re-upping the containers seems to be stable now.

This solution worked for me.

[gannza]

I have now resorted to a fairly drastic solution:

#!/bin/bash
if [[ "$(id -u)" != "0" ]]; then
        echo "Must be run as root!"
        exit 1
fi
cd /opt/mailu
docker-compose down
rm -rf filter/*.tmp
rm -rf filter/*.cvd
docker-compose up -d

I'm running this daily now. It means my server will have downtime, but - at least for me - that's OK. But this is still a really ugly solution.

I agree, and a really nasty problem was for me only restarting the antispam and antivirus container - thought it would be less downtime, but with this action I opened the Pandora's Box, after some monitoring I realized it would be wise also to restart the front container after antispam and antivirus is up again.

So, now I have a cron.hourly script, which checks the disk usage and acts like you described, but only stopping the anti* containers.

#!/bin/bash
diskusage=$(df -h | grep sda | awk '{print $5;}' | cut -f1 -d'%')
max=85
if [ "$diskusage" -gt "$max" ]; then
  test -f /srv/mailu/docker-compose.yml || exit 0
  cd /srv/mailu/
  docker-compose stop antispam antivirus
  rm -rf /srv/mailu/filter/*.tmp
  rm -rf /srv/mailu/filter/*.cvd
  touch /srv/mailu/filter/`date +%Y%m%d-%H%M%S`.tmp
  docker-compose up -d
  docker-compose restart front
fi

Help me too

[Nebukadneza]

Hi There,

The Mailu-Project is currently in a bit of a bind! We are short on man-power, and we need to judge if it is possible for us to put in some work on this issue.

To help with that, we are currently trying to find out which issues are actively keeping users from using Mailu, which issues have someone who want to work on them — and which issues may be less important. These a less important ones could be discarded for the time being, until the project is in a more stable and regular state once again.

In order for us to better assess this, it would be helpful if you could put a reaction on this post (use the :smiley: icon to the top-right).

• 👍️ if you need this to be able to use Mailu. Ideally, you’d also be able to test this on your installation, and provide feedback …
• 🎉 if you find it a nice bonus, but no deal-breaker
• 🚀 if you want to work on it yourself!
  We want to keep this voting open for 2 weeks from now, so please help out!

[te-online]

As far as I can see there is only one thing that's within the scope of the Mailu project: Adjust the documentation to state that 4GB is the minimum requirement if you want to have ClamAV running. I'm assuming that adjustments to memory usage (which would be preferable) can only be made within the ClamAV project itself, right?

[jerkball]

Really appreciated here...

crashed my backup:

root@omv:/sharedfolders/Docker/mailu# du -hs filter/

5,7T filter/

[MohammedAl-Mahdawi]

Mailu filled my server disk space after using it for more than one month and forced me to migrate all the email accounts to Zoho at midnight!

I do not know if it was caused by ClamAV or not, but it was a horrible experience.

I like Mailu and wish to keep supporting this project and use it in the future, I will keep my eyes on this bug hope it will be fixed soon.

[dhasilva]

Having the same issue here. Mailu 1.7, 4GB RAM.
Took around 3 months to fill the disk.

du -h /mailu/filter | sort -rh | head -30
29G /mailu/filter
334M    /mailu/filter/clamav-cc142e1fc9f4f47372e68eb1c7eb0c7a.tmp
334M    /mailu/filter/clamav-9e124c8a84afa72e43594c36185647a2.tmp
334M    /mailu/filter/clamav-80fcb774b4bbcfa6c0ffe330351ddd7c.tmp
334M    /mailu/filter/clamav-732ba24c06e29418a77d53e53090233f.tmp
334M    /mailu/filter/clamav-3f42114ee9138edd0261d7042a484088.tmp
334M    /mailu/filter/clamav-1122823576a64d6592d69e21b70b06ac.tmp
333M    /mailu/filter/clamav-ffe36321045fa039663761214088d0f1.tmp
333M    /mailu/filter/clamav-f0bfe2f9f16b4985a9345ed161b0145d.tmp
333M    /mailu/filter/clamav-e79fedc689f7a3809c7ea07645c4f71c.tmp
333M    /mailu/filter/clamav-e253067a1d2192c7463edb4fad454760.tmp
333M    /mailu/filter/clamav-d309bde638a9aca3d48d9993fe41cdac.tmp
333M    /mailu/filter/clamav-ce07f9c411759a2437023d23b622c4d1.tmp
333M    /mailu/filter/clamav-b85487563540dca18a58c22edd4fe5ac.tmp
333M    /mailu/filter/clamav-b0f2fc2410c3c1ad202fc07d25726453.tmp
333M    /mailu/filter/clamav-a448c4b5ac5be28bc4c1de0dc585c937.tmp
333M    /mailu/filter/clamav-9e9166ad01aadf3d8c690294d1dfe122.tmp
333M    /mailu/filter/clamav-9a59a8399ed3b4a1c2b14fd847850183.tmp
333M    /mailu/filter/clamav-8500f2eaf02599c70185eb3d329d75bd.tmp
333M    /mailu/filter/clamav-7169501d7210c91cfc5d60d91ff0cd1b.tmp
333M    /mailu/filter/clamav-15fea2accc08c062054d62f8f2769b05.tmp
333M    /mailu/filter/clamav-08a49f02fc066ddad6a274e137f60f45.tmp
332M    /mailu/filter/clamav-fcd289307f485c5049013df16a7f1fbe.tmp
332M    /mailu/filter/clamav-bb99494d021e7008370efecc942ac428.tmp
332M    /mailu/filter/clamav-a940e892324b74351270e3e5e7ef2b40.tmp
332M    /mailu/filter/clamav-6a1f7bb8c892d73870e459e44d06db8b.tmp
332M    /mailu/filter/clamav-66573a207f8801ea29edd5899bf2aab3.tmp
332M    /mailu/filter/clamav-5b5029f03c1c4c199cafa6df4ae38d24.tmp
332M    /mailu/filter/clamav-4bc6df6ce40553069d3397d9be82a9de.tmp
332M    /mailu/filter/clamav-4a2208485357f2d5300b6bd4d474639a.tmp

[alexanderadam]

1. --max-space switch

Is the call to clamav using the --max-files and --max-space switches?

Because if I understand this correctly, --max-space is exactly meant to avoid this issue.

2. OverlayFS

Otherwise there also seem to be the corresponding Debian issue 917648 from 2018, which would mean that it could also be related to _freshclam_ and Docker's _OverlayFS_.
Vincas Dargis wrote for this case:

Try changing:

/usr/bin/freshclam {

Into:

/usr/bin/freshclam flags=(attach_disconnected) {

muhlemmer already thought that _freshclam_ could be the reason.

3. Cleanup script

Some folks are also using an explicit clean up script. Maybe it would make sense to additionally remove all temporary ClamAV files before the ClamAV process is started.

This is also what the mailcow folks are doing:

# Cleaning up garbage
echo "Cleaning up tmp files..."
rm -rf /var/lib/clamav/clamav-*.tmp

And this was also already suggested by exhuma.

Summarize

I don't know whether any of these things are already used but it doesn't seem to harm to implement all these three.
In case these are already implemented: well, it was worth a try. :wink:

PS: Here and here are others with the same issue.