Docker-mailserver: incoming spam is sometimes bounced, please add parameter

[hanscees]

Context

Working on an email-host with local mailboxes for 4 domains facing the internet.

When testing if spammassasin works I sended some test emails coming form the internet.
Since I had defined a spam-header (####spam####) I was expecting that spam would be passed to endusers and tagged.

However, some spam was bounced to the internet.

logging:

Feb 12 20:55:30 mail amavis[1043]: (01043-01) Blocked SPAM {BouncedInternal,Quarantined}, LOCAL [172.19.3.101]:57278 <[email protected]> -> <[email protected]>, quarantine: L/spam-LMHsfmxldtMS.gz, Queue-ID: 35EE580A20, mail_id: LMHsfmxldtMS, Hits: 1005.482, size: 544, 999 ms
Feb 12 20:55:30 mail postfix/smtp[13246]: 35EE580A20: to=<[email protected]>, relay=127.0.0.1[127.0.0.1]:10024, delay=28, delays=27/0/0.01/1, dsn=2.5.0, status=sent (250 2.5.0 Ok, id=01043-01, BOUNCE)
Feb 12 20:55:30 mail postfix/qmgr[1975]: 35EE580A20: removed

Feb 12 20:55:30 mail postfix/smtp[13254]: Trusted TLS connection established to vlees-nl.mail.protection.outlook.com[104.47.2.36]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Feb 12 20:55:31 mail postfix/smtp[13254]: 88DC380B30: to=<[email protected]>, relay=vlees-nl.mail.protection.outlook.com[104.47.2.36]:25, delay=0.84, delays=0/0.01/0.18/0.65, dsn=2.6.0, status=sent (250 2.6.0 <[email protected]> [InternalId=11055245821936, Hostname=AM6PR06MB5845.eurprd06.prod.outlook.com] 10723 bytes in 0.137, 75.892 KB/sec Queued mail for delivery)

Users in my setup want to get all email (except from rbl-tagged hosts etc), since non of it can be missed. So rejecting mail on the basis of mail content is not desirable.

Secondly sending bounces to probably forged recipient email addresses is not clever.

Expected Behavior

spam should be delivered to users and tagged.

Actual Behavior

spam is quarantained and user never he he missed something. And a bounce to internet is sent.

Possible Fix

Some parameter to adjust this in /etc/amavis/conf.d/20-debian_defaults:

$final_banned_destiny = D_BOUNCE; # D_REJECT when front-end MTA ##changed to reject
$final_spam_destiny = D_BOUNCE;

I want banned to reject and spam to pass

Steps to Reproduce

send this email from internet with an acceptable recipient and see what happens (the long string tips off spamassasin / amavis):

telnet 172.19.3.172 smtp
ehlo mail.vlees.nl
mail from:<[email protected]>
rcpt to:<[email protected]>
data
please sell a loan now
buy naked women online

XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

[erik-wramner]

I think there is a configuration file where you can override this already. At least in the new debian buster branch. I'll see if I can find it. Worst case you can always use config/user-patches.sh to patch the existing files at startup with sed. See that startup script for examples.

[hanscees]

could you please add the user-patches.sh to the faq? It took me quite a while to find that solution.

and perhaps some examples in the file as it is on github if you want to be extra special fantastic toward users :)

[erik-wramner]

Feel free to write something in the FAQ and/or make a PR, this is a community effort after all.

[erik-wramner]

After a quick check I don't think the possibility to override was in this branch. There is some support in the Buster branch and perhaps it would make sense to add an amavis override to the config folder that is copied to the correct location at startup. As of now the user-patches.sh solution is the best option.

[hanscees]

Feel free to write something in the FAQ and/or make a PR, this is a community effort after all.

I didnt know that was a possibility.
I will, but better someone checks it for error

[casperklein]

I did something similar, if you are still looking for an example.

[hanscees]

I did something similar, if you are still looking for an example.

Thanks!
I have added an example of this in the faq as well. So 2 items added to the faq.
1- example howto add custom certificates to the server
2- example of user script that overrides default config settings.

[Nebucatnetzer]

Passing spam should be the default IMO.
I just noticed the bouncing because I missed a registration email.
It might be mentioned somewhere but I probably missed that and since some spam came through I thought that would be the case for all mails.

[Nebucatnetzer]

BTW, wouldn't it be better to add the configuration to a amavis.cf file and place it in the config directory?
In the start-mailserver.sh script is a place:
https://github.com/tomav/docker-mailserver/blob/c075a0e43685437969a3b6dedcbd5411968be8dc/target/start-mailserver.sh#L1432-L1434
where it copies the amavis.cf file to /etc/amavis/conf.d/50-user

The user-patches.sh file feels a bit hacky and is maybe something I would use for a config file which doesn't have a way to override it but since the 50-user file should override the parameters from the main file this seems to be cleaner solution.

[casperklein]

The user-patches.sh file feels a bit hacky and is maybe something I would use for a config file which doesn't have a way to override it but since the 50-user ...

You are right. I overlooked that.

For completeness: There are DNS blacklists enabled by default, which could prevent (spam) mail delivery.
I use also config/postfix-main.cf:

postscreen_dnsbl_sites =

# DNS blacklists removed
smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination, check_policy_service unix:private/policyd-spf, reject_unauth_pipelining, reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname, reject_unknown_recipient_domain

[erik-wramner]

Actually in that case (clean use of amavis.cf) I would say this is good enough and the issue can be closed?

[VanVan]

It is indeed not complicated to set up, but I think it should be the default operation to let SPAM pass by marking it. The user is free to make a SIEVE rule to move the SPAM to a specific folder.
It is not for nothing that this is the default operation for all webmails, if the user doesn't have the possibility to check his spam folder, it represents a significant risk of missing an email, which can sometimes be important.

[erik-wramner]

I'm afraid I don't agree. Personally I prefer the current behavior, but if it was just me I could configure my own installation the way I like it and change the default. More importantly this is how @tomav set it up once upon a time and changing the default would affect all current installations. That is not a good idea. There is no strong reason for "breaking" existing installations by changing the default; better to let newcomers set the configuration the way they want it. It should probably be a FAQ though.

[casperklein]

I personally prefer the "mark spam mails" over "reject spam mails (silently from receiver perspective)" approach. That's because missing an important email due a false positive spam detection is not acceptable for me.
IMHO, that should have been the default from the beginning. However I fully agree with @erik-wramner, that changing the default now and breaking existing installations is a no-go.

[Nebucatnetzer]

Maybe a good opportunity to

[Nebucatnetzer]

Ups sorry that went of faater than intended.

Maybe this is a good opportunity to implementing a way to mark options as depricated?
I noticed on other docker projects that changes are bit hard to communicate.
Maybe something to display in the logs?

[erik-wramner]

Well, I don't consider this as such a case as anyone who wants the other behavior can make the change; it is just a matter of documenting it. However, the idea about using the logs might help with the buster upgrade. Unfortunately that only works when people restart with a new image and there are many who run with the same version for a very long time without docker pull.

[VanVan]

1431 To put everyone in agreement, I've added an option to put SPAM in the inbox, without changing the default operation. To do so, you just have to change an environment variable.

[erik-wramner]

Excellent. It would be even better if you could add a test for the option. The old behavior should be covered already. Also note that as I recall this is handled a bit differently in the next branch.

[VanVan]

I've looked at the tests, it would be a bit long to write, since an SMTP conversation doesn't make a specific return, and then you have to check the inbox to verify that the spam has been received.
You can check in a minute that enabling the environnement variable do the same modification as mentioned above.
However, the code is short, very careful in its approach, as it checks the content of the files before modifying them, and makes the same modifications as the code already mentioned above. I checked and tested it successfully. Everything works correctly.

[erik-wramner]

I'm sure you did but I think this changes in Buster (as I recall the option is in another file) and it might change again in the future. Having tests for all options help us find that before someone complains. The easiest way would be to clone one of the existing tests and change the expected values. Still, I think the PR looks safe, so I'll probably merge it anyway if you don't have the time.

[VanVan]

Thank you, I understand your position, but the time to perform the test is longer than the functionality itself.
After verification, the code is compatible with the current buster version on the "next" branch.

[erik-wramner]

Merged PR, thanks!

[polarathene]

There is no strong reason for "breaking" existing installations by changing the default;

That's the purpose of respecting semver. A major version is free to introduce breaking changes, and such changes can be communicated clearly via a changelog(which exists in the repo and migrated to releases).

So providing the generally expected marked as spam and sieved to a folder as a default is perfectly valid. The changelog can point to wiki documentation for users upgrading that would prefer to keep the prior behaviour.

[youtous]

A major version is free to introduce breaking changes, and such changes can be communicated clearly via a changelog

Seen this behavior is silent and have a negative impact on mail delivery, could we introduce a breaking change in the next major release?

I will also submit a PR to add SPAMASSASSIN_SPAM_TO_INBOX in the README.me

Note: this spamassassin setting needs ENABLE_SPAMASSASSIN=1. By default, the mailserver is configured to quarantine spam emails. If emails are quarantined, they are compressed and stored in a location dependent on the ONE_DIR setting above. If ONE_DIR=1 the location is /var/mail-state/lib-amavis/virusmails/. If ONE_DIR=0 it is /var/lib/amavis/virusmails/. These paths are inside the docker container. To inhibit this behaviour and deliver spam emails, set this to a very high value e.g. 100.0.

_current README_

This paragraph should warn the user about bouncing reject behavior of spams.

I would like to add SPAMASSASSIN_SPAM_TO_INBOX=1 in README examples. It will raise curiosity of users and provide a way to recommend the spam to inbox behavior (which is default on the majority of MSP) without introducing breaking changes.

I consider the default bouncing reject behavior for spams as risky: users are not informed and the administrator will certainly miss this (https://github.com/tomav/docker-mailserver/issues/1471).

[erik-wramner]

Better documentation is always good. As stated above I personally prefer the current default. I get about 1000 spam per day to my personal accounts, they date to 1998 and are on too many lists by now. I can't reasonably go through all that so bouncing is cleaner. If it is simple to change then people can choose what they like and we don't have to change the default and possibly break things.

@tomav what is your view?

[polarathene]

If it is simple to change then people can choose what they like and we don't have to change the default and possibly break things.

Then it should be made very clear early on to the user thinking of adopting the image that by default they may have email not delivered because it was accidentally treated as spam.

Is the email never kept/stored by default? So if something important was sent and accidentally considered spam... it's gone? If you have it automatically going to a spam folder and being cleaned up after a certain duration has passed, that sounds much better...?

Some users may not realize this default until it's already too late?

[youtous]

Bounce spam, the default option for buster is D_PASS to deliver

From target/amavis/conf.d/49-docker-mailserver

Default debian configuration is $final_spam_destiny=D_PASS, it's confusing. Moreover, SPAMASSASSIN_SPAM_TO_INBOX is not working as expected, amavis/conf.d/49-docker-mailserver is not edited when SPAMASSASSIN_SPAM_TO_INBOX=1; spams are bounced.

@erik-wramner Could you reopen this issue?

[VanVan]

You are right. Here is a new PR to fix this problem. #1484

[youtous]

PR https://github.com/tomav/docker-mailserver/pull/1485 introduces a "move to Junk" possibility.

For the next major release, we could stick to the default debian behavior ( SPAMASSASSIN_SPAM_TO_INBOX=1), having more messages is not a breaking change, current behavior results in messages lost which is dangerous.

[erik-wramner]

Reopened. I have a suggestion then. Let's fix this so that there are two options. One is backwards compatible (no change to existing images) the other delivers spam. The backwards compatible version (SPAMASSASSIN_SPAM_TO_INBOX=0) is the default in the script. That way existing installations that upgrade without setting the option won't break. However, you change the environment file to use the new option (SPAMASSASSIN_SPAM_TO_INBOX=1). New users will most likely copy the environment file as a starting point, so they will get spam delivered rather than bounced.

Is that an acceptable compromise?

[VanVan]

@erik-wramner That's exactly what I thought was the best option.

[Nebucatnetzer]

That sounds like a very good solution :)