Docker-mailserver: Problem with strict DMARC policies

[mindrunner]

I am struggling with a problem resulting from a mixture of strict DMARC policies and a forwarding scheme. I found this reddit (without a proper solution) from a guy who runs exactly the same setup than I do. I hope it is okay to just post the link here.

https://www.reddit.com/r/email/comments/6ivppl/forwarding_emails_and_dkimspfdmarc_whats_correct/

My question is, if there might be a solution for this. At the moment, I completely lose this emails and do not get any notification that this happens. The only way to figure out about this is the log file which I do not regularly check.

mail              | Dec 28 14:55:21 mx0 postfix/smtp[2064862]: CDD1762E74: to=<[email protected]>, relay=gmail-smtp-in.l.google.com[74.125.71.27]:25, delay=1.1, delays=0.04/0.01/0.34/0.74, dsn=5.7.26, status=bounced (host gmail-smtp-in.l.google.com[74.125.71.27] said: 550-5.7.26 Unauthenticated email from dhl.com is not accepted due to domain's 550-5.7.26 DMARC policy. Please contact the administrator of dhl.com domain if 550-5.7.26 this was a legitimate mail. Please visit 550-5.7.26  https://support.google.com/mail/answer/2451690 to learn about the 550 5.7.26 DMARC initiative. x17si33555950wrr.148 - gsmtp (in reply to end of DATA command))

I think, there might be different possible solutions:

1) Implement ARC (http://arc-spec.org/). Do we already support this with docker-mailserver? I am not even sure if that would help, but I am willing to give it a try.

2) forward the emails in a way, so that Gmail accepts them. (Can we automatically check the senders DMARC policy and rewrite the From header for those emails? I really do not want to rewrite every single from header and lose the senders information.

3) Notify me in case gmail rejects and mail because of strict DMARC policies.

Any other thoughts are welcome. :)

Cheers
Lukas

[erik-wramner]

I don't think we have ARC, but it seems to be supported by Postfix so that could be an option. If you implement it try to be backwards compatible so that it is off by default or can easily be turned off. Some tests are also nice! Good luck.

[mindrunner]

Can you provide any source for your statement about ARC and postfix?

[erik-wramner]

Sure, http://arc-spec.org:

...and the milters authentication_milter and OpenARC can be used to deploy ARC with the Postfix

[mindrunner]

Ah okay. Yeah i saw that. However, the openARC project does not look really alive to me.
Heaps of open PRs, abandoned open bugs and no commit for more than one year.
I actually wanted to open a discussion here about what the best practice is to solve the problem I mentioned earlier.

[erik-wramner]

Right. Let's see if someone has a better idea then.

[mindrunner]

Another project (Mailu) implemented this with a tiny patch:

https://github.com/Mailu/Mailu/pull/616/files

Will that be as easy here or do we need more work previously?

[mindrunner]

Ah, we need rspamd first...

https://github.com/tomav/docker-mailserver/issues/975

[mindrunner]

@erik-wramner I would like to give it a go. As far as I understand, I need to compile the OpenARC module and then add it as a milter to postfix? Is that correct? I am still a bit lost with this. Is there an equivalent milter module already implemented in docker-mailserver?

Thanks for pointing me into the right direction. :)

[erik-wramner]

@mindrunner this turned out to be easier said than done. I can't find any good instructions on how to go about this (didn't spend a lot of time searching though) and OpenARC is still in pre-release. See https://github.com/trusteddomainproject/OpenARC/issues/10 for some pointers, but they are not very good for Postfix.

I think you you should use a two-phase build where you first install the packages and build OpenARC from source, downloading the latest release. Then the second phase is the current docker file, which would copy the compiled binary from the first step and also install the necessary per-requisites. Plus it would install the authentication_milter and configure it to use OpenARC.

Given how bleeding edge this is I don't think I would want it in the master branch for a while, but I have no problems creating an experimental branch. That way you and other interested users can try it out. Ideally I'd like OpenARC to be released and published in a repository for Debian before we integrate it into the mainline version. Some documentation for using it with Postfix wouldn't hurt either!

[mindrunner]

Yeah, figured out the same. It also does not seem a quite active project. I think ARC is going to be more and more important. I wonder why so less people care about this.

Anyways. I might give it a go. Do we have a contributors quick-start document for docker-mailserver. What is the fist steps to build it locally? git clone and makefile?

[erik-wramner]

See https://github.com/tomav/docker-mailserver/blob/master/CONTRIBUTING.md. Please note that you need to build it on Linux. I tried with a Mac the first time and that fails. Don't forget git submodule init.

[mindrunner]

Yea, no Macs on my desk... No worries :)

Build seems to run fine. Will dive into this as soon as there is some time. :)

Thanks so far!

[mindrunner]

Alright, we have a branch here:
https://github.com/mindrunner/docker-mailserver/tree/ARC-1

And an image built from that branch here:
https://hub.docker.com/repository/docker/runmymind/docker-mailserver

At the moment, I am compiling and installing OpenArc from sources while building the docker image. Furthermore, I am using the default port 8894 for OpenArc and I added it to the milters for smtp and non_smtp.

However, it seems not functional. :( Not sure where to look next.

# Milters used by DKIM
milter_protocol = 6
milter_default_action = accept
dkim_milter = inet:localhost:8891
dmarc_milter = inet:localhost:8893
arc_milter = inet:localhost:8894
smtpd_milters = $dkim_milter,$dmarc_milter,$arc_milter
non_smtpd_milters = $dkim_milter,$arc_milter

[le@nx]: ~/src/mail>$ docker-compose logs -f | grep openarc
mail              | 2020-01-05 00:52:24,393 INFO spawned: 'openarc' with pid 1015
mail              | 2020-01-05 00:52:24,393 INFO success: openarc entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
mail              | openarc: started

root@mx0:/# supervisorctl status
amavis                           RUNNING   pid 1442, uptime 0:09:16
changedetector                   RUNNING   pid 1341, uptime 0:09:16
clamav                           RUNNING   pid 1240, uptime 0:09:16
cron                             RUNNING   pid 1002, uptime 0:09:18
dovecot                          RUNNING   pid 1008, uptime 0:09:17
fail2ban                         RUNNING   pid 1138, uptime 0:09:16
fetchmail                        STOPPED   Not started
filebeat                         STOPPED   Not started
mailserver                       RUNNING   pid 9, uptime 0:09:23
openarc                          RUNNING   pid 1015, uptime 0:09:17
opendkim                         RUNNING   pid 1023, uptime 0:09:17
opendmarc                        RUNNING   pid 1031, uptime 0:09:17
postfix                          RUNNING   pid 1038, uptime 0:09:16
postgrey                         STOPPED   Not started
postsrsd                         EXITED    Jan 05 12:52 AM
rsyslog                          RUNNING   pid 1004, uptime 0:09:17
saslauthd_ldap                   STOPPED   Not started
saslauthd_mysql                  STOPPED   Not started
saslauthd_pam                    STOPPED   Not started
saslauthd_rimap                  STOPPED   Not started
saslauthd_shadow                 STOPPED   Not started

[mindrunner]

Seems like it is working:

ARC-Seal: i=1; a=rsa-sha256; d=domain.com; s=201808; t=1578189874; cv=none; b=[.......]
ARC-Message-Signature: i=1; a=rsa-sha256; d=domain.com; s=201808; t=1578189874; c=relaxed/simple; bh=[...]; h=Received-SPF:MIME-Version:Message-ID:Date:From:To:Subject:
     Content-Type:Content-Transfer-Encoding; b=[.......]
ARC-Authentication-Results: i=1; mx0.domain.com; arc=none

[mindrunner]

@tomav @erik-wramner I know OpenArc is considered experimental and there are no official releases nor packages left. But it would be really nice to see this in the official image. How can we proceed with this? Any ideas?

[erik-wramner]

My proposal would be to create a new branch for it with a corresponding docker image. We would then have:

• master/latest, Debian stretch with latest fixes
• stable, Debian stretch from master at known good states
• debian-buster (currently work in progress in my repo), basically an upgraded master
• experimental, Debian stretch or possibly buster with ARC and other new features
• versioned images for releases

Then we'll add your fix to experimental and move it to master when it can be considered stable enough; I would like it to be adopted by Debian first. However, anyone who wants to use it before that can simply use the experimental image. @fbartels and @tomav what is your view?

The alternative for getting it into master would be if we can get it in without a lot of extra (possibly dangerous) dependencies and in a disabled state, so that it has to be activated with an option. With a https://docs.docker.com/develop/develop-images/multistage-build (so that we don't install build tools) that might be possible.

[mindrunner]

Cool, that sounds good to me.

debian-buster (currently work in progress in my repo), basically an upgraded master

I was thinking of doing this. Glad to hear that you are already on it. Is there information about the issues you have at the moment? (Sorry, this is a bit off-topic here)

experimental, Debian stretch or possibly buster with ARC and other new features

Do we already have this branch so that I can rebase my changes to this and file a PR?

I would like it to be adopted by Debian first.

I would prefer having it in master/stable before I retire, though

The alternative for getting it into master would be if we can get it in without a lot of extra (possibly dangerous) dependencies and in a disabled state, so that it has to be activated with an option. With a https://docs.docker.com/develop/develop-images/multistage-build (so that we don't install build tools) that might be possible.

I could work on .deb packages build regularly, so we can integrate them into the image without the need of building it. This would have been my next step. There is already instructions upstream for building .rpm, so should not be a big deal.

[fbartels]

Hi, in general that sounds like a sound idea, just some remarks

• master/latest, Debian stretch with latest fixes

+1

• stable, Debian stretch from master at known good states

that is basically already the tagged release. if releases are maintained in a branch, then this usually comes with the request to maintain these further on.

• debian-buster (currently work in progress in my repo), basically an upgraded master

+1, but imho this could also be named e.g. "next" to indicate that its a work in progress to be merged back to master

• experimental, Debian stretch or possibly buster with ARC and other new features

I do get the idea behind this, but what would be the plan going forward for this branch? (especially with the "and other new features"). Will you merge back changes in master into this branch and then cherry-pick new features to master once they can be considered stable? That sounds a bit messy.

Thinking further maybe it would be better to have an "arc" branch just with this feature?

• versioned images for releases

this is then just the tagged releases on the docker hub, or also a/multiple branches?

[erik-wramner]

@fbartels master/latest, stable and the versioned images already exist, they are what we have now.

We could certainly name the buster branch "next" or something like that. Then the branch can live on when we merge it back to master as well and represent the next major release.

I intended to cherry-pick changes from experimental (and @mindrunner it does not exist yet), but yes it will get a bit messy. We could have an arc branch that is basically the master branch but with arc. That will also get messy if other features are introduced. We don't want too many branches.

Perhaps it is best to try to get it into master quickly after all. A deb package is one option, a multi-stage build also works. The key to that is to ensure that it doesn't affect the stability of the image in any way for people who don't use it. If we use a deb package I would prefer to build it in a docker container from the make file rather than adding it as a binary file built elsewhere. It could be an optional step (reuse deb file from last build if it exists, build it if missing) to keep the normal builds fast. Then we will always get the latest code when we build for real.

[mindrunner]

but imho this could also be named e.g. "next" to indicate that its a work in progress to be merged back to master

+1

experimental, Debian stretch or possibly buster with ARC and other new features

Do we need next and experimental? I think all new changes could go to next. If we have a super experimental feature (e.g. ARC), it should remain in it's own branch with rebases on experimental until it is ready to get staged.

this is then just the tagged releases on the docker hub, or also a/multiple branches?

I would say former. Tagged releases on docker hub and optimally git-tags as well.

--->
master -> current stable version
next -> development branch
arc -> my current changes which go to next eventually

A deb package is one option, a multi-stage build also works.

I think, there will be official packages soon. (maybe this year). I am already in contact with some of the upstream devs. Until then, let's go for the multistage build. (I already implemented this in my branch)

[erik-wramner]

I've created an arc branch based on master, feel free to make a PR with your changes.

[mindrunner]

PR filed! Also, i made sure, this is an optional feature, disabled by default.

So I do not see any big danger of merging this to master. (After some others have tested it of course)

[mindrunner]

I still have issues with this, though. :(

If I purely forward the Emails to Gmail, they get rejected (see first post)

If I forward them to Gmail AND store them into a docker-mailserver, they seem to be accepted by Gmail, but as SPAM and dangerous. :/

Also, docker-mailserver is then sending them twice to Gmail. (see https://github.com/tomav/docker-mailserver/issues/1355)

[erik-wramner]

Too bad. Perhaps you could ask in the Postfix forum? This should not be specific to docker-mailserver.

[mindrunner]

No, I know this is not an issue in docker-mailserver. The problem is rather that there is no solution. ARC does not create a trust between servers. It just re-signs messages. IMHO all this DMARC ARC stuff is big bullshit which creates more problems than it solves. But I am still learning.... I think the issue here is Google, which has some weird automagic and does not let me configure my mailsettings properly... :(

[aendeavor]

@mindrunner Implementing ARC seems very reasonable to me. Any progress with this?

[mindrunner]

We have an experimental branch with ARC working. Probably need to be rebased to master if someone needs it. However, I am not using ARC anymore because even though it is a good idea, it brings more problems than it solves (Especially if you forward to blackboxes like Gmail)

Did you test the arc-branch yet? Let me know if you need any help.

https://github.com/tomav/docker-mailserver/tree/arc

[aendeavor]

Alright. I just needed to know, as I'm going through old issues to close them if needed. This seems to be somewhat "solved". Would you like to keep this issue open due any other reason I can't think of right now? :)

[mindrunner]

I'd be happy to see the changes in master, however my personal need for this is more or less zero, so I am also happy to leave everything as is if there is no interest.

[aendeavor]

I get it. First of all: Thank you for investigating in this. Contribution is always nice to see. We'll leave the arc branch open. Maybe someone in the future is interested.

Currently there seems to be no interest however and there are a lot of other things going on related to master (looking at you, #1605). Therefore, I'll close this and mark it as frozen. If you or anyone else feels like re-opening this issue because of some reason, feel free to do so.