Docker-mailserver: Docker MailServer V2

[tomav]

Here is the list of what should be done in version 2 of this image

• [x] Reduce image size
• [ ] Move to Alpine or other lightweight image
• [X] Use encrypted user database + generator from clear password text file (#123)
• [x] Comment start-mailserver.sh to help maintenance
• [x] Move from /tmp folder to /var/docker-mailserver or equivalent
• [x] Rename files (some are .cf, some are not)
• [x] Rename env using convention
• [x] Move static files to target folder
• [x] git clone + docker-compose up should work _out of the box_
• [x] Add tests on SSL certificates, at least for letsencrypt
• [x] Clean Fail2ban configuration
• [ ] Refactor integration tests workflow in Travis
• [X] Add Sieve
• [x] Review documentation (README.md and Wiki)
• [ ] Find another project name :blush:
• [ ] Create Github organization
• [ ] Find 1 or 2 core contributors to help project maintenance

[mgansler]

Hi!

Take a look at https://gitlab.martingansler.de/docker/mail
I basically took your postfix configuration and integrated it in an alpine/s6 based image.
Currently it doesn't result in a working server but it should help you figuring out how to start the services etc.
Unfortunately I don't have much time at the moment, so that all I could hack together in the last 30 minutes or so.

[tomav]

The challenge for us if to find a lightweight and secure distro with all our needed packages.
After a first look with @KyleOndy it seems that some packages are not available on Alpine.

[mgansler]

Do you remember which ones are missing?

[00angus]

First of all ... what a great project @tomav :)
Really! I'm enjoying using it, while trying to contribute some changes.
One of the key issue is the 'encrypted database': well, I'm currently running a modified docker-mailserver image where all _cyrus-sasl_ and _courier-*_ has been replaced with dovecot: it builds and it WORKS :)

My current implementation:

• postfix auth for sending out emails is now based on dovecot (sasl)
• imap, pop3 are all served by dovecot, and SSL is configured accordingly to DMS_SSL env
• imap, pop3, postfix all use (thanks to dovecot) the same user-database with passwords encrypted
• users are still managed via accounts.cf but now the passwords are encrypted
• the supported encryption scheme is wide: CRYPT MD5 MD5-CRYPT SHA SHA1 SHA256 SHA512 SMD5 SSHA SSHA256 SSHA512 PLAIN CLEAR CLEARTEXT PLAIN-TRUNC CRAM-MD5 SCRAM-SHA-1 HMAC-MD5 DIGEST-MD5 PLAIN-MD4 PLAIN-MD5 LDAP-MD5 LANMAN NTLM OTP SKEY RPA SHA256-CRYPT SHA512-CRYPT
• services startup has been adapted accordingly, of course using the same logic
• moreover I've added an option to import (from a volume) the DKIM keys: the idea being I don't want to change my dns at every restart ! It's not a dovecot thing, but I found it useful.

Having to manage a single file for the accounts is considerably simple than my previous patch on a similar subject (not yet merged BTW). Moreover now every auth service in the container use the same DB (well, except for mail relay).

What remains to be done:

• I think an overall look at the dovecot configuration is a good idea
• integration tests need to be reworked: I've not found the time to do that, but it's needed

Are you interested in this fork ? How do you suggest to contribute (since the lack of the integration tests) ?

Regards.

[tomav]

Hi @00angus, of course we're interested for the v2 milestone.
How do you manage alias/forwards?

[00angus]

@tomav postfix/virtual as in your project.
I've maintained the same architecture you introduced, with the same overall approach... We should however test the config and the security of the solution.
Anyway I've also tested my setup using free online services like checktls.com.

[joshmanders]

Regarding packages in Alpine, have you thought about building from source? Building on Alpine is a highly suggested route. I'd like to incorporate this project into my services but I am not keen on 600MB images.

[tomav]

Could be more difficult to maintain...

[00angus]

@tomav how do you want to proceed? Would you like to create a new branch, say dovecot, where I can make PR?

[tomav]

@00angus branch will be v2.
Empty branch or from master?

[00angus]

Maybe it's better from master ? My branch is derived from your on master
anyway, with not so much things changed.
They can merge, I think.
Anyway for me is the same.

On Fri, Apr 8, 2016 at 3:12 PM, Thomas VIAL [email protected]
wrote:

@00angus https://github.com/00angus branch will be v2.
Empty branch or from master?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/tomav/docker-mailserver/issues/109#issuecomment-207425866

[tomav]

https://github.com/tomav/docker-mailserver/tree/v2

[KyleOndy]

I would like to see what you have gotten so far with your approach @00angus.

[00angus]

@KyleOndy I'm integrating the latest changes of master-v2 ... and I'll call for a PR.
Anyway CI tests don't work: @tomav do I have to take that into consideration someway when making the PR? Because I don't know in case the Travis-CI get activated if it would build (I don't know how it works).

[00angus]

PR submitted.
The Travis-CI build is started ... If it's going to use the make all command for the build, it could end-up in a green state:

$ make all
docker build --no-cache -t tvial/docker-mailserver:dovecot .

I've changed the makefile to:

• comment out fixtures, tests and clean targets from the _all_ rule
• give a version to the image, the version being dovecot. This way on the same host I can have two images compiled for courier (v1) or dovecot (v2) at the same time.

[tomav]

Merged https://github.com/tomav/docker-mailserver/pull/123

A v2 tag should appear soon on Docker Hub.
https://hub.docker.com/r/tvial/docker-mailserver/tags/

[tomav]

I'm about to push some enhancements.

[tomav]

Done.
Tests are now a high priority.

[00angus]

Waiting for it :)
We will have to merge some commit, moreover ;)

[tomav]

@00angus the build is now named tvial/docker-mailserver:v2 (from the branch name)

[00angus]

Ok ... then I'd suggest to put somewhere the information that v2 == dovecot
... Readme or something else...

-- Marco

On Tue, Apr 12, 2016 at 9:50 AM, Thomas VIAL [email protected]
wrote:

@00angus https://github.com/00angus the build is now named
tvial/docker-mailserver:v2 (from the branch name)

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/tomav/docker-mailserver/issues/109#issuecomment-208762795

[tomav]

Yeah, we'll do that when work will be finished. As you can see at the top of this thread, migration to dovecot is not the only task.

[00angus]

Well, it's only an opinion, but since v2 is downloadable maybe people want
to know what's inside..
But OK, let's wait.
On Apr 12, 2016 12:53, "Thomas VIAL" [email protected] wrote:

Yeah, we'll do that when work will be finished. As you can see at the top
of this thread, migration to dovecot is not the only task.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/tomav/docker-mailserver/issues/109#issuecomment-208844245

[mgansler]

Great work so far!
Did you think about migrating from v1 to v2?
With IMAP you can always just start ab v1 and v2 parallel and copy all messages with your mail client, but maybe there is a better way?
Or is it possible to just use the same volume?

[tomav]

Good question @mgansler
@00angus dovecot uses Maildir format like postfix no?

[tomav]

@00angus I really don't want to release a new version to quicly.
Tests are disabled right now on this branch and documentation is not up to date.
I think that Clamav doesn't start properly and I haven't tested it myself for the moment.

Of course, code and build are available for testing but really not recommended for production right now.

I should get some time at the end of the week to close some of the tasks but I will need your help, especially on tests regarding dovecot.

If some other guys want to contribute, let me know, let's talk about what are the things to do and how we should do them. I'm open to suggestions.

Thank you.

[00angus]

@tomav I'm not sure that v2 is still working by looking at your recent patches, but I may be wrong.
Because...

this line of start-mailserver.sh look for a different path from this other one for user config file.

I'll have a closer look at it this afternoon, anyway, and I'll try it out.
I will definitively help on tests.

[tomav]

I missed this line. I'll have a look.

[00angus]

Dovecot uses maildir inboxes, as courier, as for configs.

[00angus]

Haven't tested/used it:

http://wiki.dovecot.org/Migration/Courier

[KyleOndy]

I'm putting my work on the alpine based container on indefinite hold. At this point I can't see it working without jumping through a lot of hoops.

I know image size is important, but perhaps we should get a solid v2, then work to reduce size in v2.x?

[tomav]

I agree. Again we need tests to be sure that nothing breaks on big changes like changing the distribution.
I don't know what is the amount of work... Perhaps the best thing is to have the exact list of packages and see if they exist in stable alpine distro.

I created this file with the v2 packages:
https://docs.google.com/spreadsheets/d/1VFkt-741KVjFqOTpu60Qj--PYEtWHqwYA3SC0DY6TFs/edit?usp=sharing

You should have write access.

[tomav]

@00angus: I just pushed fixes.

Now we have 49 tests, 26 failures.

[tomav]

49 tests, 21 failures

[tomav]

I let you have a check for authentication tests on imap. You have a better dovecot knowledge than me.

[tomav]

BTW, I think you've disabled accounts in postfix, so smtp auth does not work.
No more time for today, let me know if you have suggestion.

[00angus]

I'll have a look but not today as I'm busy. Anyway I'm using in production the dovecot image I PR and it does smtp auth (but I've also custom main.cf ...?). I'll have a closer look tomorrow.

[tomav]

Could you diff your main.cf?

[00angus]

I'll look at my conf later today... However master.CF should open up
submission port!?
Anyway I'll let you know as soon as I have time to dedicate at the subject.
On Apr 15, 2016 00:10, "Thomas VIAL" [email protected] wrote:

Could you diff your main.cf?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/tomav/docker-mailserver/issues/109#issuecomment-210176428

[tomav]

49 tests, 12 failures. Better.

Most of remaining tests are related to authentication.
@00angus I'll need your help, I'm not a Dovecot expert.

BTW, /etc/dovecot/userdb contains clear login/passwords.

[00angus]

This all sounds very strange as pwd are taken from accounts.cf, which if
encrypted should remain encrypted.

I'm out of time these days, sorry. But on the WEnd I'll find enough time.

[00angus]

@tomav I've tried to understand the problems on v2 you have commented here:

• smtp auth: is disabled in your version of V2 simply because the main.cf you are ADDing (via Dockerfile) to the image _is almost empty_, i.e.:

 max_idle = 600s
 readme_directory = /tmp

_It cannot works_. I'm reverting the configs from my commits, while keeping your dir layout

• dovecot userdb: it contains clear password simply because you are providing postfix-accounts.cf _that contains clear passwords_! You haven't commented the README.md file: is it clear to understand and to follow for setting up the users ? Please let me know :)

I'm trying to understand your recent changes (e.g. which layout for the host volume shared...) and since no docs exist on the tests I'm trying to figure out how they work (i.e. auth login which user and password is using? etc.). Anyway... I'm working on it.

[tomav]

smtp auth: is disabled in your version of V2 simply because the main.cf you are ADDing (via Dockerfile) to the image is almost empty

That's what I found yesterday and pushed a new version. Are you sure you got the updated version?

[00angus]

No - it seems I've fetched v2 a few moments before you pushed, and I started debugging on that version.
Glad to know you worked it out :)

[00angus]

@tomav Can I ask why in the Dockerfile do you need to add "sasl2-bin" package ??

 This is the Cyrus SASL API implementation ...

[tomav]

Testing SASL only. testsaslauthd in part of this package.
If you have other options, let me know. I'd like not to have to install specific binaries for testing (but having a large test coverage)

[00angus]

Ok, I'll let you know.
On Apr 16, 2016 22:46, "Thomas VIAL" [email protected] wrote:

Testing SASL only. testsaslauthd in part of this package.
If you have other options, let me know. I'd like not to have to install
specific binaries for testing.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/tomav/docker-mailserver/issues/109#issuecomment-210898214

[00angus]

testsaslauthd can safely be substituted with doveadm auth test (parte of dovecot-core) and that's what I've already done in my branch.

I finally managed to have :

49 tests, 2 failures

But I've a question for you @tomav :) since I don't know very much of how the tests work.
The two failures comes from fail2ban checks:

• how can fail2ban effectively ban via iptables inside the container if we are not running the mail_fail2ban with the needed privilege?

Even in maste branch the Makefile that startup containers for tests do not add special privilege to that container ... and so I'm surprised that tests are working there. Do you have any idea on that ?

Thanks.

[tomav]

Great news!
I can work on fail2ban if you want. We should not try using 127.0.0.1 so I'll do it from another container.

Push your current work and I'll finish fail2ban.

[tomav]

Commit https://github.com/tomav/docker-mailserver/commit/72d72076901ea2e6f4c6d53c46c7a7a03c4cdfa3

Building on Travis...

[tomav]

Works locally, not on Travis. Trying something else.

[tomav]

Done. Your build exited with 0.
https://travis-ci.org/tomav/docker-mailserver/builds/123774801

[00angus]

Good :)

[00angus]

@tomav in the new commits I've found a new file:

target/fail2ban/jail.conf

It seems that the start-mailserver.sh doesn't need any more the fail2ban setup for jail.conf ... what do you think ?

[tomav]

What has been done in start-mailserver.sh should have been done in Dockerfile because it's core configuration. The jail.conf I added is the generated version + a modification regarding backend configuration. I also removed some unused lines.

I'll clean this today.

[KyleOndy]

In regards to running tests, are you having you docker-compose/yml point to your locally build image? I have having issues getting it to see the local image, docker-compose keeps trying to pull from docker.io and I get ERROR: Tag V2 not found in repository docker.io/tvial/docker-mailserv.

Can someone share their docker-compose.yml with me?

[tomav]

@KyleOndy it's v2 not V2
https://hub.docker.com/r/tvial/docker-mailserver/tags/

[tomav]

@00angus I think we should add a generator that output the whole line needed that can be added using a redirect to postfix-accounts.cf using:

$ docker run --rm -ti tvial/docker-mailserver:v2 doveadm pw -s MD5-CRYPT -u [email protected] > config/postfix-accounts.cf

What do you think regarding removing the pipe in this file?

[00angus]

We could provide a command, e.g. something like:

  u="[email protected]" && docker run --rm -ti tvial/docker-mailserver:v2 /bin/bash -c "echo ${u}|`doveadm pw -s CRAM-MD5 -u $u`" > config/postfix-accounts.cf

A space instead of a pipe may not be part of the password and username, so probably would be better.

[tomav]

That's it. But I think a generator will be easier.

[00angus]

Ok, but then I'm missing what do you mean with 'generator' :) The above command isn't a sort of a generator?

[tomav]

Perhaps a shell script like the certificate generator. We can start without but I think it will provide more simplicity to generate users.

[tomav]

BTW, My last commits broke the tests. I'm fixing it.

[tomav]

Just forgot to commit a file...

[tomav]

Just for information, I did not have time to check a migration to Alpine.
But a quickly tried cloudgear/ubuntu which allow to save 38MB (only...)

[KyleOndy]

I'm was working on a debian:jessie branch today, but it ended up being larger than the ubuntu:14.04 image.

[tomav]

Migrated my mail server from latest to v2. Everything works as expected so far.

[00angus]

Great!
:)
I've tested mine with several online check tools, but the best way to
improve our configuration is by using it :)

On Wed, Apr 20, 2016 at 9:31 AM, Thomas VIAL [email protected]
wrote:

Migrated my mail server from latest to v2. Everything works as expected
so far.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/tomav/docker-mailserver/issues/109#issuecomment-212298734

[tomav]

Yep. I'm currently working on the README. I'll push something in few minutes.

For those who ask, /var/mail remains the same so you will keep your content when upgraded but we recommend to backup everything before, and we're still in Beta.

[tomav]

v2 is now 411.5 MB while latest is 525.3 MB

[KyleOndy]

So now that I've spent a few days hacking at the alpine image, I don't see anyway its going to work.

It seems that alpine:3.3 is based off of busybox, which breaks the test suite since the ps exec doesn't support the proper command line arguments.

Even if we were to rewrite all the tests, alpine doesn't have an init system, so there is no way to start multiple services needed for this container.

Its a shame, it was floating around 130MB before I hit the init roadblock.

[mgansler]

@KyleOndy would you mind sharing your progress so far?
I think it should be possible to install ps.
And I also played around a little with s6, a very small init like system.

[KyleOndy]

Let me clean up (a bit) and I'll push it. Full disclosure, it is in rough shape.

[00angus]

As for the init system, normally in a docker container it's not used. But
since we start several services it could be useful. Anyway what if we
script the full command line startup of each service?
Consider the following:

• postfix binary self manage the usual stop, reload. That means that on
  cmdline or script we can use that method
• the same apply to dovecot

BTW as @tomav said, I'm a little bit worried about maintenance of the
image. Would it be still applicable to use a workflow like:

• pull image
• docker build
• docker run
  ??

Interesting work however and the image size is very... appealing :)
On Apr 21, 2016 23:33, "Kyle Ondy" [email protected] wrote:

Let me clean up (a bit) and I'll push it. Full disclosure, it is in rough
shape.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/tomav/docker-mailserver/issues/109#issuecomment-213122111

[tomav]

@00angus for a better user experience, keeping pull/run is a better option.

@KyleOndy please push your branch so we'll have a look to see if we can make the necessary effort to make it work, and also check what it means regarding maintenance.

[tomav]

By the way, thank you guys for the work so far, I don't have a lot of time these weeks.
It's nice to know that some of you are involved in this project.

[KyleOndy]

I agree that the upgrade should be a simple pull/run. It fits the ideology of docker.
It also mean I can keep on using watchtower.

---

While I think it is good to strive for a small image size, I'm in the opinion that maintainability should be a higher priority.

[tomav]

I do agree, it should work, maintain and upgrade easily, evan if it's 150 additional MB.

[tomav]

https://hub.docker.com/r/tvial/docker-mailserver/tags/

Image is smaller and smaller, just got 25 additional MB free.

[tomav]

7 additional MB removing man pages.
https://hub.docker.com/r/tvial/docker-mailserver/tags/

[tomav]

I plan to release V2 early May.

[00angus]

Good! The image size is more than acceptable.

Anyway... I'm having an hard time let CI tests pass while building an image
locally. I'm working on a minor patch but 2 or more tests keep failing
(fail2ban, opendkim...). I do not understand how on Travis they passed.
On Apr 25, 2016 06:12, "Thomas VIAL" [email protected] wrote:

_I plan to release V2 early May._

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/tomav/docker-mailserver/issues/109#issuecomment-214120950

[tomav]

We can talk about that is you want. I'm currently traveling for work so I won't be really available before wednesday, but leave a comment here, I'll try to have a look in the evening.

[tomav]

Added tests for letsencrypt.

[untoreh]

docker-slim gave an image of 150MB~ , clamAV is the biggest burden being >100MB, and alpine is not gonna help with virus signatures. When running multiple instances of the mail server the best approach would be running clamav with updates disabled and with /var/lib/clamav mounted in readonly, and manage the updates with like an helper container or manually I guess

[tomav]

If I follow you, an idea can be to add an env variable to disable automatic updates. So freshclam will move out from Dockerfile but I'm not sure that tests still pass without a first update.

Regarding docker-slim, I'll have a look but we need something compliant with docker hub automated builds.

[00angus]

DOVECOT:
By looking at Dockerfile I noticed I made a small error.
The line:

ADD target/dovecot/10-*.conf /etc/dovecot/conf.d/

unfortunately do not add the configuration for mailboxes.conf.
I've changed that, by including 15-mailboxes.conf (altready available in target).
From my checks on my production system it still works, it is only more polished IMHO.
My upcoming PR in including - among other small fixes - also this one.

[tomav]

Will release tonight.

[tomav]

(or in the next minutes)

[tomav]

Dear contributors, v2 is the new master on github and latest + v2 on docker hub.
Legacy version remains available in v1 branch on github & tag on docker hub.

I wanted to thank all of you for the work on this project.

I decided not to change the name of the project for the moment, but have mailody in mind... let me know what you think about it.

[00angus]

Good project and good work. Glad to have contributed and hope to continue.
On May 8, 2016 20:30, "Thomas VIAL" [email protected] wrote:

Dear contributors, _v2 is the new master on github and latest + v2 on
docker hu_b.
Legacy version remains available in v1 branch on github & tag on docker hub.

*
_I wanted to thank all of you for the work on this project. *_
I decided not to change the name of the project for the moment, but have
mailody in mind... let me know what you think about it.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/tomav/docker-mailserver/issues/109#issuecomment-217738141