Mailu: Looking for maintainers for Rancher, Swarm and K8s configurations

[kaiyou]

I am not a user of Rancher, Swarm of Kubernetes for Mailu and does not have enough time to maintain the documentation and configuration. Also, any help is more than welcome in maintaining those. If anyone has the skills to do this, please step up and I'll gladly merge your work or PRs, or simply listen to your comments.

[ghost]

I can do it for rancher, next to tackle kubernetes, as I have both... swarn... no idea... there was an article someone did also on k8s

[nielsvandermolen]

Hi @kaiyou,

I am currently working to get the stack of Mailu to work on a Docker swarm cluster (of one node atm).

It is an existing cluster that host a number of other applications therefore I have to integrate it with my own NGINX proxy. I have a custom Let's Encrypt solution that is somewhat similar as what you did.

As I have no experience in Python and ran into some issues with the NGINX auth_http script. Therefore I decided to decouple that part and use my own authentication script and use Mailu purely for the admin configuration part and the integrations between the different applications.

Mainly recreating this part in PHP:
"Mailu-master/core/admin/mailu/internal/nginx.py"

Some general comments so far:

• The "front" address was used in some places in the postfix and dovecat containers. To change this configuration I had to override the images.
• The .env file is not supported (and it looks like it will not be supported) in Docker swarm. It is best to at least list what environment variables are needed per application (check my current status in the attached file)
  wip_swarm_config.txt
• Using NGINX upstream to point to a Docker service like in this blog article seems to work well with mail:
  how-to-set-up-and-deploy-to-a-1000-node-docker-swarm/
• Currently working on password hashing. Since I have to authenticate on a different machine it seems needed to provide a salt for the hash in models.py:

     def set_password(self, password, hash_scheme=app.config['PASSWORD_SCHEME'], raw=False):
        """Set password for user with specified encryption scheme
           @password: plain text password to encrypt (if raw == True the hash itself)
        """
        # for the list of hash schemes see https://wiki2.dovecot.org/Authentication/PasswordSchemes
        if raw:
            self.password = '{'+hash_scheme+'}' + password
        else:
            **hash = sha512_crypt.using(salt='<Custom salt>', rounds=<Custom Rounds>).hash(password)
            self.password = '{'+hash_scheme+'}' + hash**

• Port 25 is blocked by most internet providers from the Netherlands. It can save you some time to check this when you run into some issues during testing.
• Reserve some time when you decide to work on this. Email has proven to be much harder than I estimated and I now understand some of these offerings by third parties.

Thanks for your work so far on this effort to create a mail server which is easy to use and free for everyone.

[kaiyou]

On your comments :

• I'll check most images for configuration-time DNS resolution and fix that according to recent @mildred PR ;
• Documenting which configuration is for which image is all part of preparing the Swarm image indeed.

Regarding the authentication issue, the current philosophy for Mailu is: all should go through nginx. It does not mean that you cannot use your own nginx in front of it, this is what notls and mail encryption modes are for. Could you open a separate issue for this or ping me on IRC/Matrix so we can discuss it?

[nielsvandermolen]

Thanks!

Regarding the DNS:

Mailu/core/postfix/start.py
os.environ["FRONT_ADDRESS"] = socket.gethostbyname("front")

Mailu/core/postfix/conf/main.cf

# Only the front server is allowed to perform xclient
smtpd_authorized_xclient_hosts={{ FRONT_ADDRESS }}

Similar as in the start.py of dovecot
Mailu/core/dovecot/conf/dovecot.conf
submission_host = front
FRONT_ADDRESS is used in
Mailu/core/dovecot/conf/dovecot-sql.conf.ext

[nielsvandermolen]

Managed to get the stack working on Docker swarm including antispam (Rspamd) and webmail (RainLoop). Still using my custom PHP script for the NGINX authentication to point to the correct upstream Docker services but the changes made in the PR might make it possible to switch back to the authentication system of Mailu.

Tested with multiple domains to send and receive with SMPT and the IMAP connection. Server passed all checks on MX Toolbox. Kudos again on the work on this project so far.

Here are the files I used. Feel free to take and adjust them and use them in your project(s). Note that the authentication script contains some debugging code (which logs the passwords!) and hard-coded values so adjust if you use it in a production system.

• Mail nginx config:
  nginx.conf.txt

• Upstream server configuration:
  upstream_nginx.conf.txt

• PHP Nginx authentication script that links it together:
  authentication.php.txt

• Docker Swarm file
  swarm_config.yml.txt

Some small issues that are not working:

• Authentication tokens in my custom authentication script
• Docker system performance in Mailu admin does not work

[ghost]

It would be much better if we get environment variable documentation (for ALL of them) - related to https://github.com/Mailu/Mailu/issues/284

I also have a working swarm template (with or without .env file hack, can do both) but it is really messy in env vars parts. Every time we have a new env var added to upstream I need to lookup for it in code.

I can only help with some issues, can not be a maintainer - have no time for it =(, can't even fix my last merge request...

Here is my current template if you need it (this one is for env hack - start with echo "$(docker-compose -f /mnt/docker/apps/mailu/docker-compose.yml config 2>/dev/null)" | docker stack deploy -c- mailu, but it works with 1.5.1 without any additional fixes)

version: '3.3'

services:
  front:
    image: mailu/nginx:$VERSION
    env_file: .env
    ports:
      - target: 80
        published: 80
        mode: host
      - target: 443
        published: 443
        mode: host
      - target: 110
        published: 110
        mode: host
      - target: 143
        published: 143
        mode: host
      - target: 993
        published: 993
        mode: host
      - target: 995
        published: 995
        mode: host
      - target: 25
        published: 25
        mode: host
      - target: 465
        published: 465
        mode: host
      - target: 587
        published: 587
        mode: host
    volumes:
      - "$ROOT/certs:/certs"
    deploy:
      restart_policy:
        condition: on-failure
      mode: global
      endpoint_mode: dnsrr
      placement:
        constraints:
           - node.labels.mailer == true

  redis:
    image: redis:alpine
    volumes:
      - "$ROOT/redis:/data"
    deploy:
      restart_policy:
        condition: on-failure
      mode: replicated
      replicas: 1
      placement:
        constraints:
           - node.labels.mailer == true

  imap:
    image: mailu/dovecot:$VERSION
    env_file: .env
    volumes:
      - "$ROOT/data:/data"
      - "$ROOT/mail:/mail"
      - "$ROOT/overrides:/overrides"
    depends_on:
      - front
    deploy:
      restart_policy:
        condition: on-failure
      mode: replicated
      endpoint_mode: dnsrr
      replicas: 1
      placement:
        constraints:
           - node.labels.mailer == true

  smtp:
    image: mailu/postfix:$VERSION
    env_file: .env
    volumes:
      - "$ROOT/data:/data"
      - "$ROOT/overrides:/overrides"
    depends_on:
      - front
    deploy:
      restart_policy:
        condition: on-failure
      mode: replicated
      replicas: 1
      endpoint_mode: dnsrr
      placement:
        constraints:
           - node.labels.mailer == true

  antispam:
    image: mailu/rspamd:$VERSION
    env_file: .env
    volumes:
      - "$ROOT/filter:/var/lib/rspamd"
      - "$ROOT/dkim:/dkim"
      - "$ROOT/overrides/rspamd:/etc/rspamd/override.d"
    depends_on:
      - front
    deploy:
      restart_policy:
        condition: on-failure
      mode: replicated
      replicas: 1
      placement:
        constraints:
           - node.labels.mailer == true

  antivirus:
    image: mailu/$ANTIVIRUS:$VERSION
    env_file: .env
    volumes:
      - "$ROOT/filter:/data"
    deploy:
      restart_policy:
        condition: on-failure
      mode: replicated
      replicas: 1
      placement:
        constraints:
           - node.labels.mailer == true

  webdav:
    image: mailu/$WEBDAV:$VERSION
    env_file: .env
    volumes:
      - "$ROOT/dav:/data"
    deploy:
      restart_policy:
        condition: on-failure
      mode: replicated
      replicas: 1
      placement:
        constraints:
           - node.labels.mailer == true

  admin:
    image: mailu/admin:$VERSION
    env_file: .env
    volumes:
      - "$ROOT/data:/data"
      - "$ROOT/dkim:/dkim"
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on:
      - redis
    deploy:
      restart_policy:
        condition: on-failure
      mode: replicated
      endpoint_mode: dnsrr
      replicas: 1
      placement:
        constraints:
           - node.labels.mailer == true

  webmail:
    image: "mailu/$WEBMAIL:$VERSION"
    env_file: .env
    volumes:
      - "$ROOT/webmail:/data"
    deploy:
      restart_policy:
        condition: on-failure
      mode: replicated
      endpoint_mode: dnsrr
      replicas: 1
      placement:
        constraints:
           - node.labels.mailer == true

  fetchmail:
    image: mailu/fetchmail:$VERSION
    env_file: .env
    volumes:
      - "$ROOT/data:/data"
    deploy:
      restart_policy:
        condition: on-failure
      mode: replicated
      replicas: 1
      placement:
        constraints:
           - node.labels.mailer == true

[muhlemmer]

I don't think this issue was meant to discuss the exact implementation of Swarm, but a call for arms.

With regards to Swarm, I'm already deep into the subject after bug hunting the last week or so. I need it working for my own environment. So for now, yeah I would like to volunteer to the task. (I'll just keep sending Pull requests)

[ghost]

Ive already stated id be happy to maintain them once they are known
working, i fought with mine, but did finally get it done on rancher

On Thu, Aug 9, 2018 at 1:43 PM Tim Möhlmann [e-mail]
wrote:

I don't think this issue was meant to discuss the exact implementation of
Swarm, but a call for arms.

With regards to Swarm, I'm already deep into the subject after bug hunting
the last week or so. I need it working for my own environment. So for now,
yeah I would like to volunteer to the task. (I'll just keep sending Pull
requests)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Mailu/Mailu/issues/375#issuecomment-411729554, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABXFTqkEYFNWnDNWykjWeD1UomljscYOks5uPCB6gaJpZM4RvPHi
.

[kaiyou]

I am closing here regarding the latest really good pull requests. I added @muhlemmer as a reviewer, also any help is of course still welcome :)

[consideRatio]

@kaiyou I'm curious about helping out regarding k8s and the development of a Helm chart to deploy Mailu. Feel free to ping me regarding these matters!

---

Helm is like a package manager for k8s, an abstraction on top of using kubectl.

helm install stable/mailu --name mailu --values myconfig.yaml

Examples:

• Mattermost Helm chart
• GitLab Helm chart
• JupyterHub Helm chart (This is my entrypoint to working with k8s / Helm)