Docker-mailserver: Kubernetes support

[Rubytastic2]

Loving this project but abandoned docker to many issues on OSX
Is there any interest in coverting this project to kubernetes or add support for it?

Im currently looking at this,
perhaps its an idea to make this project "hybrid"

[tomav]

Hi, @Rubytastic2
I think @tyranron and @keslerm are using Kubernetes.

[tomav]

@iMartyn too

[iMartyn]

I switched to https://github.com/Mailu/Mailu fwiw

[tyranron]

@Rubytastic2 and what is the problem to use this image in Kubernetes?

Just write a Deployment & ConfigMaps you require and that's it.

[tomav]

@tyranron is there a value to maintain a page in the wiki for Kubernetes users? If yes, could you do that? Thanks.

[iMartyn]

deploy.yml.txt
service.yml.txt
fwiw these are what I used to set it up in a single-node cluster (so no need for shared storage). Note that the deployment is set up to my dockerhub image because the original dev hasn't merged my pull request to add the hostname override.

[tyranron]

@tomav yes, we can help people with that page, because there are some some tips which are not obvious enough. Like v1.PodSpec.subdomain usage, or placing docker-mailserver beyond reverse-proxies (which requires proxy protocol enabling).
I can do that but cannot guarantee doing it fast.

Also, +1 for merging #390 of @iMartyn .

[tomav]

@tyranron thanks, this page will be helpful.
Regarding @iMartyn PR, I'll merge as soon as he resolves the conflict. No problem.
Do not hesitate to review other's PR on topics that are important for you.

[iMartyn]

Unfortunately my patch was before the script was effectively rewritten. I'm not going to have time to refactor it so completely. Hopefully it will serve as inspiration to someone else but as I'm not using this any more, my time is limited to work on it.

[scottstensland]

I have this repo running inside Docker using Kubernetes to run on Amazon AWS
and am keen to pitch in to help others reach this possibility ... however my pressing concern
is to port onto Ubuntu 16.04 ... not to jackknife this post but can someone point me to
current state of affairs on any ongoing port to 16.04 ? If not I will share my progress on that port

[tomav]

Hi @scottstensland, thanks for your help on Kubernetes. It could be usefull to have a Wiki page on that topic for docker-mailserver.
Regarding 16.04, we can probably upgrade without major issue. Want to contribute on that point?

[tyranron]

@tomav I played recently a bit more with latest (2.1) image version and Kubernetes and finally stuck in situation that cannot run this image version at all. The problem is in hostname detection.

If I specify in PodSpec data like:

subdomain: server
hostname: docker

The result is

/ # hostname
docker
/ # hostname -f
docker.server.mail.svc.cluster.local
/ # hostname -d
server.mail.svc.cluster.local

So image does not run as it requires dot in results of hostname command.

And I am unable to specify subdomain or hostname in PodSpec too, because Kubernetes requires [a-z0-9]([-a-z0-9]*[a-z0-9]) matching:

The Deployment "server" is invalid: spec.template.spec.hostname: Invalid value: "main.docker": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc')

1) The simplest way to resolve this is to use $(hostname -f) instead of $(hostname) in start-mailserver.sh. But I don't know if it will be correct for rest part of configuration. May it be okay?

2) The longer way is to complete #390, which requires detailed testing of each service inside image and may result in configuration complication (as far as many configurations uses hostname command result implicitly and by default).

[tomav]

Hi @tyranron, thanks for the test. The option 1 will be OK if you want to submit a patch (it won't break test I think)

[styxlab]

Thanks for this project and for providing some support for Kubernetes. I'm currently running docker-mailserver on Kubernetes but had to use the image provided by @iMartyn in order to get it working. So, I'd love to see the changes of #390 incorporated into the official project.

I think that you should consider option 2. (#390) more seriously. The hostname is not well suited for Kubernetes and for a clustered environment in general. The Kubernetes way of accessing endpoints is always through a DNS routable name which is most easily done through environment variables. Using the subdomain/hostname properties are more a workaround than a clever solution.

Going for option 2 will save you a lot of headaches in the long run. Just search some of the "hostname" discussions on Kubernetes, if you are not convinced.

[tyranron]

@styxlab just use subdomain without hostname. It will make hostname -f command to return Pod's FQDN and that FQDN won't be static (as we don't specify hostname). No headache.

It's okay to have option 2 as it allows to specify domain you want (like mail.example.com) in mail headers, as currently Pod's FQDN is used there. But this is not something critical.

There are some major problems like placing docker-mailserver behind ingress controllers. This requires PROXY protocol usage between ingress controller and docker-mailserver to preserve real clients IPs. Postfix works okay with HAProxy, however there is no proper HAProxy ingress controller. For Nginx ingress controller things can be more complicate.
Currently working on it.

[styxlab]

I have the standard nginx ingress controller running in front of docker-mailsever without any issues, so I do not understand the problem - at least if you have the option of setting the hostname with env variables. As stated before, I'm using @iMartyn 's image where I can use the env variable OVERRIDE_HOSTNAME=mail.example.com.

[tomav]

@tyranron could you create a Wiki page regarding Kubernetes when you have time?

@styxlab no problem to merge @iMartyn's PR if:

• really needed
• conflicts resolved

Feel free to contribute

I just reopened https://github.com/tomav/docker-mailserver/pull/390

[tyranron]

@styxlab did you check what client IPs you receive in docker-mailserver logs?

@tomav sure, it's on my TODO list. I didn't forget about it.

[styxlab]

@tyranron I'm seeing only cluster IPs in docker-mailserver logs, not external IPs. I think this is a limitation of Kubernetes nginx ingress controller. Currently, external IP forwarding only works on http/https endpoints.

[iMartyn]

The reason I'm not going to do this is it was a really simple pull request before the script was majorly refactored.
If it were a simple "resolve conflicts" request, I would, but I don't have the time on a project I don't use any more to understand all the changes that were made to that script.
In my not-so-humble opinion, the person who re-wrote the script should be able to trivially add this by looking at my actual change, otherwise it's gonna be a from-scratch effort by the next person.

[styxlab]

Isn't #484 the solution we are looking forward to?

[tomav]

@iMartyn yep, and I'm this person who rewrote the startup script but had not time/usage of that PR (especially to fill in a lack of kubernetes stack).

And you're right @iMartyn, it would have been easier, but the point was not clear when some users like @tyranron make it work on Kubernetes (and he's not the only one). So merging code just to merge code without discussing the different usages is not how I want things to be done in this project.

Now, if there's a real need for that part of code, as I said earlier in this thread, I'm open to integrate it has soon conflicts are resolved, AND test added as mentioned in contribution guidelines. This was missing in your initial PR.

Thanks to @kcrawley, @iMartyn's PR will be merged soon (I already commented this PR this morning).

[tyranron]

@styxlab

I'm seeing only cluster IPs in docker-mailserver logs, not external IPs.

Yup. That's exactly the issue I was referring to. It's not limitation of Kubernetes ingress controller, it's rather lack of required options. This can happen in any environment when you place Postfix behind reverse-proxy, not only Kubernetes. It can be solved via using PROXY protocol.

There is issue for adding PROXY protocol usage option in Nginx Ingress Controller.
Also, some work maybe required from Postfix side.

[tomav]

@tyranron if you have a feedback to provide regarding #484, do not hesitate

[ghost]

I concatenated @iMartyn yaml & the deployment + service is created; however I have no clue where to go from here to test/configure this properly. Can someone help me through this? I have 5 heavy bare-metal servers on a custom pacemaker/kubernetes getup & need mail ASAP. I can drop a few bones for the efforts.

[scottstensland]

@webeindustry shoot me an email ... I have a send email only container running on AWS using kubernetes ... [email protected]

[tyranron]

@webeindustry

To test it you just need to assure that following common scenarios run well:

• You can connect from local MUA (Mail User Agent) to your server via IMAP (and POP3 if you require).
• You can sent emails from local MUA via SMTP (on authorization port 587).
• Server is able to receive emails on 25 port.

To configure it you should mount into container all required configuration files using Kubernetes ConfigMaps and Secrets. And specify desired env vars in Deployment spec.

There is no much difference in using this image inside/outside Kubernetes. The steps you perform to get up this image with Docker Compose are just same to get up this image in Kubernetes. And instead of regular files Kubernetes abstractions are used (ConfigMaps, Secrets, etc).

The only major concept to understand is how you are exposing your mailserver outside the cluster. There are some ways, not only one.
The way provided by @iMartyn is to bind mailserver Service to concrete external IP.
But this can be done also with:

• using Port Proxy
• direct binding Pod to concrete Node and using hostNetwork: true
• using Ingresses and Ingress Controllers (like Nginx)

Image configuration depends on what way for exposing you choose.

[ghost]

Think I'm getting closer :) Thanks for the replies so far!

I am currently using @iMartyn repo, is this necessary or should I move to this main repo for improvements? I don't know much about configmaps or secrets yet, they seem to be "the kube way" but not necessary. This is my current YAML. Does this appear in order?

https://pastebin.com/ztfeTWWR

running kubectl create -f mail-server.yaml successfully creates deployment + service. From here would I need to create a config file & place inside of /storage/gv0/mail/config ? Or is this generated for me?

Looking inside /tmp/docker-mailserver I see nothing, as well on host at /storage/gv0/mail/config nada. I should create this config in some manner?

I stumbled on the wiki:
https://github.com/tomav/docker-mailserver/wiki/Configure-Accounts

Will create config files & place inside /storage/gv0/mail/config try to see how far that gets me.
Thanks again!

I created an admin acct, DKIM key, the record. Went on to create MX records, SPF records, A record... never could get my device to connect. Not sure what I'm missing. I see both SPF record & DKIM key are set from https://www.mail-tester.com/spf-dkim-check.

I've attempted to use sendmail to a mail-tester.com address but it's not being received. Hmmm.

[tomav]

@webeindustry do you see something in the logs?
If yes, it will be a good lead to find the answer. Otherwise, you probably don't reach the server.

[ghost]

@tomav Hi @tyranron Helped me sort this out :heart: There were more than a couple of things that needed reworking, but we got it settled.

[kachkaev]

Hi guys,

Just curious: would anyone be interested in summarizing their experience of using this project in a Kubernetes cluster in a form of a blog post or a wiki page? I've been enjoying using docker-mailserver for about a year now, but am looking into upgrading my environment to k8s at some point. Pretty sure I'm not alone among 1.5K github stargazers! :–)

[tomav]

A Wiki page would be perfect. @tyranron @webeindustry ?

[tomav]

Hi guys, anybody has 10min to explain how things work with Kubernetes? Thanks

[tyranron]

@tomav I've written some wiki doc recently, but I'm unable to push it. Would you be so kind to add me to project collaborators temporary?

[tomav]

Done! Thanks @tyranron

[tyranron]

@tomav done! Here it is. I hope that information will help someone.

It's sad, but I don't see "just deploy and use" solution for Kubernetes. Things hardly depend on concrete cluster configuration and constructing Helm chart for mailserver is a huge pain.

[tomav]

Thanks @tyranron!

[kachkaev]

This is terrific, thanks @tyranron!

Would you be interested in making your recipe an official helm chart to turn docker-mailserve into a one-liner on any k8s cluster? That would just rock!

I have quite little experience with help and k8s as such, but after installing wordpress and a couple of other things with helm, I can't imagine any better setup workflow!

[tyranron]

@kachkaev Helm chart is a great idea, but as I referred above (and in wiki page) is non-trivial and requires decisions that come in price.
What exact way of exposing mailserver should we use, and why?

The problem is not about Helm, but more about Kubernetes ecosystem nowadays. Exposing HTTP services in HA, manageable and scalable manner in Kubernetes is easy, while TCP/UDP services is not.

[kachkaev]

I guess it's technically possible to make a chart so that it includes all options, one of which being default and others being available by tweaking the configuration. All the yamls you've nicely crafted would be included at once, just having something like {{- if eq .Values.mode "option_1" -}}...{{- end }} in various places (like in minio or other charts).

Instead of following the instructions by hand, people would just be able to wave a magic wand after providing helm a couple of settings and that's it :–) The docs will not harder as README.md for wordpress.

It's probable that k8s will implement something cool for dealing with TCP traffic in future and then it'll be just a matter of changing the default mode in the chart. Your chart might be already used by many by that time already, because a mailserver is one of the most common services that people want to have on top of the standard web stuff.

[iMartyn]

Whilst I'm not active in this project, there are a couple of things to add
to this thread, that others may want to pick up :

K8SNIff is a TLS TCP SNI ingress, so it can indeed do the tcp HA is
possible. I'd always use traefik in front of the web interfaces and I
wrote a little container to pull the certs from it into the imap server of
my setup (I like mail.mydomain not imap.mydomain and webmail.mydomain).

Another project to look at is linkerd-tcp which is also a TCP ingress from
the great folks at linkerd.

On 28 June 2017 at 20:18, Alexander Kachkaev [e-mail]
wrote:

I guess it's technically possible to make a chart so that it includes all
options, one of which being default and others being available by tweaking
the configuration. All the yamls you've nicely crafted would be included at
once, just having something like {{- if eq .Values.mode "option_1"
-}}...{{- end }} in various places (like in minio
https://github.com/kubernetes/charts/blob/0b1692af04f1b1d22fa1c21e42823c38e160f79b/stable/minio/templates/minio_pvc.yaml
or other charts).

Instead of following the instructions by hand, people would just be able
to wave a magic wand after providing helm a couple of settings and that's
it :–) The docs will not harder as README.md for wordpress
https://github.com/kubernetes/charts/tree/master/stable/wordpress.

It's probable that k8s will implement something cool for dealing with TCP
traffic in future and then it'll be just a matter of changing the default
mode in the chart. Your chart might be already used by many by that time
already, because a mailserver is one of the most common services that
people want to have on top of the standard web stuff.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tomav/docker-mailserver/issues/407#issuecomment-311760304,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA6xXGrkai170leC9JAlIxES0U8BN5Kkks5sIqbvgaJpZM4LQpjl
.

[tyranron]

@iMartyn thank you for pointing to those projects. They really rock! However, looking through I didn't found mention about PROXY protocol implemented. This one is required to preserve real client IP for incoming mails SPF checks actually to work.
Can you share how is problem with SPF checks is solved on your side?