Mailcow-dockerized: Gitlab SMTP Integration

Created on 10 Sep 2019 · 30Comments · Source: mailcow/mailcow-dockerized

Prior to placing the issue, please check following: (fill out each checkbox with a X once done)

[x] I understand that not following below instructions might result in immediate closing and deletion of my issue.
[x] I have understood that answers are voluntary and community-driven, and not commercial support.
[x] I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description of the bug:
First: I am not sure if this is a gitlab or a mailcow Bug. I am trying to find people, who also use gitlab with an linked mailcow mail server.

To my issue: I am trying to configure the SMTP Mail service for my docker gitlab instance with the following documentation: https://docs.gitlab.com/omnibus/settings/smtp.html (See section mailcow).
So I copied the config options to my gitlab.rb config file and edited them that they fit my mailcow installation (I anonymized the sensible output with variables):

gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = $MAILCOW_HOSTNAME
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "gitlab@my_mail_address.de"
gitlab_rails['smtp_password'] = "my_password"
gitlab_rails['smtp_domain'] = $MAILCOW_HOSTNAME
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'none'

So when I try to send an email now (I tried different situations to verify it's not an issue with one function of gitlab) I get an 500/'Something went wrong' error code.
The docker logs of my gitlab instance:

==> /var/log/gitlab/gitlab-rails/production.log <==
Sent mail to my_user_address@my_domain.de (30008.4ms)
Completed 500 Internal Server Error in 30942ms (ActiveRecord: 40.5ms)

Net::OpenTimeout (execution expired):

app/services/emails/confirm_service.rb:6:in `execute'
app/controllers/profiles/emails_controller.rb:30:in `resend_confirmation_instructions'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Following logs are in the dovecot container of the mailcow instance:

Sep 10 18:46:39 mail dovecot: lmtp(32483): Connect from 172.22.1.6
Sep 10 18:46:39 mail dovecot: lmtp(32483): Disconnect from 172.22.1.6: Client has quit the connection (state=READY)
Sep 10 18:46:39 mail dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=172.22.1.6, lip=172.22.1.250, TLS, TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Sep 10 18:46:39 mail dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=172.22.1.6, lip=172.22.1.250
Sep 10 18:46:39 mail dovecot: managesieve-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=172.22.1.6, lip=172.22.1.250
Sep 10 18:47:01 mail CRON[32513]: pam_unix(cron:session): session opened for user vmail by (uid=0)
Sep 10 18:47:01 mail CRON[32512]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 10 18:47:01 mail CRON[32515]: (vmail) CMD (/usr/local/bin/trim_logs.sh >> /dev/console 2>&1)
Sep 10 18:47:01 mail CRON[32516]: (root) CMD ( /usr/local/bin/imapsync_cron.pl 2>&1 | /usr/bin/logger)
Sep 10 18:47:01 mail CRON[32513]: pam_unix(cron:session): session closed for user vmail
Sep 10 18:47:01 mail CRON[32512]: pam_unix(cron:session): session closed for user root
Sep 10 18:47:28 mail dovecot: lmtp(32483): Connect from 172.22.1.6
Sep 10 18:47:28 mail dovecot: lmtp(32483): Disconnect from 172.22.1.6: Client has quit the connection (state=READY)
Sep 10 18:47:28 mail dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=172.22.1.6, lip=172.22.1.250, TLS, TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Sep 10 18:47:28 mail dovecot: imap-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=172.22.1.6, lip=172.22.1.250
Sep 10 18:47:28 mail dovecot: managesieve-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=172.22.1.6, lip=172.22.1.250

If I understand the logs right the gitlab instance is not correctly authenticating with the mailcow server? Maybe this could be the information for me to isolate the error wether it is the gitlab instance or mailcow.

__I have tried or I do...__

[x] Try different settings in the gitlab.rb config file
[x] Tried these debugging tips: https://docs.gitlab.com/ee/administration/troubleshooting/debug.html

My hope of this issue is, that anybody who uses gitlab and mailcow together successfully, can share his configuration with me or finds the error.
Thanks in advance!

System information
Further information (where applicable):

| Question | Answer |
| --- | --- |
| My operating system | Ubuntu 18.04 |
| Is Apparmor, SELinux or similar active? | NO |
| Virtualization technlogy (KVM, VMware, Xen, etc) | Proxmox |
| Server/VM specifications (Memory, CPU Cores) | 8 cores, 30GB RAM |
| Docker Version (docker version) | 19.03.1 |
| Docker-Compose Version (docker-compose version) | 1.24.1 |
| Reverse proxy (custom solution) | NO |

Source

mkapra

All 30 comments

Hi, you need to post your postfix-mailcow logs. :) Are both dockerized services on the same host?

andryyy on 10 Sep 2019

I just set up a Mailcow SMTP with my Gitlab and it works fine. Gitlab and Mailcow are on different VMs with different IPs.

Braintelligence on 10 Sep 2019

He probably just needs to connect both Docker networks and let them 'see' each other.

andryyy on 10 Sep 2019

@Braintelligence Is your SMTP configuration the same or do you have extra options?

mkapra on 10 Sep 2019

@andryyy Yes, both are on the same server.
I looked at the postfix logs but there aren't any logs from a gitlab connection :(
Maybe I sould wait for @Braintelligence 's answer of his configuration.

mkapra on 10 Sep 2019

No, there are no connections then. You need to edit the compose file and add the mailcow network to your GitLab stack.

andryyy on 10 Sep 2019

@andryyy

He probably just needs to connect both Docker networks and let them 'see' each other.

It already worked in past versions. After an update of mailcow (I guess) the mail function 'passed' away. I never had it in one network.

mkapra on 10 Sep 2019

My config looks different. It is identical with the config that Gitlab describes in their docs.
SMTP tls false looks fishy for example.

Braintelligence on 10 Sep 2019

@Braintelligence Is it possible that you send me your anonymized configuration please? My configuration is from the original gitlab page (should be updated if your configuration should work on my machine *with my data of course)

mkapra on 10 Sep 2019

Your config is not identical to the docs. It doesn't even mention the setting

gitlab_rails['smtp_tls'] = false

there. You're also missing the setting of the from_mail - mailcow doesn't let you send from any mail address with the provided user, only those that are viable aliases.

I'm already in bed and can't provide my config now.
I literally copypasted the Gitlab settings from the current docs right after I read your issue.

Braintelligence on 10 Sep 2019

@Braintelligence i have the from_mail option but accidently not mentioned on this issue. It's on top of my config file.

The smtp_tls option is listed here:
https://docs.gitlab.com/omnibus/settings/smtp.html#mailcow

mkapra on 10 Sep 2019

Wow, I didn't even try to check if there were settings for mailcow o_O.
I just used the example settings at the top with 587 as port.

Braintelligence on 10 Sep 2019

@Braintelligence okay i will try that thank you!
what do you have for the openssl_verify_mode? Do you know that eventually? :)

mkapra on 10 Sep 2019

Same as example doc settings:

gitlab_rails['smtp_openssl_verify_mode'] = 'peer'

Braintelligence on 10 Sep 2019

@Braintelligence still failing on my environment :(

mkapra on 10 Sep 2019

My config looks like that now:

gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "MAILCOW_HOSTNAME"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "MAILUSER@MAILDOMAIN"
gitlab_rails['smtp_password'] = "PASSWORD"
gitlab_rails['smtp_domain'] = "MAILCOW_HOSTNAME"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'

# If your SMTP server does not like the default 'From: gitlab@localhost' you
# can change the 'From' with this setting.
gitlab_rails['gitlab_email_from'] = 'MAILUSER@MAILDOMAIN'
gitlab_rails['gitlab_email_reply_to'] = 'MAILUSER@MAILDOMAIN'

mkapra on 10 Sep 2019

Any suggestions how to proceed with this problem?

mkapra on 12 Sep 2019

why you're using port 587?
recommendation, use 465 instead.

Port 465

gitlab_rails['gitlab_email_from'] = "[email protected]"
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "mail.example.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "[email protected]"
gitlab_rails['smtp_password'] = "abcdefghi"
gitlab_rails['smtp_domain'] = "mail.example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'none'

Port 587

gitlab_rails['gitlab_email_from'] = "[email protected]"
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "[email protected]"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "[email protected]"
gitlab_rails['smtp_password'] = "abcdefghi"
gitlab_rails['smtp_domain'] = "mail.example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false

scrape out everything else you have regarding SMTP in your config file because those are enough.
it's as simple as copy/paste but only change the email and mail server addresses.

Always good to read: https://docs.gitlab.com/omnibus/settings/smtp.html

Viktova on 12 Sep 2019

@M034B5 As mentioned above I use the settings from the gitlab documentation.

Also with this settings the same problem :(

mkapra on 12 Sep 2019

👎1

They probably cannot talk to each other.

andryyy on 12 Sep 2019

@M034B5 As mentioned above I use the settings from the gitlab documentation.

Also with this settings the same problem :(

you're not using the

gitlab_rails['gitlab_email_from'] = "[email protected]"

please reread and copy paste
also make sure you do "gitlab-ctl reconfigure"

I tested both ports just now on our company's live server lol they're working fine.

Viktova on 12 Sep 2019

@M034B5 My current configuration:

gitlab_rails['gitlab_email_from'] = "gitlab@MAILDOMAIN"
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "MAILDOMAIN"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "gitlab@MAILDOMAIN"
gitlab_rails['smtp_password'] = "PASSWORD"
gitlab_rails['smtp_domain'] = "MAILDOMAIN"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_tls'] = true
gitlab_rails['smtp_openssl_verify_mode'] = 'none'

Also executed gitlab-ctl reconfigure every time.

mkapra on 12 Sep 2019

Open rails console and show me the logs please:

do:

gitlab-rails console

and then type

Notify.test_email('[email protected]', 'Message Subject', 'Message Body').deliver_now

Dont show the docker logs, i want to see rails logs from inside the container

First in order to get into the container you need to execute something like:

docker exec -it <container_name> bash

Viktova on 12 Sep 2019

@M034B5

Notify#test_email: processed outbound mail in 1.8ms
Sent mail to MAILADDRESS (30003.4ms)
Date: Thu, 12 Sep 2019 14:12:09 +0000
From: Gitlab <gitlab@MAILDOMAIN>
Reply-To: Gitlab <gitlab@MAILDOMAIN>
To: MAILADDRESS
Message-ID: <[email protected]>
Subject: Message Test Subject
Mime-Version: 1.0
Content-Type: text/html;
 charset=UTF-8
Content-Transfer-Encoding: 7bit
Auto-Submitted: auto-generated
X-Auto-Response-Suppress: All

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body><p>Message Test Body</p></body></html>

Traceback (most recent call last):
        1: from (irb):1
Net::OpenTimeout (execution expired)

mkapra on 12 Sep 2019

Do you have any firewall rules enabled?
if so please disable them temporarily and retest.
your gitlab instance seems not able to communicate with the mail server on port 587 nor 465 (as you said earlier)

Another thing or two,
1- Please post your firewall rules for the mail server if applicable.
2- the timeout seems also to be caused by fail2ban from mailcow. so try to disable that container or restart it and retest.

Viktova on 12 Sep 2019

@M034B5
Now it is an authentication error. Seems to be a firewall problem. Thanks for your help!

Net::SMTPAuthenticationError (535 5.7.8 Error: authentication failed: UGFzc3dvcmQ6
):

app/services/emails/confirm_service.rb:6:in `execute'
app/controllers/profiles/emails_controller.rb:30:in `resend_confirmation_instructions'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

mkapra on 12 Sep 2019

👍1

@M034B5
With this firewall configuration it is working now.
Thanks for your help!

Status: active

To                         Action      From
--                         ------      ----
80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
22/tcp                     ALLOW       Anywhere
1022                       ALLOW       Anywhere
1194/udp                   ALLOW       Anywhere
465                        ALLOW       Anywhere
587                        ALLOW       Anywhere
25                         ALLOW       Anywhere
993                        ALLOW       Anywhere
143                        ALLOW       Anywhere
80/tcp (v6)                ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)
22/tcp (v6)                ALLOW       Anywhere (v6)
1022 (v6)                  ALLOW       Anywhere (v6)
1194/udp (v6)              ALLOW       Anywhere (v6)
465 (v6)                   ALLOW       Anywhere (v6)
587 (v6)                   ALLOW       Anywhere (v6)
25 (v6)                    ALLOW       Anywhere (v6)
993 (v6)                   ALLOW       Anywhere (v6)
143 (v6)                   ALLOW       Anywhere (v6)

mkapra on 12 Sep 2019

🎉1

@M034B5 What's wrong with using 587? Works fine for me.

Braintelligence on 12 Sep 2019

@M034B5 What's wrong with using 587? Works fine for me.

Port 465 is for smtps
SSL encryption is started automatically before any SMTP level communication.

unless you don't care about encryption and security then 587 is good for you

Viktova on 12 Sep 2019

It's a hard stretch to say I wouldn't care about encryption at all, just because I am alright with a STARTTLS handshake.
gitlab_rails['smtp_enable_starttls_auto'] = true will keep the connection itself encrypted as long as the mailserver on the other hand provides STARTTLS, which Mailcow does.

I concur, though, that it is safer to go with 465.

Braintelligence on 12 Sep 2019

Was this page helpful?

0 / 5 - 0 ratings