Docker-gitlab: SMTP doesn't work and no way to debug

[globalcitizen]

IMHO you really need some sort of test process for SMTP configuration.

In my case I have tried STARTTLS as well as SSL for SMTP and am not getting anywhere.

Outbound messages are supposed to be getting delivered...

Sent mail to [email protected] (60140.2ms)
Sent mail to [email protected] (60130.0ms)
Sent mail to [email protected] (60142.5ms)

... but the emails never arrive, and there appears to be no means to debug the problem.

Testing the very same SMTP credentials using Mozilla Thunderbird, emails are delivered fine to the very same recipients.

[r2evans]

I suggest that gitlab needs some sort of test process for SMTP configuration. If it were implemented in this repo, then all you would be testing would be their test, not the gitlab configuration, since gitlab would still be doing its own thing.

In fact, it's been asked already (twice).

Please realize the difference between issues against the server component (via GitLab issues) and the docker container provided by @sameersbn and friends.

(Now, if you can get it working in another instance of gitlab and identical configuration is broken in this image, then that would be appropriate here.)

[sameersbn]

@globalcitizen who is your email provider, maybe if you share some information, other's with the same provider/configuration may be able to help.

You may also want to look at the sidekiq logs for mail delivery failures.

[globalcitizen]

Thanks for the pointer, unfortunately that file just shows lots of ...

"error_message":"Net::ReadTimeout","error_class":"Net::ReadTimeout"

However, I have full network access from within the container and can ping and telnet to the servers in question.

My provider is a Chinese domestic provider sold through a virtual hosting reseller - bnet.cn - it is unlikely many others are using it.

The only other thing I noticed is that retry_count seems to increment from 0 to 3 before failure.

[xueshanf]

@globalcitizen Have you configured SMTP_PORT explicitly? The default is 587, which might not be the one your email server opens to the world. When you do telnet from within the container, did you use that port?

[globalcitizen]

587 is open on the server in question.

[lenovouser]

@globalcitizen I saw your comment on HN. I am pretty sure this is not a issue with the container / image. Do you mind sharing your environment variables or the docker-compose.yml that you're using? Also what e-mail provider are you using? If you're self hosting your e-mail what does:

docker exec -it gitlab bash
ping <email server hostname>
telnet <email server hostname> <port>

do?

[globalcitizen]

No environment variables. Standard docker-compose.yml with credentials added. Have tried various SMTP and IMAP options. Provider-wise, we are using bnet.cn (apparently a ~popular reseller platform for ISPs in China). Yes, I can ping and telnet to appropriate ports on the server, and I have tested the same credentials using Mozilla Thunderbird and can SMTP and IMAP without issue.

[lenovouser]

What I meant is if you can ping the e-mail server from the container? If yes, what is the output?

[globalcitizen]

host # ./gitlab-shell
root@fdaa1ff1a0fe:/home/git/gitlab# ping smtp.bnet.cn
PING smtp-189ent.cdn.21cn.com (183.61.185.66) 56(84) bytes of data.
^C
--- smtp-189ent.cdn.21cn.com ping statistics ---
15 packets transmitted, 0 received, 100% packet loss, time 14324ms

root@fdaa1ff1a0fe:/home/git/gitlab# ping imap.bnet.cn
PING imap-189ent.cdn.21cn.com (183.61.185.66) 56(84) bytes of data.
^C
--- imap-189ent.cdn.21cn.com ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9252ms

root@fdaa1ff1a0fe:/home/git/gitlab# telnet imap.bnet.cn 993
bash: telnet: command not found
root@fdaa1ff1a0fe:/home/git/gitlab# exit
exit
host # telnet imap.bnet.cn 993
Trying 183.61.185.66...
Connected to imap.bnet.cn.
Escape character is '^]'.





^]close
telnet> close
Connection closed.
host # ping imap.bnet.cn
PING imap-189ent.cdn.21cn.com (183.61.185.66) 56(84) bytes of data.
^C
--- imap-189ent.cdn.21cn.com ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2055ms
host #

ie. It's configured to drop ICMP but telnet works fine from the host machine, connections work fine from the container (because not configuring correctly caused errors which are now resolved, and temporary connectivity issues cause errors to appear in the logs) and there are no issues using the server from other IPs/clients.

[slim12kg]

@globalcitizen how did you finally fix issue in configuration

[globalcitizen]

@slim12kg As far as I can remember there were some relatively undocumented options available which resolved the issue. I am now on a new server hosting its own SMTP with its own appropriate DNS entries for DKIM etc.

[jstdoit]

I am testing with Notify.test_email('[email protected]', 'Message Subject', 'Message Body').deliver_now, and got these errors:

>
Sent mail to [email protected] (60250.3ms)
Date: Thu, 08 Feb 2018 14:23:30 +0530
From: GitLab [e-mail]
Reply-To: GitLab [e-mail]
To: [email protected]
Message-ID: [e-mail]
Subject: Message Subject
Mime-Version: 1.0
Content-Type: text/html;
charset=UTF-8
Content-Transfer-Encoding: 7bit
Auto-Submitted: auto-generated
X-Auto-Response-Suppress: All

Message Body

Net::ReadTimeout: Net::ReadTimeout
from /usr/lib/ruby/2.3.0/net/protocol.rb:176:in rbuf_fill' from /usr/lib/ruby/2.3.0/net/protocol.rb:154:inreaduntil'
from /usr/lib/ruby/2.3.0/net/protocol.rb:164:in readline' from /usr/lib/ruby/2.3.0/net/smtp.rb:955:inrecv_response'
from /usr/lib/ruby/2.3.0/net/smtp.rb:556:in block in do_start' from /usr/lib/ruby/2.3.0/net/smtp.rb:965:incritical'
from /usr/lib/ruby/2.3.0/net/smtp.rb:556:in do_start' from /usr/lib/ruby/2.3.0/net/smtp.rb:521:instart'
from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mail-2.7.0/lib/mail/network/delivery_methods/smtp.rb:109:in start_smtp_session' from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mail-2.7.0/lib/mail/network/delivery_methods/smtp.rb:100:indeliver!'
from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mail-2.7.0/lib/mail/message.rb:2160:in do_delivery' from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mail-2.7.0/lib/mail/message.rb:260:inblock in deliver'
from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionmailer-4.2.10/lib/action_mailer/base.rb:543:in block in deliver_mail' from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:164:inblock in instrument'
from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications/instrumenter.rb:20:in instrument' from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:164:ininstrument'
from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionmailer-4.2.10/lib/action_mailer/base.rb:541:in deliver_mail' from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/mail-2.7.0/lib/mail/message.rb:260:indeliver'
from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/actionmailer-4.2.10/lib/action_mailer/message_delivery.rb:85:in deliver_now' from (irb):1 from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/commands/console.rb:110:instart'
from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/commands/console.rb:9:in start' from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:68:inconsole'
from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/commands/commands_tasks.rb:39:in run_command!' from /home/git/gitlab/vendor/bundle/ruby/2.3.0/gems/railties-4.2.10/lib/rails/commands.rb:17:in from ./bin/rails:9:in require' from ./bin/rails:9:in

'

>

[r2evans]

@jstdoit how did you get to the gitlab console? I can't seem to get gitlab-rails console to work (no gitlab-rails). I found vendor/bundle/ruby/2.4.0/bin/rails, but it fails with can't find gem omniauth-oauth2-generic (>= 0.a) (Gem::GemNotFoundException.

I'd really like to be able to get to the point where I can do Notify.test_email(...).

[emilklindt]

@r2evans I just stumpled upon this issue. Have you tried with the console for source installations, with bundle: bundle exec rails console production?

You might find this page useful: https://docs.gitlab.com/ee/administration/troubleshooting/debug.html

[HackerTao]

mark,

[r2evans]

@emilklindt, I can no longer comment on the success or fixes to this, I switched over to gitlab/gitlab-ce.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity for the last 60 days. It will be closed if no further activity occurs during the next 7 days. Thank you for your contributions.