Synapse: Synapse uses TLS1.0 for smtp which is rejected by some mail servers

[gjabell]

Description

Requesting a password reset from a brand-new Synapse installation returns a 500 error, with the error twisted.mail._except.SMTPConnectError: Unable to connect to server.

Steps to reproduce

• On a vanilla homeserver, add the following configuration to homeserver.yaml:

email:
  enable_notifs: false
  smtp_host: [hostname or ip]
  smtp_port: 587
  smtp_user: [username]
  smtp_pass: [password]
  notif_from: "Your friendly %(app)s Home Server <[email]>"
  app_name: Matrix

• Restart synapse to apply changes
• Using riot, change the homeserver url and then select "Set a new password"
• Enter the valid email address and a new password
• Select "Send Reset Email"

After the last step, the server will respond with a 500 error, and the following will be displayed in synapse's log:

Oct 17 15:19:00 [hostname] synapse[11936]: synapse.handlers.identity: [POST-49] Error sending threepid validation email to [email]
                                                Traceback (most recent call last):
                                                  File "/nix/store/1al2bnj8f2y66jxmzhi00aw3a7wp1jgw-matrix-synapse-1.4.0/lib/python3.7/site-packages/synapse/handlers/identity.py", line 347, in send_threepid_validation
                                                    yield send_email_func(email_address, token, client_secret, session_id)
                                                twisted.mail._except.SMTPConnectError: Unable to connect to server.

And this is displayed in the postfix log of the receiving server:

Oct 17 15:19:00 [hostname] postfix/smtpd[2546]: connect from unknown[ip]
Oct 17 15:19:00 [hostname] postfix/smtpd[2546]: SSL_accept error from unknown[ip]: -1
Oct 17 15:19:00 [hostname] postfix/smtpd[2546]: warning: TLS library problem: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol:ssl/statem/statem_srvr.c:1661:
Oct 17 15:19:00 [hostname] postfix/smtpd[2546]: lost connection after STARTTLS from unknown[ip]
Oct 17 15:19:00 [hostname] postfix/smtpd[2546]: disconnect from unknown[ip] ehlo=1 starttls=0/1 commands=1/2

I've tested this configuration with both require_transport_security: false and require_transport_security: true. Also worth mentioning that the username / password are correct, as logging into the mail server from a mail program and sending a test email from there works fine.

Version information

New personal homeserver running synapse.

• Version: 1.4.0

• Install method: Package Manager
  

• Platform: NixOS running on Hetzner Cloud VM for both Matrix and mail server
  

[gjabell]

After some more testing, this appears to be an issue with Synapse (or rather Twisted) not using an up-to-date TLS protocol when communicating with the mail server. My server is set to not accept TLSv1 or SSLv2 & 3, which explains the unsupported protocol:ssl error above. Is there a way to force Synapse/Twisted to use newer TLS protocols?

[madpsy]

I second this - seeing exactly the same issue.

Interestingly, require_transport_security doesn't seem to do anything. When set to true or false it still attemps STARTTLS which is obviously wrong.

[fadenb]

I am experiencing the exact same issue.

[thereapman]

same here.

[ghost]

Yes, I have the same behavior here as well.

[sbiberhofer]

While looking at this bug w/ @henry-nicolas yesterday, we did some testing which might be helpful in pinpointing the bug:

• It seems like synapse's support of STARTTLS is hardcoded to TLSv1.0. Regardless of federation_client_minimum_tls_version, the TLS Client Hello following STARTTLS in an smtp connection is using TLSv1.0 and is thus rejected by any mailserver not supporting such an old TLS version.
• As @madpsy already noticed, twisted is using STARTTLS in an opportunistic manner by default. The only thing that require_transport_security controls is whether or not STARTTLS is a hard requirement. This isn't "wrong" per se but it would be nice to have more control over this behaviour. Additionally, synapse doesn't support smtps (i.e. smtp wrapped in TLS), which should probably be at least mentioned in the documentation.
• I'm not sure this is actually synapse's fault. It might well be a bug in twisted since - as far as I can tell - synapse simply relies on twisted's sendmail() functionality to "do the right thing".

[plamenh]

+1

[vmario89]

same problem here. I am running Plesk which controls Postfix/Dovecot. The systems are configured to only use TLS 1.2 or higher

the log output:

2019-11-27 12:55:05,982 - synapse.handlers.identity - 357 - ERROR - POST-1239- Error sending threepid validation email to
Traceback (most recent call last):
File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/handlers/identity.py", line 354, in send_threepid_validation
yield send_email_func(email_address, token, client_secret, session_id)
twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion.

hopefully it belongs to this issue. Some releases before everything worked fine

[vmario89]

i was able to validate that its a TLS problem. i re-enabled old TLS 1.0 and 1.1 to test:
plesk bin server_pref -u -ssl-protocols 'TLSv1 TLSv1.1 TLSv1.2'

that makes it work directly after enabling TLS 1.0

[Chatcloud]

Same here on ubuntu server 18.04. Please provide a solution.

2019-12-02 10:30:12,234 - synapse.http.site - 203 - WARNING - POST-14- Error processing request <XForwardedForRequest at 0x7f76febbcfd0 method='POST' uri='/_matrix/client/r0/account/3pid/email/requestToken' clientproto='HTTP/1.1' site=8008>: <class 'twisted.internet.error.ConnectionDone'> Connection was closed cleanly.
2019-12-02 10:30:12,325 - twisted - 172 - INFO - - SMTP Client retrying server. Retry: 5

[MisterAlainDev]

the problem was that twisted mail smtp wrapper does not call SSL connect at all (inside synapse virtual env > /twisted/mail/smtp.py)
in this python file

1. add this import from twisted.internet.ssl import optionsForClientTLS
2. inside the class ESMTPSender(SenderMixin, ESMTPClient) > fix the minimal tls version (for example : context.method = ssl.SSL.TLSv1_1_METHOD)
3. inside the def sendmail > replace the line connectTCP with

if (requireTransportSecurity):
    contextFactory = optionsForClientTLS(smtphost)
    connector = reactor.connectSSL(smtphost, port, factory, contextFactory)

else:  
    connector = reactor.connectTCP(smtphost, port, factory)

[schwukas]

It seems that @sbiberhofer is right with the hardcoded part.
I noticed that in line 2038 the smtp module has the context method overwritten to a lower TLS version.
The context factory which is called by the smtp part actually provides the more modern TLS versions 2 and 3. However it states to also allow for TLSv1, which doesn't explain why the below works.

At any rate, commenting line 2038 in the smtp module where the method gets overwritten to TLSv1 worked for me.
I don't know if this has other implications. It might be there for a reason. I will try to find someone in the #twisted matrix room to ask.

I have created a ticket on twisted's issue tracker, that you can find here.

[kaiyou]

For what is worth, here is a temporary fix on our side, thank you so much for pinpointing the actual culprit in Twisted sources: https://forge.tedomum.net/tedomum/synapse/blob/aac748e3720e001a2fc9e42ef2add49ce815443e/docker/Dockerfile#L57

If required, I can PR this ugly jewel.

[babolivier]

Since I don't see it mentioned here, here's the ticket on Twisted's side for tracking: https://twistedmatrix.com/trac/ticket/9740

[mmilata]

Submitted a patch to Twisted. If it's not accepted we can make Synapse use ESMTPSenderFactory (with explicit ~ClientContextFactory~ context returned by optionsForClientTLS) directly instead of the higher-level function sendmail() (with hardcoded TLSv1.0 ClientContextFactory).

[n3m3s1s]

The problem seems to be fixed. Could you update the dependency or is it more complicated to get the fix into 1.12.4 maybe?

[clokep]

@n3m3s1s The fix is not yet in a release version of Twisted. After they've done their next release it should just be a matter of ensuring the version of Twisted used is the latest.

[n3m3s1s]

@clokep Right, sorry. Got a little confused by the 1k branches they have. Looking forward to their next release.

[Neustradamus]

@clokep, @Half-Shot: Any news with this bug?
TLS 1.2 and TLS 1.3 are important!

[Half-Shot]

@Neustradamus We can't fix it until Twisted release a fix. There's no point in asking until then.

[mmilata]

@Neustradamus if you don't want to wait you should be able to rebuild your twisted package to include this patch.

[Neustradamus]

@adiroiban, @ashfall, @cyli, @exarkun, @glyph, @habnabit, @hawkowl, @hynek, @jml, @Julian, @Lukasa, @lvh, @markrwilliams, @mithrandi, @radix, @wallrj, @warner, @wsanchez: Can you create a release with this important fix?

[exarkun]

@Neustradamus That's really obnoxious and unappreciated.

[babolivier]

@Neustradamus Please don't harass people. The Twisted devs already answered your question on Twitter (https://twitter.com/twistedmatrix/status/1256852143425773568) giving you a viable solution if you really can't wait for a release (https://twitter.com/twistedmatrix/status/1256852515586338816), this should be more than enough while waiting for them to release a new version.

To reiterate and elaborate a bit on the advice the Twisted devs gave, here's how you install the patch if you _really really really_ can't wait for a release:

# Activate Synapse's virtualenv
source env/bin/activate
# Uninstall the mainline version of twisted[tls]
pip uninstall twisted[tls]
# Reinstall twisted[tls] from github at the commit provided by the Twisted folks
pip install https://github.com/twisted/twisted/archive/8c251edc95b48d578660343c5de072691ff75e8b.zip#egg=twisted[tls]

Let me just empathise that this isn't a recommended procedure as unreleased patches are known to be less stable than releases.

Note that you'll need to run pip uninstall twisted[tls] && pip install twisted[tls] when the fix is released.

[Neustradamus]

People ask news about this security fix:

• https://twitter.com/CaptainKurtis/status/1292278623110660096