Mumble: Public server registration not working due to SSL error

[pb-programmer]

Quick disclaimer: I've been running my mumble server for several years and haven't touched it (except: apt upgrade) until I noticed this problem, so outdated configuration/regression is absolutely possible! I also upgraded my server to Ubuntu 20.04 LTS recently (started with 16.04 LTS, upgraded to 18.04 LTS in 2018, now switched to 20.04 LTS) and that might very well be the time when the bug occured first. I personally didn't notice the exact time because murmur works just fine as long as the server is in your "favorite" list.

Steps to Reproduce
On Ubuntu 20.04.01 LTS with the stable PPA version of mumble (1.3.3-1\~ppa1\~focal1) installed and OpenSSL 1.1.1f
_Configuration file:_

• Public server registration is enabled

registerName=DotA2 german [location Karlsruhe]
registerPassword=#############
registerUrl=https://mumble.christoph-mayer.net/
registerHostname=mumble.christoph-mayer.net

• A Let's encrypt certificate is provided

sslCert=/etc/letsencrypt/live/christoph-mayer.net/cert.pem
sslKey=/etc/letsencrypt/live/christoph-mayer.net/privkey.pem
sslCA=/etc/letsencrypt/live/christoph-mayer.net/fullchain.pem

• Some standard TLS settings are provided (added them after I noticed the problem)

sslDHParams=@ffdhe2048
sslCiphers=EECDH+AESGCM:EDH+aRSA+AESGCM:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA

After startup the murmur log looks like this:

<W>2020-11-05 12:40:20.790 SSL: OpenSSL version is 'OpenSSL 1.1.1f  31 Mar 2020'
<W>2020-11-05 12:40:20.802 Initializing settings from /etc/mumble-server.ini (basepath /etc)
<W>2020-11-05 12:40:23.901 MetaParams: TLS cipher preference is "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AE
S128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA"
<W>2020-11-05 12:40:24.213 ServerDB: Opened SQLite database /var/lib/mumble-server/mumble-server.sqlite
<W>2020-11-05 12:40:24.215 ServerDB: Using SQLite's default rollback journal.
<W>2020-11-05 12:40:24.243 DBus registration succeeded
<W>2020-11-05 12:40:24.272 MurmurIce: Endpoint "tcp -h 127.0.0.1 -p 6502 -t 60000" running
<W>2020-11-05 12:40:24.727 Murmur 1.3.3 (1.3.3-1~ppa1~focal1) running on X11: Ubuntu 20.04.1 LTS: Booting servers
<W>2020-11-05 12:40:24.763 1 => Server listening on 0.0.0.0:64738
<W>2020-11-05 12:40:26.066 1 => Announcing server via bonjour
[...]
<W>2020-11-05 12:41:58.285 1 => <2:(-1)> New connection: 52.201.18.254:41916
<W>2020-11-05 12:41:58.290 1 => <2:(-1)> Connection closed: Error during SSL handshake: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol [13]
<W>2020-11-05 12:41:58.487 1 => <3:(-1)> New connection: 52.201.18.254:41918
<W>2020-11-05 12:41:58.500 1 => <3:(-1)> Connection closed: Error during SSL handshake: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol [13]
<W>2020-11-05 12:41:58.604 1 => Registration: DNS name callback mumble.christoph-mayer.net=>82.165.163.205 failed: Source-based callback failed. Server not reachable.

Noticed problems
The DNS callback problem is weird. The domain name and the IP are correct, haven't changed in years and can be resolved just fine (checked several DNS servers all over the world). Another quick test showed the server is reachable with Mumble 1.3.3 (Win10), can be pinged and connected to via openssl. So I think this is just because the registration server can't connect.

Which brings us to real problem: The registration server can't connect

Registration used to work just fine a couple months ago (I don't look into the logs that often) and as a user mumble/mumur just work as intended (except the lack of "public server"-listing).

Further investigation
I started digging around and noticed:
If I connect to my server via openssl with standard settings it works just fine (successful connection):
openssl s_client -connect mumble.christoph-mayer.net:64738
resulting murmur log:

<W>2020-11-05 12:51:05.415 1 => <4:(-1)> New connection: 46.142.168.###:52999
<W>2020-11-05 12:51:12.320 1 => <4:(-1)> Connection closed: The remote host closed the connection [1]

But if I connect to my server via openssl and force TLS1.0, the connection is immediately closed and I get the same error message the registration server produces
openssl s_client -connect mumble.christoph-mayer.net:64738 -tls1
resulting murmur log:

<W>2020-11-05 12:51:22.016 1 => <5:(-1)> New connection: 46.142.168.###:53001
<W>2020-11-05 12:51:22.022 1 => <5:(-1)> Connection closed: Error during SSL handshake: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol [13]
<W>2020-11-05 12:51:22.959 Ignored OpenSSL error 140E0197 for 0x55b09f2da5b0

So the registration server probably only tries to connect with TLS 1.0. Is there any way to allow murmur under Ubuntu 20.04 LTS to accept TLS 1.0 connections or is it possible to upgrade the registration server to offer newer TLS versions?

[Krzmbrzl]

The registration server can't connect

What do you mean by that?

I think I still didn't understand when this error occurs. Is someone actually trying to connect to your server (a client) or does the error appear in the log simply after having started the server? And is the server still visible in the public server list?

[Popkornium18]

I also couldn't get registration to work and I have configured the cipherlist. After reading this bug report I tried it with the default cipherlist and sure enough: Registration: New registration successful

So I can confirm this.

[Krzmbrzl]

Not a crypto-expert here, but just to be sure: The problem is not the cipher-list but the handling of a custom list? Aka: Does it matter which ciphers you actually list there?

[Popkornium18]

I think it's just a cipher mismatch when the murmur server tries to contact the registration server with a limited cipher list. I was using sslCiphers=EECDH+AESGCM:EDH+AESGCM and the registration server does not support these (I guess - All these cipher names can get really confusing).

The registration server also seems to be lacking TLS1.3 support.
https://www.ssllabs.com/ssltest/analyze.html?d=publist-registration.mumble.info

[pb-programmer]

@Krzmbrzl I start murmur, it tries to register itself as a public server but when the registration server tries to connect to my server the connection is closed.
So my server does not appear on the public server list.

If you know the url/ip everything works just fine. The log I showed appears all the time. It's the mumble registration server trying to connect to my server (but failing) to verify it's live.

@Popkornium18 Okay, I out commented everything cipher related from my mumble-server.ini and restarted murmurd.
But sadly the same error still occurs

<W>2020-11-05 17:25:43.074 SSL: OpenSSL version is 'OpenSSL 1.1.1f  31 Mar 2020'
<W>2020-11-05 17:25:43.074 Initializing settings from /etc/mumble-server.ini (basepath /etc)
<W>2020-11-05 17:25:43.609 MetaParams: TLS cipher preference is "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA"
<W>2020-11-05 17:25:43.648 ServerDB: Opened SQLite database /var/lib/mumble-server/mumble-server.sqlite
<W>2020-11-05 17:25:43.649 ServerDB: Using SQLite's default rollback journal.
<W>2020-11-05 17:25:43.660 DBus registration succeeded
<W>2020-11-05 17:25:43.661 MurmurIce: Endpoint "tcp -h 127.0.0.1 -p 6502 -t 60000" running
<W>2020-11-05 17:25:43.721 Murmur 1.3.3 (1.3.3-1~ppa1~focal1) running on X11: Ubuntu 20.04.1 LTS: Booting servers
<W>2020-11-05 17:25:43.737 1 => Server listening on 0.0.0.0:64738
<W>2020-11-05 17:25:44.287 1 => Announcing server via bonjour
<W>2020-11-05 17:25:46.134 1 => <1:(-1)> New connection: 46.142.168.###:57410
<W>2020-11-05 17:25:46.237 1 => <1:(-1)> Client version 1.3.3 (Win: 1.3.3)
<W>2020-11-05 17:25:46.246 1 => Starting voice thread
<W>2020-11-05 17:25:46.253 1 => CELT codec switch ffffffff80000010 0 (prefer ffffffff80000010) (Opus 1)
<W>2020-11-05 17:25:46.263 1 => <1:###(3)> Authenticated
<W>2020-11-05 17:28:02.964 1 => <2:(-1)> New connection: 52.201.18.254:34284
<W>2020-11-05 17:28:02.973 1 => <2:(-1)> Connection closed: Error during SSL handshake: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol [13]
<W>2020-11-05 17:28:03.174 1 => <3:(-1)> New connection: 52.201.18.254:34286
<W>2020-11-05 17:28:03.180 1 => <3:(-1)> Connection closed: Error during SSL handshake: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol [13]
<W>2020-11-05 17:28:03.282 1 => Registration: DNS name callback mumble.christoph-mayer.net=>82.165.163.205 failed: Source-based callback failed. Server not reachable.
<W>2020-11-05 17:28:06.097 Ignored OpenSSL error 140E0197 for 0x558e73ac7060

My config is attached so you can check I didn't make a stupid mistake in there
mumble-server.txt

[Popkornium18]

So if I comment out sslCiphers murmur uses ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA as the default for me, which is different than your default, which makes sense, since you use Ubuntu 20.04 and I use Debian 10.

Are you able to register if you explicitly set the ciphers that work for me?

[pb-programmer]

@Popkornium18 I'm not good with SSL cipher names and what to use where, so correct me if I'm wrong!
I set my mumble-server.ini to:
sslCiphers=ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA

If I put that same string into "openssl cipher $YOURLIST" I get:
TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA

Which matches what murmur prints in it's log:
<W>2020-11-05 18:02:19.987 MetaParams: TLS cipher preference is "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA"

Sadly the registration server still can't connect:

<W>2020-11-05 18:05:14.178 1 => <2:(-1)> New connection: 52.201.18.254:37152
<W>2020-11-05 18:05:14.183 1 => <2:(-1)> Connection closed: Error during SSL handshake: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol [13]
<W>2020-11-05 18:05:14.379 1 => <3:(-1)> New connection: 52.201.18.254:37154
<W>2020-11-05 18:05:14.384 1 => <3:(-1)> Connection closed: Error during SSL handshake: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol [13]
<W>2020-11-05 18:05:14.485 1 => Registration: DNS name callback mumble.christoph-mayer.net=>82.165.163.205 failed: Source-based callback failed. Server not reachable.
<W>2020-11-05 18:05:16.117 Ignored OpenSSL error 140E0197 for 0x562477b4a7a0

EDIT:
Also tried sslCiphers=HIGH:MEDIUM and ended up with this behemoth in the log:
<W>2020-11-05 18:26:55.192 MetaParams: TLS cipher preference is "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:RSA-PSK-AES256-GCM-SHA384:DHE-PSK-AES256-GCM-SHA384:RSA-PSK-CHACHA20-POLY1305:DHE-PSK-CHACHA20-POLY1305:ECDHE-PSK-CHACHA20-POLY1305:AES256-GCM-SHA384:PSK-AES256-GCM-SHA384:PSK-CHACHA20-POLY1305:RSA-PSK-AES128-GCM-SHA256:DHE-PSK-AES128-GCM-SHA256:AES128-GCM-SHA256:PSK-AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:ECDHE-PSK-AES256-CBC-SHA384:ECDHE-PSK-AES256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:RSA-PSK-AES256-CBC-SHA384:DHE-PSK-AES256-CBC-SHA384:RSA-PSK-AES256-CBC-SHA:DHE-PSK-AES256-CBC-SHA:AES256-SHA:PSK-AES256-CBC-SHA384:PSK-AES256-CBC-SHA:ECDHE-PSK-AES128-CBC-SHA256:ECDHE-PSK-AES128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:RSA-PSK-AES128-CBC-SHA256:DHE-PSK-AES128-CBC-SHA256:RSA-PSK-AES128-CBC-SHA:DHE-PSK-AES128-CBC-SHA:AES128-SHA:PSK-AES128-CBC-SHA256:PSK-AES128-CBC-SHA"

Sadly the registration still does not work, so I think it's a TLS version mismatch, not a cipher mismatch

[Krzmbrzl]

The public server registration implementation is ancient. Thus it is definitely possible that it is just outdated

[davidebeatrici]

We received quite a few reports about this, it's almost certainly due to OpenSSL not being up-to-date on our server.

[pb-programmer]

Just as a quick follow up: With Ubuntu 20.04 LTS all system TLS libraries (OpenSSL, GnuTLS, NSS) have been set to default to security level 2, which in turn means TLS 1.2 or TLS 1.3 is enforced (+corresponding ciphers) for secure connections. TLS 1.0, TLS 1.1 or any SSL connections are instantly dropped.
See here: https://discourse.ubuntu.com/t/default-to-tls-v1-2-in-all-tls-libraries-in-20-04-lts/12464
Apparently there has also been a blog post about this (from May 20th 2020), but it has since been withdrawn and you can't access it anymore.

Possible workaround

I personally think updating the public registration server would be the way to go, but I don't know if or when this is going to happen. Since a public mumble server without public registration is kind of pointless, I've got a workaround for anyone using murmur on Ubuntu 20.04 LTS for the time being.

This is NOT recommended, do this at your own risk and only if you are sure you fully understand all the drawbacks and possible security vulnerability this change brings!

In your mumble-server.ini change the cipher line to:
sslCiphers=HIGH:MEDIUM:@SECLEVEL=1
Follow up with a sudo systemctl daemon-reload and/or a sudo systemctl restart mumble-server and you should be good to go.

You can verify the change worked when with openssl s_client -connect localhost:64738 -cipher "MEDIUM:HIGH:@SECLEVEL=1" -tls1 a connection is established and stays open. Or just wait a couple of minutes and look if there has been a
(...) => Registration: Registration updated in your mumble-server.log

[davidebeatrici]

Our server now runs Ubuntu 20.04.

Please let us whether the issue is fixed.

[pb-programmer]

Wow that was fast, thanks a lot. I reverted my sslCipher back to default (from mumble wiki)
sslCiphers=EECDH+AESGCM:EDH+aRSA+AESGCM:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA

Looks like the public registration server can successfully connect to murmur now, sadly there still seems to be something wrong:

<W>2020-11-15 13:13:30.678 1 => <1:(-1)> New connection: 52.201.18.254:33016
<W>2020-11-15 13:13:30.684 1 => <1:(-1)> Connection closed: The remote host closed the connection [1]
<W>2020-11-15 13:13:30.751 1 => <2:(-1)> New connection: 52.201.18.254:33018
<W>2020-11-15 13:13:30.756 1 => Registration: DNS name callback mumble.christoph-mayer.net=>82.165.163.205 failed: Source-based callback failed. Server not reachable.
<W>2020-11-15 13:13:30.760 1 => <2:(-1)> Connection closed: The remote host closed the connection [1]

This seems like a separate issue though, so I'll investigate a bit to make sure the fault is not on my side...

[Popkornium18]

Same for me

[davidebeatrici]

Thank you for your report!

Two different issues have just been fixed:

1. Registration script attempting to connect using TLSv1 for Murmur 1.2.0+ and SSLv3 for older versions.
   The logic is now completely removed and we let OpenSSL negotiate the best cipher. The minimum TLS version is 1.2.
2. No IPv6 support due to publist-registration.mumble.info being an alias to the VPS' IPv4 hostname.
   An AAAA record has been added to mumble.info and publist-registration.mumble.info is now an alias to mumble.info.

The last issue appears to be #4184, which we will fix as soon as possible

[pb-programmer]

Thank you very much for addressing the issue. I just checked my logs and without any further changes to the config and without a restart registration is now working as expected.

<W>2020-11-16 04:46:42.111 1 => <38:(-1)> New connection: 52.201.18.254:54176
<W>2020-11-16 04:46:42.116 1 => <38:(-1)> Connection closed: The remote host closed the connection [1]
<W>2020-11-16 04:46:42.204 1 => Registration: DNS name callback mumble.christoph-mayer.net=>82.165.163.205 failed: Source-based callback failed. Server not reachable.
[...]
<W>2020-11-16 07:50:18.123 1 => <41:(-1)> New connection: 52.201.18.254:37772
<W>2020-11-16 07:50:18.225 1 => <41:(-1)> Connection closed: The TLS/SSL connection has been closed [1]
<W>2020-11-16 07:50:18.230 1 => Registration: Registration updated

[davidebeatrici]

No problem, thank you very much for testing!