Mumble: [Debian] Murmur can only read certificate if it is file owner

[notinecrafter]

I tried to get murmur (or mumble-server, as the Debian packages so adamantly call it) to use my Let's Encrypt certificates. However, I continuously got the error: "Failed to read /etc/letsencrypt/live/example.com/fullchain.pem".

The Mumble wiki suggests that I have to give mumble-server execute permission in the /etc/letsencrypt/live and /etc/letsencrypt/archive directories. I did this using setfacl, but no change occurred.

I then tried to make a new group called ssl (for exactly this purpose), made mumble-server a member of this group, and set ownership of both directories to root:ssl, but to no avail.

Only when I made mumble-server the owner of (a copy of) the files, as suggested in issue 1949, it managed to successfully read the certificates. However, in a production environment with other things running having mumble be the owner of your certificate files is obviously not entirely desirable.

@janschejbal 's solution to issue 1949, having mumble run with priority, does not solve this issue on my Debian 9 (Stretch) system.

UPDATE: @janschejbal's solution didn't work because dpkg-reconfigure didn't actually change /etc/default/mumble-server. Manually changing /etc/default/mumble-server did solve the issue.

I'm leaving this issue open because it's still unexpected behaviour to say the least, and because of this other issue I encountered with dpkg-reconfigure. If you, as a developer, think this needs to be a separate issue I suggest you make it one.

[mkrautz]

I suppose this would be better suited as a Debian bug?

cc @crknadle

[crknadle]

@mkrautz thanks for the heads-up on this.
I just tested 'dpkg-reconfigure mumble-server' with mumble-server 1.2.18-1. The second Debconf quesiton is what sets MURMUR_USE_CAPABILITIES in /etc/default/mumble-server -- it appears to work correctly, but isn't obvious when this is being set because of the wording used right now, which is:
"Mumble-server (murmurd) can use higher priority process and network priority to ensure low latency audio forwarding even on highly loaded servers.
Allow mumble-server to use higher priority?"

I expect that the file permissions issue concerning reading the SSL cert(s) isn't Debian-specific, though it's worth understanding what's going on there either way. This is going to become more of an issue soon because Let's Encrypt is going to start supporting wildcard SSL certs next month, as wildcard certs will need to be sharable between a number of daemons.
https://letsencrypt.org/2017/07/06/wildcard-certificates-coming-jan-2018.html

[somniumAeternam]

Since the last update on openSUSE Tumbleweed for the mumble-server 1.2.19-4, I have the same issue that I cannot read the certificates of Lets Encrypt. These certificates are working for all other services (webserver, prosody etc.) but not for mumble-server anymore. On openSUSE Tumbleweed there is no option "MURMUR_USE_CAPABILITIES". And even if I start as root and increase the permissions (rwx), the the server fails to read the certificates. What can I do now? I'm forced to use a self-signed certificate.

[storrgie]

I've just been bit by this as well in debian

[john-penntech]

Having the same issue in Ubuntu 18.04.1 LTS

[RenHWall]

I just migrated my mumble server to a new system and ran into this problem again.
Old system: Ubuntu 16.04, mumble-server 1.2.12-1
New system: Ubuntu 18.04, mumble-server 1.3.0-1
On both mumble cannot load certificates unless allow mumble server to use higher priority has been set with dpkg-reconfigure.
This is regardless of what permissions the mumble-server user has on the folders containing the keys.
IMO this is not just a bug on Debian, but a bug on Ubuntu as well. There just happens to be an easier workaround on Ubuntu.

[bobonov]

I can confirm same issue on ubuntu server 20.04 and mumble server 1.3.0
It fails to load certbot certificate

[Krzmbrzl]

@bobonov could you please list the permissions for the certificate file? And could you also check that all directories in the path of the certificate file are accessible (have the x flag for the respective user/group).

As which user (and in what group) is murmur running? I assume it's a user separate from your default user account?

[bobonov]

Sorry for the late answer:
certificate are letsencrypt one and are generated with certbot.
Since I use nginx, at installation time I used the following option

certbot --nginx

certificate are working properly in nginx.

in mumble-server.ini I configured the following:

sslCert=/etc/letsencrypt/live/my.domain/fullchain.pem
sslKey=/etc/letsencrypt/live/my.domain/privkey.pem

It start as root, (service at boot time), then it switch to default user
uname=mumble-server
ps -axu | grep mumble
mumble-+ 1024 0.0 0.1 154784 10316 ? Sl mag02 0:03 /usr/sbin/murmurd -ini /etc/mumble-server.ini

Here follow the ls -la /etc/letsencrypt/live/my.domain/

lrwxrwxrwx 1 root root   43 apr 30 14:37 cert.pem -> ../../archive/my.domain/cert3.pem
lrwxrwxrwx 1 root root   44 apr 30 14:37 chain.pem -> ../../archive/my.domain/chain3.pem
lrwxrwxrwx 1 root root   48 apr 30 14:37 fullchain.pem -> ../../archive/my.domain/fullchain3.pem
lrwxrwxrwx 1 root root   46 apr 30 14:37 privkey.pem -> ../../archive/my.domain/privkey3.pem

These are symlink to the files which permission are:

ls -la /etc/letsencrypt/archive/my.domain/

-rw-r--r-- 1 root root 1923 apr 30 14:37 cert3.pem
-rw-r--r-- 1 root root 1647 apr 30 14:37 chain3.pem
-rw-r--r-- 1 root root 3570 apr 30 14:37 fullchain3.pem
-rw-r--r-- 1 root root 1704 apr 30 14:37 privkey3.pem

Files currently in use are the *3.pem, certbot always add the new certificate incrementally at renew times and symlink to the latest one.

Probably it should read the certificate before switching the user.

[Krzmbrzl]

@bobonov Thanks for the input. I assume you are also getting the "Failed to read ..." error message in the log?

To me your file permissions look correct, but on the other hand our code also looks correct:
https://github.com/mumble-voip/mumble/blob/5be2bb965faa5e0b6c691d8dc3a6f2692355c97e/src/murmur/Meta.cpp#L477-L484
Maybe the opening of read-only files doesn't work well in Qt making this a Qt bug?

Are you using the static server binary or are you using a dynamically linked (I think if you installed it from the repos, it should be dynamic). In the latter case what's the output of qmake --version?

[streaps]

Private keys shouldn't be readable by everyone. Some applications refuse to use certs that have read permissions for others. I don't think this is the problem here, but with 644 permissions you could also make mumble-server (or murmur) the owner.

[RenHWall]

@streaps
The private keys are not really world readable since the parent folder has restricted permissions, this is the default behavior of certbot. Here is a more complete example from my Ubuntu 18.04 server.

~$ sudo ls -l /etc/letsencrypt/
total 40
drwx------ 4 root root     4096 Dec 22 02:09 accounts
drwx------ 3 root root     4096 Dec 22 02:08 archive
-rw-r--r-- 1 root root      121 Feb 10  2019 cli.ini
drwxr-xr-x 2 root root     4096 Apr 20 08:10 csr
drwx------ 2 root root     4096 Apr 20 08:10 keys
drwxr-x--- 3 root ssl-cert 4096 Dec 22 02:08 live
-rw-r--r-- 1 root root     1143 Dec 22 02:08 options-ssl-nginx.conf
drwxr-xr-x 2 root root     4096 Apr 20 08:10 renewal
drwxr-xr-x 5 root root     4096 Dec 22 02:08 renewal-hooks
-rw-r--r-- 1 root root      424 Dec 22 02:08 ssl-dhparams.pem

~$ sudo ls -l /etc/letsencrypt/live/
total 8
drwxr-xr-x 2 root root 4096 Apr 20 08:10 my.domain.name
-rw-r--r-- 1 root root  740 Dec 22 02:08 README

~$ sudo ls -l /etc/letsencrypt/live/my.domain.name
total 4
lrwxrwxrwx 1 root root  34 Apr 20 08:10 cert.pem -> ../../archive/my.domain.name/cert3.pem
lrwxrwxrwx 1 root root  35 Apr 20 08:10 chain.pem -> ../../archive/my.domain.name/chain3.pem
lrwxrwxrwx 1 root root  39 Apr 20 08:10 fullchain.pem -> ../../archive/my.domain.name/fullchain3.pem
lrwxrwxrwx 1 root root  37 Apr 20 08:10 privkey.pem -> ../../archive/my.domain.name/privkey3.pem
-rw-r--r-- 1 root root 692 Dec 22 02:08 README

The mumble-server user is a member of the ssl-cert group and so has permissions to read the folder, and as such should be able to load the certs.
Edit: I forgot how symlink permissions work.

[streaps]

@RenHWall

In your example the /etc/letsencrypt/archive directory has "drwx------" permissions and can only be accessed by root.

The files in /etc/letsencrypt/live/my.domain.name are only symlinks that point to the archive directory where the cert files are.

Try to make archive directory and the files inside readable for the ssl-cert group.

[RenHWall]

Thanks to @streaps for pointing out my misunderstanding symlink permissions mumble is now correctly reading the certificates without starting as root.
Since new files are placed in /etc/letsencrypt/archive/my.domain.name/ every time the certificate is updated rather than updating the existing files the permissions of the new certificate needs to be changed as well. To accomplish this I have edited /etc/letsencrypt/renewal-hooks/deploy/mumble-restart.sh to include the following.

#!/bin/sh
#Update permissions on new certificates so the ssl-cert group (mumble-server) can read them.
chmod 640 /etc/letsencrypt/archive/my.domain.name/privkey*
chgrp -R ssl-cert /etc/letsencrypt/archive/

# Actually restart mumble
systemctl restart mumble-server.service

[Krzmbrzl]

@RenHWall that's great to hear. Also thanks for sharing your solution :)

@bobonov any chance you are experiencing the same problem? :thinking:

[TredwellGit]

On Ubuntu with Certbot you can:

sudo chmod 755 /etc/letsencrypt/{archive,live}/
sudo chgrp mumble-server -- /etc/letsencrypt/archive/mumble.example.net/privkey*.pem
sudo chmod 640 -- /etc/letsencrypt/archive/mumble.example.net/privkey*.pem

In /etc/mumble-server.ini set:

sslCert=/etc/letsencrypt/live/mumble.example.net/fullchain.pem
sslKey=/etc/letsencrypt/live/mumble.example.net/privkey.pem

In /etc/letsencrypt/renewal-hooks/deploy/mumble.example.net:

#!/usr/bin/env sh
exec pkill -USR1 -F /run/mumble-server/mumble-server.pid

This fixes the overly restrictive default Certbot permissions scheme, uses live reloading of certificates so there is no need to restart the server, and does not require the Mumble server to start as root. The group permissions from the previous privkey file will be used on new renewed privkey files created by Certbot.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information.
With only the information that is currently in the issue, we don't have enough information to take action.

Please reach out if you have or find the answers we need so that we can investigate further (or if you feel like this issue shouldn't be closed for another reason).