Jitsi-meet: On Debian Buster, fresh install, coturn server fails to start

[julienfastre]

## Description

I did a fresh install on Debian Buster using package jitsi-meet.

sudo apt install jitsi-meet
sudo /usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh

Warning: this is a fresh install, not an upgrade.

After install, the coturn server does not start, altough it is marked as started by systemd.

## Current behavior

The coturn service does not start properly:

1. the port 4445 is not listening:

root@test-visio-443:/home/debian# netstat -ltnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      14191/nginx: master 
tcp        0      0 0.0.0.0:4444            0.0.0.0:*               LISTEN      14191/nginx: master 
tcp        0      0 0.0.0.0:5280            0.0.0.0:*               LISTEN      14146/lua5.2        
tcp        0      0 127.0.0.1:5347          0.0.0.0:*               LISTEN      14146/lua5.2        
tcp        0      0 0.0.0.0:5222            0.0.0.0:*               LISTEN      14146/lua5.2        
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      14191/nginx: master 
tcp        0      0 0.0.0.0:5269            0.0.0.0:*               LISTEN      14146/lua5.2        
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      9009/sshd           
tcp6       0      0 :::4444                 :::*                    LISTEN      14191/nginx: master 
tcp6       0      0 :::5280                 :::*                    LISTEN      14146/lua5.2        
tcp6       0      0 ::1:5347                :::*                    LISTEN      14146/lua5.2        
tcp6       0      0 :::5222                 :::*                    LISTEN      14146/lua5.2        
tcp6       0      0 :::80                   :::*                    LISTEN      14191/nginx: master 
tcp6       0      0 :::5269                 :::*                    LISTEN      14146/lua5.2        
tcp6       0      0 :::22                   :::*                    LISTEN      9009/sshd           
tcp6       0      0 :::8888                 :::*                    LISTEN      11797/java

The service is marked as active:

sudo systemctl status coturn
● coturn.service - coTURN STUN/TURN Server
   Loaded: loaded (/lib/systemd/system/coturn.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-04-01 08:37:41 UTC; 12s ago
     Docs: man:coturn(1)
           man:turnadmin(1)
           man:turnserver(1)
  Process: 16438 ExecStart=/usr/bin/turnserver --daemon -c /etc/turnserver.conf --pidfile /run/turnserver/turnserver.p
  Process: 16440 ExecStartPost=/bin/sleep 2 (code=exited, status=0/SUCCESS)
 Main PID: 16439 (turnserver)
    Tasks: 3 (limit: 4915)
   Memory: 3.9M
   CGroup: /system.slice/coturn.service
           └─16439 /usr/bin/turnserver --daemon -c /etc/turnserver.conf --pidfile /run/turnserver/turnserver.pid

By adapting the service definition, I could get some logs:

Apr 01 09:43:01 test-visio-443 turnserver[17105]: 0: Trying to bind fd 16 to <127.0.0.1:443>: errno=13
Apr 01 09:43:01 test-visio-443 turnserver[17105]: 0: Cannot bind DTLS/UDP listener socket to addr 127.0.0.1:443
Apr 01 09:43:01 test-visio-443 turnserver[17105]: 0: Trying to bind DTLS/UDP listener socket to addr 127.0.0.1:443, again..

## Expected Behavior

The coturn server should start and listen to port 4445.

## Possible Solution

• /etc/turnserver.conf is maybe broken ? This is the content:

# jitsi-meet coturn config. Do not modify this line
lt-cred-mech
use-auth-secret
keep-address-family
static-auth-secret=xxxxxx
realm=visio443.xxx.be
cert=/etc/letsencrypt/live/visio443.xxx.be/fullchain.pem
pkey=/etc/letsencrypt/live/visio443.xxx.be/privkey.pem

no-tcp
listening-port=443
tls-listening-port=4445
external-ip=visio443.xxx.be

## Steps to reproduce

See under title "Description" :smile:

# Environment details

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:    10
Codename:   buster

[julienfastre]

I also opened a thread on community.jitsi.org here: https://community.jitsi.org/t/problem-with-coturn-on-new-jvb2-installation-on-debian-buster/27966/10

[saghul]

Looks like this is the culprit: https://github.com/jitsi/jitsi-meet/blob/master/doc/debian/jitsi-meet-turn/turnserver.conf#L11

@damencho Why did we leave that there, since nginx will actually listen on 443 and do the dispatching?

[saghul]

Ah, wait, that should be UDP!

[saghul]

Trying to bind fd 16 to <127.0.0.1:443>: errno=13

Errno 13 is EPERM, maybe a capability is missng, hum.

[rfrederick]

Trying to bind fd 16 to <127.0.0.1:443>: errno=13

Errno 13 is EPERM, maybe a capability is missng, hum.

I believe systemd on Buster starts coturn as the "turnserver" user and not as root, so coturn isn't able to bind to privileged ports.

[julienfastre]

This is a good path. Coturn start if I add no-udp to the file /etc/turnserver.conf

I also tryed to create a socket, which seems more elegant, but until now it fails:

[Unit]
Description=coturn socket

[Socket]
ListenDatagram=443

I also added a line Requires=coturn.socket in the coturn service definition.

I think that the coturn service definition is done by the coturn debian package, not jitsi. May we find a solution which implies to change it ?

[julienfastre]

May I also suggest to add a line syslog to /etc/turnserver.conf ?

[saghul]

Can you try to add this under [service]? AmbientCapabilities=CAP_NET_BIND_SERVICE

Then you probably need to systemctl daemon-reload and then systemctl restart coturn

[gspu]

@saghul
Yes, thats the solution!
No crashes of coTURN anymore on my maschine.

Thanks allot!

[saghul]

Excellent! We are adding this to the Debian package.

[rolnas]

I just tested with last changes. turnserver starts, but doesn't listen on 4445/tcp in default config. Probably because of no-tcp option in config file. But if we will change no-tcp (by commenting it), then coturn listen on 443 and 4445 what will be problem with nginx on 443/tcp. In this case we should change listening-port=443 to 4445 (if it will not break functionality). Should we also allow 4445/udp on firewall ?

[saghul]

The default setup uses UDP and TLS. Why do youi want to enable TCP?

[rolnas]

But in this case nginx config to pass tcp packets to turnserver from 443 to 4445 will not work, because nginx can pass only tcp. Also if we start with default config, 4445 port is not open (tcp or udp) so it is useless. There are no such IP protocol TLS, only UDP or TCP. TLS is a layer over UDP or TCP.

[julienfastre]

As far as I understood,

• the coturn is able to handle the protocol TCP and UDP.
• nginx receive TCP packages on port 443. Nginx is able to filter the https / http2 / http conversations and send those package to... himself, but on a second listening port, the port 4444. The conversation which does not concerns https / http2 / http is send to the turn server, listening on port 4445.

This is done by the file /etc/nginx/modules-enabled/60-jitsi-meet.conf:

stream {
    upstream web {
        server 127.0.0.1:4444;
    }
    upstream turn {
        server 127.0.0.1:4445;
    }
    # since 1.13.10
    map $ssl_preread_alpn_protocols $upstream {
        "h2"            web;
        "http/1.1"      web;
        "h2,http/1.1"   web;
        default         turn;
    }

    server {
        listen 443;

        # since 1.11.5
        ssl_preread on;
        proxy_pass $upstream;

        # Increase buffer to serve video
        proxy_buffer_size 10m;
    }
}

You are right that with default config, port 4445 is not open, and that is the subject of this issue: coturn does not start.

[rolnas]

After improving systemd unit on Debian 10/buster, coturn starts, but it listen only on 443/UDP. It doesn't listen on tls-listening-port=4445, because tcp is disabled with option no-tcp.

[julienfastre]

You are right, @rolnas : coturn start but does not listen on port 4445/TCP.

We add "AmabientCapabilities" when overriding the service definition (and other stuff to improve logging):

root@test-visio-443:/var/log# diff /etc/systemd/system/coturn.service /usr/lib/systemd/system/coturn.service 
9c9
< Type=simple
---
> Type=forking
11c11,12
< ExecStart=/usr/bin/turnserver -c /etc/turnserver.conf  -v --syslog
---
> PIDFile=/run/turnserver/turnserver.pid
> ExecStart=/usr/bin/turnserver --daemon -c /etc/turnserver.conf --pidfile /run/turnserver/turnserver.pid
17d17
< AmbientCapabilities=CAP_NET_BIND_SERVICE

Reload systemd and start coturn:

root@test-visio-443:/var/log# sudo systemctl daemon-reload 
root@test-visio-443:/var/log# sudo systemctl start coturn

Nobody is listening on port 4445:

root@test-visio-443:/var/log# sudo netstat -lntp | grep 44
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      14191/nginx: master 
tcp        0      0 0.0.0.0:4444            0.0.0.0:*               LISTEN      14191/nginx: master 
tcp6       0      0 :::4444                 :::*                    LISTEN      14191/nginx: master 

starting coturn as root:

root@test-visio-443:/var/log# sudo systemctl stop coturn
root@test-visio-443:/var/log# sudo turnserver -c /etc/turnserver.conf --daemon --syslog -v
# ...
# snipped
# ...
root@test-visio-443:/var/log# sudo netstat -lntp | grep 44
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      14191/nginx: master 
tcp        0      0 0.0.0.0:4444            0.0.0.0:*               LISTEN      14191/nginx: master 
tcp        0      0 213.xxx.xxx.xxx:4445        0.0.0.0:*               LISTEN      7533/turnserver     
tcp        0      0 127.0.0.1:4445          0.0.0.0:*               LISTEN      7533/turnserver     
tcp        0      0 213.xxx.xxx.xxx:4445        0.0.0.0:*               LISTEN      7533/turnserver     
tcp        0      0 127.0.0.1:4445          0.0.0.0:*               LISTEN      7533/turnserver     
tcp6       0      0 :::4444                 :::*                    LISTEN      14191/nginx: master 
tcp6       0      0 ::1:4445                :::*                    LISTEN      7533/turnserver     
tcp6       0      0 ::1:4445                :::*                    LISTEN      7533/turnserver 

[rolnas]

coturn on 4445/tcp with tls doesn't start because as a turnserver user cannot read server certificate private key. Adding turnserver to ssl-cert solves problem and turnserver starts to listen on 4445.

[julienfastre]

mmmh... it seems not to be sufficient for me:

$ id turnserver
uid=110(turnserver) gid=117(turnserver) groups=117(turnserver),114(ssl-cert)

And I still get the error:

Apr 02 15:41:24 test-visio-443 turnserver[8591]: 0: WARNING: cannot find certificate file: /etc/letsencrypt/live/visio443.xxx/fullchain.pem (1)
Apr 02 15:41:24 test-visio-443 turnserver[8591]: 0: WARNING: cannot start TLS and DTLS listeners because certificate file is not set properly
Apr 02 15:41:24 test-visio-443 turnserver[8591]: 0: WARNING: cannot find private key file: /etc/letsencrypt/live/visio443.xxxx.be/privkey.pem (1)
Apr 02 15:41:24 test-visio-443 turnserver[8591]: 0: WARNING: cannot start TLS and DTLS listeners because private key file is not set properly

Did you add something else ?

[rolnas]

I'm also using Let's encrypt certificates, but with puppet I copies them to /etc/ssl with standard permissions (-rw-r----- 1 root ssl-cert 1679 Apr 2 17:25 /etc/ssl/example.key). So turnserver starts with both ports listening after aduser turnserver ssl-cert.

[julienfastre]

FYI: https://github.com/coturn/coturn/issues/496

[julienfastre]

As I could read, there are two ways of making certificates available for user turnserver:

1. change permissions on /etc/letsencrypt (which may be overriden each time the letsencrypt certificate is renewed)
2. add a hook which will copy keys and certificates in another directories and make them available for coturn.

It seems the second soluce is more resilient.

I am ok for writing a script, but I do not know in which repo I should write it and I do not have any experience in debian packaging.

[julienfastre]

Concerning the ability to listen on port 443/udp: is this really used ? If the client is able to send tcp on port 443, and udp on port 10000, why should a client have an access to udp on port 443 ?

[saghul]

That is for the case in which UDP is completely bloked. Some firewalls may allow "anything" on port 443, so we are banking on that.

[saghul]

@damencho Looks like Debian buster has more coturn trouble, sigh.

[julienfastre]

@saghul you mean that it would be prefereble to send udp on port 443 than tcp on port 443 ?

During my test, I concluded that the client:

• test a connection on port udp/10000 ;
• if not available, fallback on port tcp/443 ;

But no fallback on port udp/443. So, I wonder if we could not remain the port udp/443 close (tcp/443 will remains open by nginx).

And when coturn is started, it works great !

[saghul]

Coturn is used for media. So, it provides an alternative to port 10000, not to TCP 443. We also use it in 1-1 calls, in order to avoid connecting via the JVB.

[julienfastre]

So, you confirm that the port udp/443 is not necessary ? @saghul

In this case, I think that adding a no-udp to /etc/turnserver.conf option might be a first resolution (no need for overriding the systemd service definition).

I am ok for writing a hook to copy certbot certificates in a safe place.

[damencho]

We just think to make it port 4446 so we can still use it. Here https://github.com/jitsi/jitsi-meet/blob/7f1eb83dbd5c6d324bde2134dd103bcf11e35573/config.js#L346 and here https://github.com/jitsi/jitsi-meet/blob/06641a7dd6baff6f854591ad2f05ff18a9221c62/doc/debian/jitsi-meet-prosody/prosody.cfg.lua-jvb.example#L9 and here https://github.com/jitsi/jitsi-meet/blob/06641a7dd6baff6f854591ad2f05ff18a9221c62/doc/debian/jitsi-meet-prosody/prosody.cfg.lua-jvb.example#L10.
And remove the systemd thing.

[rolnas]

User could be on (corporate) firewall which pass only 80 and 443 and with luck both tcp and udp.

[damencho]

@rolnas So you think we should keep the 443 udp?

[rolnas]

My testing with 2 people on meeting (p2p mode) shows traffic on this port 443/udp (via firewall). I'm not expert on webrtc + stun + turn so I cannot be sure. For some people under some NAT solutions TURN service is required for webrtc p2p mode to work.

[damencho]

Well, we are moving it to a different port because of the problems we see on Debian, it will still be used if for any reason only port 443 is available it will fallback to TCP.

[julienfastre]

I wrote a pull request to solve the problems mentioned here.

Please, do not merge it: it is not yet ready.

(I still have to test the packaging of the fixes - the scripts have been tested invidually and the copied manually)

[julienfastre]

The PR https://github.com/jitsi/jitsi-meet/pull/5729 is going to fix this issue.