Laravel-websockets: Best way to configure laravel websockets + apache + ssl + letsencrypt + supervisor ?

[damienfournier]

Hello,

I've succeeded in making websockets work with ssl on apache, but I have some questions on how to configure it "the right way" with let's encrypt.

Let's encrypt update the certificate & private key every 2 months.
By default, the certificate is readable by everyone, but the private key is readable only by root (as it should be).

First question:
- should the websockets service be restarted every 2 months ? (each time the cert & key are regenerated) or does it get the new cert automatically ?

Second question:
- under which linux user should the command "php artisan websockets:serve" be run ? (and be configured in supervisor)

But to answer that question, there is another one before:

• can the websockets server write things in the laravel log (even when crashing).
  Because since I use daily logs for laravel, if a log file is created by another user than "www-data", the next time that the webapp want to log something, it will crash because it doesn't have permission on the log file.
  Other side effects could be any files written by the server (uploaded files, cached views...)
  I don't know if the websocket server can have any impact on this, or if it only deals with sockets.

So, the goals are:
1 - best security possible
2 - everything is automated
3 - doesn't have side effects with permissions (log files that make the app crash...)

The easiest way is to choose root, but:
1 - running code from the root user is bad
2 - ok
3 - may cause problems with logs

2nd solution, "www-data":
1 - it's better on security, but not great either... because now the entire web app has access to the private key
2 - it can't read the private key file... so it means we should find a way to change permissions (or to copy it to another place with the correct permissions) on that file each time it's generated
3 - no side effect, since it's the apache user

3rd solution, a dedicated user "laravel-echo" (like in the docs)
1 - it seems to be the most secure way
2 - same problems with the private key file
3 - may cause problems with logs

So, what is the right way ?
Should a script (or hook) be launched each time let's encrypt update ?

Thanks

[lovetoast]

Hi,

As far as I can see the service does not need to be restarted when a new certificate is installed.

I use the standard web server users, options 2 and 3 are pretty much the same in my opinion. My letsencrypt implementation uses a certificate and private key that is owned by the apache user anyway.

[bbashy]

If you don't need to verify the SSL cert, just use this in the configs;

websockets.php

'ssl' => [
    //...

    'verify_peer' => false,
],

broadcasting.php

'pusher' => [
    // ...
    'options' => [
        // ...
        'curl_options' => [
            CURLOPT_SSL_VERIFYHOST => 0,
            CURLOPT_SSL_VERIFYPEER => 0,
        ],
    ],
],

[damienfournier]

Hello,

@lovetoast
What do you mean by "my implementation is owned by the apache user anyway" ?
Have you just changed the permissions on the cert & key files, or does the call to certbot in crontab is changed to be ran by the www-data user ? (so that new generated cert & key will be owned by www-data)

(For info, I use a vanilla debian install. No forge, no plex, no nothing...)

@bbashy
I've seen this config options (curl options) in several places, but it seems to be more for self signed certificates in dev, and it was not recommended for security in production (which is the case here).
So, yes, I need to verify the ssl cert, and I hope the "best way to configure ssl with websockets" is not "let's disable the security" :)
That being said, I made it to work without the curl_options modifications, but I had to add the "verify_peer" to false... don't know if it has big impact on security

Thanks

[bbashy]

Since it's only websocket traffic (I guess there's nothing major being sent?) I would say it's fine to do. If someone has taken over your domain and issued a certificate, you've got other problems.

[lovetoast]

Hello,

@lovetoast
What do you mean by "my implementation is owned by the apache user anyway" ?
Have you just changed the permissions on the cert & key files, or does the call to certbot in crontab is changed to be ran by the www-data user ? (so that new generated cert & key will be owned by www-data)

(For info, I use a vanilla debian install. No forge, no plex, no nothing...)

@bbashy
I've seen this config options (curl options) in several places, but it seems to be more for self signed certificates in dev, and it was not recommended for security in production (which is the case here).
So, yes, I need to verify the ssl cert, and I hope the "best way to configure ssl with websockets" is not "let's disable the security" :)
That being said, I made it to work without the curl_options modifications, but I had to add the "verify_peer" to false... don't know if it has big impact on security

Thanks

Hi,

I'm using LetsEncrypt with VirtualMin. It appears that the user, as you have concluded, is run by the user to whom the website belongs. This isn't the case across all letsencrypt installations.

@bbashy Makes a valid point, and it appears if you want to use Apache, without a reverse proxy and letsencrypt, his workaround is required. I have tried this on two separate servers with different domains and I needed to do this on both.

Remember pecl event/ev, and changing your supervisor and user file limits!

[lovetoast]

If you don't need to verify the SSL cert, just use this in the configs;

websockets.php

'ssl' => [
    //...

    'verify_peer' => false,
],

broadcasting.php

'pusher' => [
    // ...
    'options' => [
        // ...
        'curl_options' => [
            CURLOPT_SSL_VERIFYHOST => 0,
            CURLOPT_SSL_VERIFYPEER => 0,
        ],
    ],
],

@bbashy

Hi.

I actually implemented these options on the advice of another posted, but I'm wondering, what does verify peer false in websockets.php do exactly? Because it's normally a curl option to check certificate of the receiving server, but what does it do in this file? Essentially who are we NOT verifying in this case?

[rennokki]

Please see the TLS context settings: https://www.php.net/manual/en/context.ssl.php

Regarding your concerns, I think the best way with the Let's Encrypt certificate is to make sure you get the server restarted. This behavior isn't tested. but luckily the 1.6.0 version came with a websockets:restart command that does it for you automatically.

While you got www-data for weblogs, it shouldn't access the private key at all. You can instead make a copy of it somewhere else as you said and give the proper permissions to the user that runs the php artisan command.

[damienfournier]

Hello,

thank you all for the answers.
I have some new questions (and I think it will also help anyone in a similar situation).

1. If I understood well, as soon as we set 'verify_peer = false' in websockets.php, the ssl cert won't be verified.
   That means that we can also set the curl options in broadcasting since it's not verified anyway...

So the connection will still be encrypted but with no guarantee that it's correct ssl cert.
Does it have big security consequences ? since the websockets code needs an authentified user anyway, is it a real problem for a saas product ?

1. If we want to have verify_peer = true, are we forced to have a reverse proxy ?
   is it doable on a single server or do I need 2 ? (one for the proxy, another for hosting)

if we can do it on apache without a reverse proxy, what is missing ?

1. @rennokki, you said that the user running "php artisan websockets:serve" shouldn't be root or www-data, but another one dedicated to that use, right ? (so, the 3rd solution in my first message ?)

So I need to create a shell script, ran by a cron or by a hook from a let's encrypt update, that will copy it and give read access to that user, and then restart the websocket server to be sure (with the restart command or by killing the existing process).

But, what about the side effects of not using www-data (for logs & files), is it a real concern ? can a log file in the laravel app (storage/logs) be created by the websocket code ?

1. Is there another simpler way ? (having the websocket on a subdomain ? 2 separate laravel apps ?)

Thanks again

[lovetoast]

Please see the TLS context settings: https://www.php.net/manual/en/context.ssl.php

Regarding your concerns, I think the best way with the Let's Encrypt certificate is to make sure you get the server restarted. This behavior isn't tested. but luckily the 1.6.0 version came with a websockets:restart command that does it for you automatically.

While you got www-data for weblogs, it shouldn't access the private key at all. You can instead make a copy of it somewhere else as you said and give the proper permissions to the user that runs the php artisan command.

Hi @rennokki , thanks for your reply, with regards to the verify_false setting with the SSL certificate, does it have a major impact on security? I had to use verify peer false as many others have, to get the web socket connection to work in chrome.

[rennokki]

It does, I mean if somebody signs a certificate on their own for your WebSockets domain and can intercept any of your clients' WebSockets connection, they can use Man-In-The-Middle Attack. Basically they get what the client sends (for example, triggering an event), and when the Server replies back, they steal that, modify the response with their certificate, and sends it back to the client, and the client won't notice because the handshake has already been made on the message channel: https://serverfault.com/a/915583

[lovetoast]

It does, I mean if somebody signs a certificate on their own for your WebSockets domain and can intercept any of your clients' WebSockets connection, they can use Man-In-The-Middle Attack. Basically they get what the client sends (for example, triggering an event), and when the Server replies back, they steal that, modify the response with their certificate, and sends it back to the client, and the client won't notice because the handshake has already been made on the message channel: https://serverfault.com/a/915583

Hi Again @rennokki but does this not relate to verify_peer at the client side, not the server side option?

[damienfournier]

Hello,

I think I got it right...
So, if anyone had the same questions, here's my stuff:

Permissions for logs
It seems to be a non-issue.
Since the websocket server doesn't seem to write things in the laravel log, it's not a problem if it's run by another user.
And parts where it could happen (the channels route file, or the broadcast events) are run by the apache user, so no problems).

So it's ok to create a 'laravel-websockets' user and use it to launch 'php artisan websockets:serve'

Supervisor
in /etc/supervisor/conf.d, I have a file that's like

[program:program_name_websockets]
command=php artisan websockets:serve

user=laravel-websockets
directory=/var/www/domain.com/www

numprocs=1

redirect_stderr=true
autostart=true
autorestart=true

stdout_logfile=/var/www/domain.com/www/storage/logs/domain_supervisord_websockets.log
stderr_logfile=/var/www/domain.com/www/storage/logs/domain_supervisord_websockets_error.log

Let's encrypt
I created a folder like: /etc/letsencrypt-secret-folder/domain.com to put copies of the certificates in it that will be readable by the laravel-websockets user

To automate the changes when a certificate is renewed I added a script in /etc/letsencrypt/renewal-hooks/deploy/domain.com.sh

#!/usr/bin/env bash

# put this script in /etc/letsencrypt/renewal-hooks/deploy, and chmod to 700
# if it doesn't work, do a dos2unix cmd on it to be sure
# you can test if it works with the command (in root): certbot -q renew --force-renew

for domain in $RENEWED_DOMAINS
do
    if [ "$domain" = domain.com ]
    then
        # copy new certificates (-H means we want the target of the symlink)
        cp -H /etc/letsencrypt/live/domain.com/fullchain.pem /etc/letsencrypt-secret-folder/domain.com/fullchain.pem
        cp -H /etc/letsencrypt/live/domain.com/privkey.pem /etc/letsencrypt-secret-folder/domain.com/privkey.pem

        # give correct rights
        chgrp laravel-websockets /etc/letsencrypt-secret-folder/domain.com/fullchain.pem
        chgrp laravel-websockets /etc/letsencrypt-secret-folder/domain.com/privkey.pem

        # chmod 640 => RW- R-- ---
        chmod 640 /etc/letsencrypt-secret-folder/domain.com/fullchain.pem
        chmod 640 /etc/letsencrypt-secret-folder/domain.com/privkey.pem

        # restart supervisor
        supervisorctl restart program_name_websockets
    fi
done

And do a certbot -q renew --force-renew to see if it works

Conclusion
Does it seems good to you ?

The last missing thing, is to know if there is a way to have it working with verify_peer = true in the config...
Is it possible on a single server ?

[makedirectory]

@damienfournier I tried your method above and it seems to work however in the network tab of safari I get the following message:

Status: 426 Upgrade header MUST be provided

Any ideas on where my setup could be wrong?

[damienfournier]

@makedirectory sorry, I have no idea because I haven't got that message and I don't use safari...

For other people, to close this ticket (and maybe ajust the documentation), it still needs the info about how to make it working with verify_peer = true in the config... any idea ?