Rocket.chat: snap.rocketchat-server.rocketchat-caddy.service ACTIVE: failed

[vatixxx]

Hi, I've got problem with new rocketchat-server, after snap install rocketchat-server , rocket not working.
OS:ubuntu 17.10
snap:
core:16-2.28.5 Rev.3247
rocketchat-server 0.59.0 Rev.1170
I have checked logs of rocket and every time problem is in caddy.

paź 19 13:35:30 ubuntuserver systemd[1]: Started Service for snap application rocketchat-server.rocketchat-caddy.
paź 19 13:35:30 ubuntuserver rocketchat-server.rocketchat-caddy[2212]: 2017/10/19 13:35:30 loading Caddyfile via flag: open /C
paź 19 13:35:30 ubuntuserver systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Main process exited, code=exited, status=1/FAILURE
paź 19 13:35:30 ubuntuserver systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Unit entered failed state.
paź 19 13:35:30 ubuntuserver systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Failed with result 'exit-code'.
paź 19 13:35:31 ubuntuserver systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Service hold-off time over, scheduling restart.
paź 19 13:35:31 ubuntuserver systemd[1]: Stopped Service for snap application rocketchat-server.rocketchat-caddy.

snap refresh rocketchat-server --candidate or beta or stable not working for me.
What I can do?

[geekgonecrazy]

Caddy actually stops like that out of the box. So caddy isn't the issue. Can you provide additional logs? Also did you try accessing at: localhost:3000 or {yourserver ip}:3000 . Double check firewall if on a vps.

[707eric]

do
sudo systemctl status snap.rocketchat-server.rocketchat-caddy.service

You should get an output like the following.

If it doesn't then:
edit your /etc/hosts file and add your public IP and the FQDN you are planning to use

Then
cd /var/snap/rocketchat-server/1188

Verify your Caddyfile looks like:

YOUR.HOSTNAME.com{
proxy / localhost:3000 {
websocket
transparent
}
}

Then run
sudo /usr/bin/snap run rocketchat-server.rocketchat-caddy

Post the results after running the last command if it still doesnt work.

[zenglicheng]

Hi,
I have a similar issue starting rocketchat-caddy service. The journal entry for the failure is:

Activating privacy features... 2018/04/24 17:58:09 open /root/snap/rocketchat-server/1254/.caddy/acme/acme-v02.api.letsencrypt.org/sites//.crt: no such file or directory

I used nautilus to observe the directory of /root/snap/rocketchat-server/1254/.caddy/acme/acme-v02.api.letsencrypt.org/sites/. I saw the crt file was briefly generated and then disappeared. So there is nothing in that directory.

This is a fresh install of rocketchat. I researched the previous discussion. There was a similar case of "missing .crt file causing rocketchat-caddy to fail". However, that was related to a refreshing of installation. For me, this is a fresh installation.

The I use is a valid Domain name registered with GoDaddy.

Much appreciate if you could help me in this case.

Kind regards
LC

[707eric]

What are the contents of you Caddy file?

It should be located in
/var/snap/rocketchat-server/current/Caddyfile

[zenglicheng]

Wow, you responded so quickly. Appreciate it, Eric.

The contents of my Caddy file are:

proxy / localhost:3000 {
websocket
transparent
}

I just follow the instructions of https://rocket.chat/docs/installation/manual-installation/ubuntu/snaps/autossl/ to configure Caddy.

Could you please advise the reason why .crt cannot be created? I saw briefly .crt.lock file was created while rocketserver-caddy service is being restarted.

[707eric]

I think you are just missing some brackets in your Caddyfile.

MYDOMAIN.COM {
proxy / localhost:3000 {
websocket
transparent
}
}

That's the how my Caddyfile looks

[zenglicheng]

Many thanks for your reply, Eric. I tried with your method, but still no luck.

However, I managed to get it working with self-signed certificate.
https://mydomain.com
tls self_signed
proxy / localhost:3000 {
websocket
transparent
}

I guess the reason my domain name causes the Caddy service to fail is that, "letsencrypt" service could not generate a SSL certificate for my domain name. And the reason it couldn't generate the certificate is because my domain name, although it is registered with a DNS provider, has no CNAME record, I haven't mapped the name to any IP address yet. I haven't tried mapping my domain name to an IP address yet, as my computer I am using is sitting behind a modem/router.

In your case, I suppose your DNS name is bound to some IP address or has a CNAME attached to it?

thank you for your help

[707eric]

Oh, in that case you just need to make an A record to the public IP where your server is located. Then setup NAT and firewall rules to point to your server and the DNS test will succeed.

I have Caddy working as a proxy for multiple sites . I prefer expose only one point of entry vs multiple.

[lzeng]

Eric,

Thanks for the advice. Next step I want to experiment is to embed rochetchat with my website. Do you experience with this? As we know the documentation is quite terse, if you can share an actual example of a custom site embedding RochetChat, that would be great.

regards

[2E0PGS]

I have the exact same issue. I have had a install just break randomly I suspect snap automatically refreshed/updated.

Running sudo /usr/bin/snap run rocketchat-server.rocketchat-caddy I get no such file or directory

In my /var/log/syslog I get the same error too.

I have had the CaddyFile working fine before and it appears to be the same config now.

[2E0PGS]

I guess since it was working fine before this rules out it being something to do with my domain name unless something changed upstream which makes my sub domain invalid for certification.

[707eric]

Run
sudo systemctl restart snap.rocketchat-server.rocketchat-caddy.service

then

journalctl -xe

and look if it says it fails and if so why it might have failed.

I have been running multiple instances of Rocket with two different users groups with the system available 24/7 for about 6 months now while updating and using customizations like a custom auth provider (Auth0) and so far it has been smooth without any issues with Caddy except the one time initially.

[707eric]

You can also check for the existence of the Caddyfile and permissions

ls -l /var/snap/rocketchat-server/current

and if it is there

cat /var/snap/rocketchat-server/current/Caddyfile

verify the contents are accurate

[2E0PGS]

The Caddy file is present and correct syntax as I stated above.
Its permissions are: -rw-r--r-- 1 root root 83 May 12 17:42 Caddyfile

Running journalctl -xe shows me the same error of file not present: MYDOMAINNAME.crt: no such file or directory

[2E0PGS]

Related posts I found while researching this issue:

• https://wyldeplayground.net/rocket-chat-snap-doesnt-start/
• https://askubuntu.com/questions/894670/rocket-chat-snap-doesnt-start
• Useful Caddy tutorial: https://caddyserver.com/docs/automatic-https

Also a side note my rocket chat is accessible without going via Caddy proxy. So I can connect to my rocket chat using its internal IP and default port non HTTPS.

[707eric]

That sounds like the similar issue above where the crt file wasn't being retained due to an with the Caddyfile. Please paste the contents of the Caddyfile.

[2E0PGS]

cat /var/snap/rocketchat-server/current/Caddyfile

https://subdomain.domain.org
proxy / localhost:3000 {
  websocket
  transparent
}

[2E0PGS]

However I just tried @zenglicheng comment and it works with self sign cert.

cat /var/snap/rocketchat-server/current/Caddyfile

https://subdomain.domain.org
tls self_signed
proxy / localhost:3000 {
  websocket
  transparent
}

[707eric]

Drop the https part and it will work. The Let's Encrypt is failing because Caddy isn't responding on port 80, its responding on 443.

MYDOMAIN.COM {
proxy / localhost:3000 {
websocket
transparent
}
}

[2E0PGS]

Your statement doesn't make much sense it contradicts itself (remove HTTPS because it responds on the port for HTTPS is what your saying in other words). Also I tried that and it didn't work.

That's the CA for the cert as far as i'm aware.

[707eric]

They changed the syntax of the file so it doesn't fully apply though it still requires having port 80 open to function.

Ports 80 and 443 must be externally open
By default, Caddy will bind to ports 80 and 443 to serve HTTPS and redirect HTTP to HTTPS. This usually requires privilege escalation. On Linux systems, you can give Caddy permission to bind to port 80 and 443 without being root using setcap, like so: setcap cap_net_bind_service=+ep caddy. Don't forget to configure all relevant firewalls to allow Caddy to use these ports for incoming and outgoing connections! Caddy must have claim on these ports to obtain certificates unless you enable the DNS challenge OR forward ports 80 and 443 to different ports internally (in which case you can change the HTTP and HTTPS ports using CLI flags).

[707eric]

example.com # Host: example.com; Ports: 80->443
http://example.com # Host: example.com; Port: 80
https://example.com # Host: example.com; Ports: 80->443

Previously when https was specified it wouldn't auto redirect 80 to 443, to have it work in that fashion you left the http and https off. They appear to have changed, maybe even a while ago.I have been using it for quite some time now and I don't really stay up to date on all the changes.

[2E0PGS]

So are you saying I need to port forward port 80 in addition to my existing port 443 forward?
Then remove the https:// from my config?

I don't need 80 to 443 redirection as I only want users accessing it via 443. So that's why only 443 is forwarded at the moment and my config is setup like so. This used to work just fine around 4-6 months ago maybe.

Also surely rocketchat would of added some rules to prevent breaking everyone's existing configs if Caddy changed something upstream?

[2E0PGS]

Related issue: https://github.com/mholt/caddy/issues/468

[707eric]

Yes, to have the Let's Encrypt SSL work you need to have port 80 and port 443 open and point to your system running Caddy.

Its a better user experience this way vs yours. It will make it so no matter what the user types it will always redirect to a secured connection. Your current method if they type http://YourDomain.com they would receive a 404 or some other error.

Using the suggested method would mean even if they typed http://YourDomain.com they would end up at https://YourDomain.com

[2E0PGS]

Yes I am fully aware it's for SSL redirect purposes. Common practice for public sites.

I don't need that user experiance it's locked down for security and is a invite only system. Ideally there should be an option to run SSL port only. There is CLI flags it seems for disabling the different types of challeneges: disable-http-challenge and -disable-tls-sni-challenge. I wonder if these flags can also be accessed via the Caddy snap for rocketchat?

[707eric]

Having your proxy server redirect port 80 to port 443 is no less secure than the method you are using. Having it locked down to port 443 only, provides no additional security benefit to the system.

I run multiple instances, all of which are invite only and all of which are related to cyber security.

[2E0PGS]

More ports open is a bigger targer especially on default port numbers. If I only need one port then why expose port 80 to attract more attackers.

⁣Peter Stevenson (2E0PGS)
http://www.m3pgs.co.uk​

On 22 May 2018, 23:07, at 23:07, Eric Brogdon [e-mail] wrote:

Having your proxy server redirect port 80 to port 443 is no less secure
than the method you are using. Having it locked down to port 443 only,
provides no additional security benefit to the system.

I run multiple instances, all of which are invite only and all of which
are related to cyber security.

--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/RocketChat/Rocket.Chat/issues/8539#issuecomment-391156198

[2E0PGS]

Opening port 80 on the NAT didn't work. Perhaps I need to set the NAT rule to translate it from 80 external to default port internally 3000? However it seemed like the caddy snap service wasn't exiting like it did before.

[707eric]

And you meet these requirements?
The hostname:
-is not empty
-is not localhost
-is not an IP address
-has no more than 1 wildcard (*)
-wildcard must be left-most label

The port is not explicitly 80
The scheme is not explicitly http
TLS is not turned off in site's definition
Certificates and keys are not provided by you
Caddy is able to bind to ports 80 and 443

[2E0PGS]

Should my RocketChat be listening on port 80?

Here are the ports open on the server itself locally:

[2E0PGS]

There are no references to port 80 within any configs or ports open...

All I see is 3000 which seems to be the default for RocketChat and 443 which is what I had specified in the Caddy file for the proxy to 3000.

Anyway I found this interesting that its now trying port 8080 for some reason:

CGroup: /system.slice/snap.rocketchat-server.rocketchat-caddy.service
           └─5803 caddy -conf=/var/snap/rocketchat-server/1265/Caddyfile -host=localhost:8080

I am trying a combination of configs in the Caddy file to see what causes it to crash and what works to bind.

[2E0PGS]

So without specifying HTTP or HTTPS in front the domain it will say its proxying via port 2015 which seems like a random port. So I cannot get it to work without specifying either HTTP 80 or HTTPS 443 in front the domain in my Caddy file.

CADDY FILE:
subdomain.mydomain.org
tls self_signed
proxy / localhost:3000 {
  websocket
  transparent
}

SERVICE LOG:
May 26 14:14:07 rocketchat rocketchat-server.rocketchat-caddy[6143]: Activating privacy features... done.
May 26 14:14:07 rocketchat rocketchat-server.rocketchat-caddy[6143]: https://subdomain.mydomain.org:2015

NMAP:
22/tcp   open  ssh
3000/tcp open  ppp

However if I specify http://subdomain.mydomain.org or https://subdomain.mydomain.org it will open and proxy via either http80 or https443 but never have I seen it proxy both at same time. Only one of the other.

[2E0PGS]

OK so the only way I have got both port 80 and port 443 proxying is by specifying two rules

CADDY FILE:
http://subdomain.mydomain.org, https://subdomain.mydomain.org
tls self_signed
proxy / localhost:3000 {
  websocket
  transparent
}

NMAP:
22/tcp   open  ssh
80/tcp   open  http
443/tcp  open  https
3000/tcp open  ppp

However the HTTP port 80 wont be redirected to HTTPS.

[2E0PGS]

More related posts:
https://caddy.community/t/rocketchat-works-only-in-self-signed-mode/3595
https://github.com/RocketChat/Rocket.Chat/issues/10268

[707eric]

What happens if you just do
subdomain.mydomain.org {
tls self_signed
proxy / localhost:3000 {
websocket
transparent
}
}

Keep the brackets I added as well

[707eric]

Oh ok, I just scrolled up and saw the port 8080 mention

I was having that issue as well. I ended up reverting one instance back to an older Rocket because I couldn't get the issue to resolve. I know where the 8080 is coming from but I don't know how to stop it sinces its in the snap which is read only

[geekgonecrazy]

A new version of the snap was released with the newest caddy version. Please give it a try and see if it solves your problems.

Going to go ahead and close this. If it persists we can open it back up.

Thanks,