Rocket.chat: Fail to start rocket.chat snap

[ricardojrgpimentel]

I'm having trouble starting my instance of rocket.chat, i'm running ubuntu 16.04 with a snap of rocket.chat and noticed this morning that the chat wasn't running, here are the logs:

systemctl status snap.rocketchat-server.rocketchat-server.service is running without errors

systemctl status snap.rocketchat-server.rocketchat-mongo.service is running without errors

systemctl status snap.rocketchat-server.rocketchat-caddy.service returns:

Mar 29 09:06:30 ubuntu-s-1vcpu-1gb-fra1-01 systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Main process exited, code=exited, status=1/FAILURE Mar 29 09:06:30 ubuntu-s-1vcpu-1gb-fra1-01 systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Unit entered failed state. Mar 29 09:06:30 ubuntu-s-1vcpu-1gb-fra1-01 systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Failed with result 'exit-code'. Mar 29 09:06:30 ubuntu-s-1vcpu-1gb-fra1-01 systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Service hold-off time over, scheduling restart. Mar 29 09:06:30 ubuntu-s-1vcpu-1gb-fra1-01 systemd[1]: Stopped Service for snap application rocketchat-server.rocketchat-caddy. Mar 29 09:06:30 ubuntu-s-1vcpu-1gb-fra1-01 systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Start request repeated too quickly. Mar 29 09:06:30 ubuntu-s-1vcpu-1gb-fra1-01 systemd[1]: Failed to start Service for snap application rocketchat-server.rocketchat-caddy.

journalctl -u snap.rocketchat-server.rocketchat-caddy.service returns:

open /root/snap/rocketchat-server/1238/.caddy/acme/acme-v02.api.letsencrypt.org/sites/example.com/example.com.crt: no such file or directory Main process exited, code=exited, status=1/FAILURE Unit entered failed state. Failed with result 'exit-code'.

So it looks like its a problem with the certificate, what can i do?

[Nepoxx]

example.com.crt certainly looks like an invalid certificate. Based on https://github.com/RocketChat/Rocket.Chat/issues/10266 it seems like the Caddyfile was overwritten by the default one, you might have to edit it to point to your certificate?

[ricardojrgpimentel]

Sorry i didnt said anything, in the log its pointing to the right domain, i just changed to show it here.
So its something to do with snap auto-refresh?

[ingokunzi]

also problem here:

Mar 29 08:03:44 Rocket systemd[1]: Started Service for snap application rocketchat-server.rocketchat-caddy.
Mar 29 08:03:45 Rocket rocketchat-server.rocketchat-caddy[2229]: Activating privacy features... 2018/03/29 08:03:45 open /root/snap/rocketchat-server/1238/.caddy/acme/acme-v02.api.letsencrypt
.org/sites/xxxxx.xxxxxxx.xxx/xxxxx.xxxxxxx.xxx.crt: no such file or directory
Mar 29 08:03:45 Rocket systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Main process exited, code=exited, status=1/FAILURE
Mar 29 08:03:45 Rocket systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Unit entered failed state.
Mar 29 08:03:45 Rocket systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Failed with result 'exit-code'.
Mar 29 08:03:46 Rocket systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Service hold-off time over, scheduling restart.
Mar 29 08:03:46 Rocket systemd[1]: Stopped Service for snap application rocketchat-server.rocketchat-caddy.
Mar 29 08:03:46 Rocket systemd[1]: snap.rocketchat-server.rocketchat-caddy.service: Start request repeated too quickly.
Mar 29 08:03:46 Rocket systemd[1]: Failed to start Service for snap application rocketchat-server.rocketchat-caddy.

[roberdaniel]

Same problem here.
Caddy can't find certificates on path ...acme-v02.api.letsencrypt.org/sites... folder is empty.
Acme V02 certificates are not currently being created?

[TwizzyDizzy]

@sefter25 in #10266 for better description:

After snap auto-refresh rocket-chat yesterday night, it was not possible to access our rocket chat server, because the "Caddyfile" was overwritten by auto-refresh. It´s normal? How we can stop/disable the auto-refresh?

Cheers
Thomas

[bfde974]

Same here, certificate crt and acme path GONE !!! #10266 does not resolve the bug, anyone ???

How restart the Caddy Lets encrypt process in order to get a new certificate ?

Thanks guys !

Bryce

[geekgonecrazy]

take a look in /root/snap/rocketchat-server/1232/.caddy and see what you have. Compare that with /root/snap/rocketchat-server/current/.caddy

If nothing in the second one copy contents from the first folder over and then restart caddy

sudo systemctl restart snap.rocketchat-server.rocketchat-caddy

[bfde974]

Thanks for the tip, it seems to work but I got a "502 Bad Gateway" when I want to browse to my https url...

Thanks !

Bryce

[bfde974]

Oups forgot to restart Rocketchat service too ! Sorry, everythings working !

Thanks !

Bryce

[geekgonecrazy]

No clue why it didn't copy those files over. Seems to be a bug in the snap system. I'll dig a bit and see if I can find the cause

[ricardojrgpimentel]

@geekgonecrazy I have less files in the 1232 directory then the current directory

[geekgonecrazy]

Still likely worth a try to copy them over

[geekgonecrazy]

Related:https://caddy.community/t/rocketchat-works-only-in-self-signed-mode/3595/2

[ricardojrgpimentel]

Still doesn't work, the logs show that its trying to look for
/root/snap/rocketchat-server/1238/.caddy/acme/acme-v02.api.letsencrypt.org/sites/example.com/example.com.crt

Non of the folders inside /root/snap/rocketchat-server/ have the v02 folder

[geekgonecrazy]

ok time for a bit of work around with the snap system. We need to basically follow the suggestion from there in the forum and get the log file. To do so we need to run something like:

sudo HOME=/root/snap/rocketchat-server/current /snap/rocketchat-server/current/bin/caddy -conf=/var/snap/rocketchat-server/current/Caddyfile -log /tmp/caddy.log

Should crash like normal... But now take and get contents of /tmp/caddy.log

Lets see what that yields us :)

[ricardojrgpimentel]

It doesn't say much

2018/03/30 18:29:00 [INFO][example.com] acme: Obtaining bundled SAN certificate 2018/03/30 18:29:01 [INFO][] Certificate written to disk: /root/snap/rocketchat-server/current/.caddy/acme/acme-v02.api.letsencrypt.org/sites/.crt

the sites folder is empty

[geekgonecrazy]

It really says /sites/.crt with no site name? :thinking:

[ricardojrgpimentel]

Yes

[geekgonecrazy]

I hope the caddy guys can give more insight. Because that doesn't make much sense to me. We may need to roll back to a previous caddy version :thinking:

[mholt]

There is a bug in the acmev2 branch of xenolf/lego, which is what Caddy uses for certificates, that apparently causes it to not report an error when there is one, or in any case, returns an empty Certificate and a nil error. I'm sure it will be resolved before too long. In the meantime, I've worked around it in the latest master branch of Caddy, which you can try. I'm not sure it gets to the root of the issue, unfortunately, but it won't write empty .crt files like that anymore.

Please build Caddy from the latest master and let me know how it goes!

(Edit: To clarify, if it's writing an empty .crt file, that _probably_ means it wasn't able to get a certificate that was needed, so the root cause is likely somewhere else like DNS configuration or firewall or something. That's my guess, but I can't be sure from the information here; running with the latest Caddy might provide more insight.)

[ricardojrgpimentel]

I guess we need to wait for a new snap update for this right?

[geekgonecrazy]

@mholt thanks for taking the time to post on this issue. I'll get another build of this going to give that a try.

@ricardojrgpimentel if you are stuck you can run:

sudo HOME=/root/snap/rocketchat-server/1232/ /snap/rocketchat-server/1232/bin/caddy -conf=/var/snap/rocketchat-server/current/Caddyfile -log /tmp/caddy.log

Basically executing the prior snap's caddy version.

1232 should be the number of the last revision.

[ricardojrgpimentel]

Thank you @geekgonecrazy it works!

[2E0PGS]

Hi I am running into the same issue. My RocketChat snap instance all of a sudden stopped working. It used to work just fine with the following config:

sudo cat /var/snap/rocketchat-server/current/Caddyfile
https://subdomain.mydomain.org
proxy / localhost:3000 {
  websocket
  transparent
}

Now I have to use self signed in order to get it working:

sudo cat /var/snap/rocketchat-server/current/Caddyfile
https://subdomain.mydomain.org
tls self_signed
proxy / localhost:3000 {
  websocket
  transparent
}

I am running the following versions:

/snap/rocketchat-server/current/bin/caddy -version Caddy 0.10.12 (unofficial)

lrwxrwxrwx 1 root root 4 May 23 20:20 current -> 1265

```Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial

34mI20180523-21:06:44.621(1) ➔ +------------------------------------------------------+
I20180523-21:06:44.621(1) ➔ | SERVER RUNNING |
I20180523-21:06:44.622(1) ➔ +------------------------------------------------------+
I20180523-21:06:44.622(1) ➔ | |
I20180523-21:06:44.623(1) ➔ | Rocket.Chat Version: 0.64.2 |
I20180523-21:06:44.623(1) ➔ | NodeJS Version: 8.9.4 - x64 |
I20180523-21:06:44.623(1) ➔ | Platform: linux |
I20180523-21:06:44.624(1) ➔ | Process Port: 3000 |
I20180523-21:06:44.624(1) ➔ | Site URL: https://subdomain.mydomain.org |
I20180523-21:06:44.625(1) ➔ | ReplicaSet OpLog: Enabled |
I20180523-21:06:44.625(1) ➔ | Commit Hash: 105715ef27 |
I20180523-21:06:44.626(1) ➔ | Commit Branch: HEAD |
I20180523-21:06:44.626(1) ➔ | |
I20180523-21:06:44.626(1) ➔ +------------------------------------------------------+

While poking around in that folder I just discovered something. I am not sure if one is simply a template but my Caddy files are different in these two locations:

cat /snap/rocketchat-server/current/bin/Caddyfile
http://:8080
proxy / localhost:3000 {
websocket
transparent
}

sudo cat /var/snap/rocketchat-server/current/Caddyfile
https://subdomain.mydomain.org
tls self_signed
proxy / localhost:3000 {
websocket
transparent
}
```

Anyway I have the same problem and most of my troubleshooting so far has happened over in this issue: https://github.com/RocketChat/Rocket.Chat/issues/8539

[mholt]

The latest versions have some bug fixes: https://github.com/mholt/caddy/releases

https://caddyserver.com/download

[2E0PGS]

@mholt I am running this via a snap on Ubuntu 16.04 is there a dev / unstable snap upstream I can switch to in order to use those changes on my LTS box? Is there a planned stable release for xenial with those "fixes"?

[mholt]

I dunno, you'll have to ask the RocketChat team.

[2E0PGS]

Appears there isn't a newer build on edge in fact its behind. Candidate is same as stable:

snap-id:   wdBUbiEuMNHmAHLBCXQXOcXaOCvbWS1e
tracking:  stable
refreshed: 2018-05-19T00:07:25+01:00
installed:   0.64.2         (1265) 218MB -
channels:                          
  stable:    0.64.2         (1265) 218MB -
  candidate: 0.64.2         (1265) 218MB -
  beta:      ↑                           
  edge:      0.59.0-develop (1163) 176MB -

[geekgonecrazy]

Ah perfect there is a newer version released of caddy. I'll give that a try

[geekgonecrazy]

New version of the snap should be in stable channel with caddy 0.11.0 in it. Please let me know if you guys still have the issue.

Going to go ahead and close this issue.

[2E0PGS]

Hi the service still will crash but it also appears port 443 is not being used:

NMAP:

PORT     STATE SERVICE
22/tcp   open  ssh
3000/tcp open  ppp

CADDYFILE:

https://subdomain.mydomain.org
proxy / localhost:3000 {
  websocket
  transparent
}

It seems to use port 8080 still once I run service xxx status on the caddy snap.

[geekgonecrazy]

What are your caddy logs?

journalctl -u snap.rocketchat-server.rocketchat-caddy