Cockpit: Cockpit failed to start with error loading certs

[kxxoling]

Cockpit version: 251-1 amd64
OS: Ubuntu 20.20 Linux gen8 5.4.0-26-generic #30-Ubuntu SMP Mon Apr 20 16:58:30 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I've installed cockpit on Gen8 since a brand new Ubuntu 20.20 installed, and had cockpit started with SSL error. So I tried to reinstall cockpit to fix this invalid cert error. But after this, I cannot start cockpit with sudo systemctl start cockpit anymore.

$ sudo systemctl status cockpit

● cockpit.service - Cockpit Web Service
     Loaded: loaded (/lib/systemd/system/cockpit.service; static; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2020-05-21 04:06:34 UTC; 7min ago
TriggeredBy: ● cockpit.socket
       Docs: man:cockpit-ws(8)
    Process: 563960 ExecStartPre=/usr/sbin/remotectl certificate --ensure --user=root --group=cockpit-ws --selinux-type= (code=exited, status=0/SUCCESS)
    Process: 563961 ExecStart=/usr/lib/cockpit/cockpit-tls (code=exited, status=1/FAILURE)
   Main PID: 563961 (code=exited, status=1/FAILURE)

May 21 04:06:34 gen8 systemd[1]: cockpit.service: Main process exited, code=exited, status=1/FAILURE
May 21 04:06:34 gen8 systemd[1]: cockpit.service: Failed with result 'exit-code'.
May 21 04:06:34 gen8 systemd[1]: Starting Cockpit Web Service...
May 21 04:06:34 gen8 systemd[1]: Started Cockpit Web Service.
May 21 04:06:34 gen8 cockpit-tls[563961]: cockpit-tls: Could not locate server certificate: Error loading certificates from /etc/cockpit/ws-certs.d: Permission denied
May 21 04:06:34 gen8 systemd[1]: cockpit.service: Main process exited, code=exited, status=1/FAILURE
May 21 04:06:34 gen8 systemd[1]: cockpit.service: Failed with result 'exit-code'.
May 21 04:06:34 gen8 systemd[1]: cockpit.service: Start request repeated too quickly.
May 21 04:06:34 gen8 systemd[1]: cockpit.service: Failed with result 'exit-code'.
May 21 04:06:34 gen8 systemd[1]: Failed to start Cockpit Web Service.

$ journalctl -u cockpit

-- Logs begin at Mon 2020-05-04 05:17:32 UTC, end at Thu 2020-05-21 04:24:00 UTC. --
May 15 14:09:28 gen8 systemd[1]: Starting Cockpit Web Service...
May 15 14:09:28 gen8 remotectl[527778]: Generating temporary certificate using: sscg --quiet --lifetime 3650 --key-strength 2048 --cert-key-file /etc/cockpit/ws-certs.d/0-self-signed.cert --cert-file /etc/cockpit/ws-certs.d/0-self-signed>
May 15 14:09:28 gen8 remotectl[527778]: Error generating temporary dummy cert using sscg, falling back to openssl
May 15 14:09:28 gen8 remotectl[527778]: Generating temporary certificate using: openssl req -x509 -days 36500 -newkey rsa:2048 -keyout /etc/cockpit/ws-certs.d/0-self-signed.347AK0.tmp -keyform PEM -nodes -out /etc/cockpit/ws-certs.d/0-se>
May 15 14:09:28 gen8 systemd[1]: Started Cockpit Web Service.
May 15 14:10:58 gen8 systemd[1]: cockpit.service: Succeeded.
May 15 14:19:34 gen8 systemd[1]: Starting Cockpit Web Service...
May 15 14:19:34 gen8 systemd[1]: Started Cockpit Web Service.
May 15 14:19:57 gen8 systemd[1]: Stopping Cockpit Web Service...
May 15 14:19:57 gen8 systemd[1]: cockpit.service: Succeeded.
May 15 14:19:57 gen8 systemd[1]: Stopped Cockpit Web Service.
May 15 14:23:23 gen8 systemd[1]: Starting Cockpit Web Service...
May 15 14:23:23 gen8 systemd[1]: Started Cockpit Web Service.
May 15 14:24:53 gen8 systemd[1]: cockpit.service: Succeeded.
May 15 14:24:56 gen8 systemd[1]: Starting Cockpit Web Service...
May 15 14:24:56 gen8 systemd[1]: Started Cockpit Web Service.
May 15 14:25:15 gen8 cockpit-tls[529527]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:25:46 gen8 cockpit-tls[529527]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:25:46 gen8 cockpit-tls[529527]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:27:16 gen8 systemd[1]: cockpit.service: Succeeded.
May 15 14:38:02 gen8 systemd[1]: Starting Cockpit Web Service...
May 15 14:38:02 gen8 systemd[1]: Started Cockpit Web Service.
May 15 14:38:02 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:38:03 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:38:03 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:38:03 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:38:11 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:38:11 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:38:11 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:39:24 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: Decryption has failed.
May 15 14:39:28 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: Decryption has failed.
May 15 14:39:32 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: Decryption has failed.
May 15 14:39:37 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:39:37 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:40:40 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:40:40 gen8 cockpit-tls[530315]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
May 15 14:42:10 gen8 systemd[1]: cockpit.service: Succeeded.
May 20 07:34:15 gen8 systemd[1]: Starting Cockpit Web Service...
May 20 07:34:15 gen8 systemd[1]: Started Cockpit Web Service.
May 20 07:35:53 gen8 systemd[1]: cockpit.service: Succeeded.
May 20 07:40:42 gen8 systemd[1]: Starting Cockpit Web Service...
May 20 07:40:42 gen8 systemd[1]: Started Cockpit Web Service.
May 20 07:42:12 gen8 systemd[1]: cockpit.service: Succeeded.
May 20 07:45:38 gen8 systemd[1]: Starting Cockpit Web Service...
May 20 07:45:38 gen8 systemd[1]: Started Cockpit Web Service.
May 20 07:47:39 gen8 systemd[1]: cockpit.service: Succeeded.
May 20 08:03:05 gen8 systemd[1]: Starting Cockpit Web Service...
May 20 08:03:05 gen8 systemd[1]: Started Cockpit Web Service.
May 20 08:03:12 gen8 cockpit-tls[559075]: cockpit-tls: gnutls_handshake failed: Decryption has failed.
May 20 08:03:42 gen8 cockpit-tls[559075]: cockpit-tls: gnutls_handshake failed: Decryption has failed.
May 20 08:05:12 gen8 systemd[1]: cockpit.service: Succeeded.
May 20 08:09:23 gen8 systemd[1]: Starting Cockpit Web Service...
May 20 08:09:23 gen8 systemd[1]: Started Cockpit Web Service.
May 20 08:10:53 gen8 systemd[1]: cockpit.service: Succeeded.

$ sudo ls -al /etc/cockpit
drwx------   2 root root 4096 May 21 04:04 ws-certs.d
$ sudo ls -al /etc/cockpit/ws-certs.d
-rw-r----- 1 root cockpit-ws 2853 May 21 04:04 0-self-signed.cert

I think there's something strange with my /etc/cockpit folder, but have no idea what to do with it.

[martinpitt]

@kxxoling : Right, the /etc/cockpit/ws-certs.d/ folder is only accessible for root. It should be 0755. This smells like some umask problem -- do you have a very tight one for root, like 077?

[kxxoling]

@martinpitt I tried to cleanly reinstall the newest version in apt, and found there would be a machines.d folder under /etc/cockpit alongside of ws-certs.d. And cockpit-ws starts well now except the https problem:

$ curl https://localhost:9090
curl: (60) SSL certificate problem: self signed certificate

Isn't 0-self-signed.cert a legal ca cert? Or should I maintain the cert myself?

[martinpitt]

@kxxoling : It is a valid cert syntactically, but as the name says it is self-signed. So curl (same as your browser) won't accept it by default. You can either accept it anyway (curl -k or click the button in the browser), or better, add your own.

[kxxoling]

@martinpitt I got it. I'll try to use a Let's Encrypt signed one instead. This may solve the cert problem.

The service start error may be caused by myself in accident, but a full reinstallation could solve it. So I will close this issue.

[kxxoling]

Thanks for you help! @martinpitt :D