Cockpit: Cockpit does not start after restart

[sebiweise]

Freshly installed version of Cockpit on Debian 9.5 just won't start. After restarting my servers the status of service cockpit status is only inactive. When I run service cockpit start, the output of service cockpit status is OK but my web interface is not available on both servers.
The error also occurs when I install Cockpit on Debian and do not make any changes.

[martinpitt]

cockpit.service is not supposed to auto-start on boot. It is activated on demand with socket activation. The unit that should be running is cockpit.socket.

"Not available on both servers" needs more information. What exactly do you see? What's the output of sudo systemctl status cockpit.service after a connection attempt? Any journal messages, firewall, etc?

[sebiweise]

"Not available on both servers" I have a vServer and Home-Server. Cockpit's running on both. Via the vServer I go to the Cockpit web interface and have integrated the Home-Server there.

● cockpit.service - Cockpit Web Service Loaded: loaded (/lib/systemd/system/cockpit.service; static; vendor preset: enabled) Active: inactive (dead) Docs: man:cockpit-ws(8)

firewalld is active. But I added firewall-cmd --add-service cockpit

[sebiweise]

I have to add, that I configured Cockpit behind a Apache2 reverse Proxy.

_/etc/cockpit/cockpit.conf_

[WebService]
Origins = https://cp.<domain> http://127.0.0.1:9090
ProtocolHeader = X-Forwarded-Proto
AllowUnencrypted = true

and my Apache2 site-conf

<IfModule mod_ssl.c>
<VirtualHost *:443>
        ServerName <domain>
        SSLCertificateFile /etc/letsencrypt/live/<domain>/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/<domain>/privkey.pem
        Include /etc/letsencrypt/options-ssl-apache.conf

        ProxyPreserveHost On
        ProxyRequests Off

        RewriteEngine On
        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /(.*)           ws://127.0.0.1:9090/$1 [P,L]
        RewriteCond %{HTTP:Upgrade} !=websocket [NC]
        RewriteRule /(.*)           http://127.0.0.1:9090/$1 [P,L]

        ProxyPass / http://127.0.0.1:9090/
        ProxyPassReverse / http://127.0.0.1:9090/
</VirtualHost>
</IfModule>

[sebiweise]

_Update:_
When I disable firewalld and then reboot the vServer, everything is working fine. Also the second Server with still activated firewalld is controllable without problems.
I think that the firewall is blocking Apache2. When it is active, all my Websites are going down.
Any ideas?

[martinpitt]

firewall-cmd --add-service only applies to the current boot. If you want it to persist, you need to add --permanent as well. Confusing, I know, but not something which we can influence on the Cockpit side.

[sebiweise]

Adding
firewall-cmd --add-service cockpit --permanent
and
firewall-cmd --add-service http --permanent
and
firewall-cmd --add-service https --permanent
didn´t work for me first, but after a reboot everything is working fine 😃
Thank u.

[martinpitt]

Right, --permanent does not affect the currently running instance, that's the --add-service without the permanent option that you used before. So if you want to add the service both "now" and "after reboots", you need to run both variants.

Great to hear that it works now!

[DavidFW1960]

Just installed cockpit.
Added repo to /etc/apt/sources.list.d/backports.list and did a apt install cockpit cockpit-docker cockpit-machines. Debian Stretch 9.5

david@debian:~$ systemctl status cockpit
● cockpit.service - Cockpit Web Service
Loaded: loaded (/lib/systemd/system/cockpit.service; static; vendor preset: enabled)
Active: inactive (dead)
Docs: man:cockpit-ws(8)

So same error. Dead.
I didn't reboot the server.. was I supposed to?
I have not installed or configured a firewall that I know of. Just did a Debian install.. no desktop.
cannot load localip:9090 in a browser.
This server runs headless.

[martinpitt]

@DavidFW1960 : This sounds rather unrelated to the original issue here, where the problem was a missing firewall config. Is systemctl status cockpit.socket running? Does it work when you are on the server (with ssh) and run curl -k https://localhost:9090? (You should see some HTML stuff, not a 404 or "Connection refused")

I didn't reboot the server.. was I supposed to?

No, not necessary.

[DavidFW1960]

I see this:

david@debian:~$ curl -k https://localhost:9090
curl: (35) Unknown SSL protocol error in connection to localhost:9090
david@debian:~$ systemctl status cockpit.socket
● cockpit.socket - Cockpit Web Service Socket
   Loaded: loaded (/lib/systemd/system/cockpit.socket; enabled; vendor preset: enabled)
   Active: failed (Result: resources) since Fri 2018-10-12 11:19:08 AEDT; 6h ago
     Docs: man:cockpit-ws(8)
   Listen: [::]:9090 (Stream)
      CPU: 1ms
david@debian:~$

If I use the IP address instead of localhost:

david@debian:~$ curl -k https://10.90.11.100:9090
curl: (7) Failed to connect to 10.90.11.100 port 9090: Connection refused
david@debian:~$

Which is the same error I get in a browser on my PC (Win10)

[martinpitt]

Active: failed (Result: resources)

That means that there's something wrong with socket units on your system. Can you please check

sudo journalctl -u cockpit.socket

for any error messages?

Also, "Unknown SSL protocol error in connection to localhost:9090" sounds like there is already something else running on port 9090. What do you see in

sudo ss -tulpen | grep 9090

[DavidFW1960]

Dammit! You're right! Sabnzdb is listening on that port. Can I change the cockpit port?

david@debian:~$ sudo journalctl -u cockpit.socket
-- Logs begin at Mon 2018-09-10 16:21:48 AEST, end at Fri 2018-10-12 17:58:46 AEDT. --
Oct 12 11:19:08 debian systemd[1]: cockpit.socket: Failed to listen on sockets: Address already in use
Oct 12 11:19:08 debian systemd[1]: Starting Cockpit Web Service Socket.
Oct 12 11:19:08 debian systemd[1]: Failed to listen on Cockpit Web Service Socket.
Oct 12 11:19:08 debian systemd[1]: cockpit.socket: Unit entered failed state.
david@debian:~$ sudo ss -tulpen | grep 9090
tcp    LISTEN     0      128      :::9090                 :::*                   users:(("docker-proxy",pid=5579,fd=4)) ino:28229875 sk:da v6only:0 <->

[martinpitt]

@DavidFW1960 : Yes, you can change it. See https://cockpit-project.org/guide/latest/listen.html

[DavidFW1960]

Genius! Thankyou so much. Working now. I'm so used to docker telling me it cannot start a container when there is a conflict.

Thanks again.

[dandelionred]

I had the same problem with cockpit not available on reboot, but I could start it manually. I used such override snippet to prevent it from listening on all interfaces

[Socket]
ListenStream=
ListenStream=192.168.1.1:9090

The IP was not established yet (a static ip set with networkmanager) when cockpit starts. As per https://cockpit-project.org/guide/latest/listen.html I fixed it with FreeBind=yes appended to the snippet.

[saadelkenawy]

How could I add auto restart in daemon of cockpit service

[marusak]

How could I add auto restart in daemon of cockpit service

Sorry, what do you mean by auto restart? cockpit service is socket activated, so there is no need to restart it.

[saadelkenawy]

How could I add auto restart in daemon of cockpit service

Sorry, what do you mean by auto restart? cockpit service is socket activated, so there is no need to restart it.
@marusak

please advise according to my search to add a parameter to service of cockpit to ON-Restart = on-failure

Configures whether the service shall be restarted when the service process exits, is killed, or a timeout is reached. The service process may be the main service process, but it may also be one of the processes specified with ExecStartPre=, ExecStartPost=, ExecStop=, ExecStopPost=, or ExecReload=. When the death of the process is a result of systemd operation (e.g. service stop or restart), the service will not be restarted. Timeouts include missing the watchdog "keep-alive ping" deadline and a service start, reload, and stop operation timeouts.

Takes one of no, on-success, on-failure, on-abnormal, on-watchdog, on-abort, or always. If set to no (the default), the service will not be restarted. If set to on-success, it will be restarted only when the service process exits cleanly. In this context, a clean exit means an exit code of 0, or one of the signals SIGHUP, SIGINT, SIGTERM or SIGPIPE, and additionally, exit statuses and signals specified in SuccessExitStatus=. If set to on-failure, the service will be restarted when the process exits with a non-zero exit code, is terminated by a signal (including on core dump, but excluding the aforementioned four signals), when an operation (such as service reload) times out, and when the configured watchdog timeout is triggered. If set to on-abnormal, the service will be restarted when the process is terminated by a signal (including on core dump, excluding the aforementioned four signals), when an operation times out, or when the watchdog timeout is triggered. If set to on-abort, the service will be restarted only if the service process exits due to an uncaught signal not specified as a clean exit status. If set to on-watchdog, the service will be restarted only if the watchdog timeout for the service expires. If set to always, the service will be restarted regardless of whether it exited cleanly or not, got terminated abnormally by a signal, or hit a timeout.

[dandelionred]

@saadelkenawy I guess systemctl edit cockpit.service is what you're looking for.

[saadelkenawy]

@saadelkenawy I guess systemctl edit cockpit.service is what you're looking for.

I will try to do it and. will feedback @dandelionred