Cockpit: Cannot access via domain name after upgrading to Fedora 33

[Alion548]

Cockpit version: 229
OS:Linux fedora_33 5.8.13-300.fc33.x86_64

The same configuration is normal in Fedora 32.

[osiu97]

I managed to get it working on Fedora 33. To be honest at least in my case it was my mistake in nginx config. It looked same as yours so maybe it will help. For me it was trailing slash in proxy_pass directive which isn't supposed to be there. Plain config taken from wiki seems to work perfect now.

EDIT: I just checked, and it was one of my problems. It actually doesn't seem to work when there is more than one server running on the same port in nginx. (Maybe something SNI related?) I don't think it's something fedora 33 related (although i do not have the means to test it on something else) since it works fine when there's only cockpit reverse proxy server running on nginx. You can test it with config like this
```user nginx;
worker_processes 2;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

Load dynamic modules. See /usr/share/nginx/README.dynamic.

include /usr/share/nginx/modules/*.conf;

events {
worker_connections 768;
}

http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log  /var/log/nginx/access.log  main;

sendfile            on;
tcp_nopush          on;
tcp_nodelay         on;
keepalive_timeout   65;
types_hash_max_size 2048;

include             /etc/nginx/mime.types;
default_type        application/octet-stream;

# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.

server {
listen 80;
listen 443 ssl;
server_name cockpit.domain.top;
ssl_certificate /etc/letsencrypt/live/cockpit.domain.top/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/cockpit.domain.top/privkey.pem; # managed by Certbot

location / {
    # Required to proxy the connection to Cockpit
    proxy_pass https://192.168.1.100:9090;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Required for web sockets to function
    proxy_http_version 1.1;
    proxy_buffering off;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";

    # Pass ETag header from Cockpit to clients.
    # See: https://github.com/cockpit-project/cockpit/issues/5239
    gzip off;
}

}
}
and run it withnginx -g "daemon off;" -c /tmp/cockpit.conf```

[Alion548]

我设法得到它在 Fedora 33 上的工作。说实话， 至少就我而言， 这是我的错误， 在 nginx 配置。它看起来和你的一样， 所以也许会有所帮助。对我来说， 这是尾随斜proxy_pass指令， 这是不应该在那里。从维基取来的纯配置现在似乎很完美。

编辑：我刚刚检查，这是我的问题之一。当 nginx 中的同一端口上运行多台服务器时，它实际上似乎不起作用。（也许与斯尼有关？我不认为这是 Fedora 33 相关的东西 （虽然我没有手段测试它别的东西）， 因为它工作正常时， 只有驾驶舱反向代理服务器运行在 nginx 上。您可以使用这样的配置来测试它

worker_processes 2;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 768;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.

server {
    listen         80;
    listen         443 ssl;
    server_name    cockpit.domain.top;
    ssl_certificate /etc/letsencrypt/live/cockpit.domain.top/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/cockpit.domain.top/privkey.pem; # managed by Certbot

    location / {
        # Required to proxy the connection to Cockpit
        proxy_pass https://192.168.1.100:9090;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Required for web sockets to function
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Pass ETag header from Cockpit to clients.
        # See: https://github.com/cockpit-project/cockpit/issues/5239
        gzip off;
    }
}
}

并运行它nginx -g "daemon off;" -c /tmp/cockpit.conf

Use your nginx configuration and get the same error.

[osiu97]

If you didn't have anything else running on nginx, then you should post your logs here journalctl -f for me it looked like this

cockpit-bridge[27435]: received invalid HTTP request headers
cockpit-bridge[27435]: 0:1: http://packages/shell/index.html: received truncated HTTP response
cockpit-ws[27362]: /shell/index.html: external channel failed: protocol-error

Also make sure you have your origins configured in /etc/cockpit/cockpit.conf.
Is your nginx running on the same server you're trying to proxy to?
For me it is working now, but for some reason I had to give it access to exclusive port and disable HTTP/2 in nginx.

[mehmooda]

Not sure if related; But my connections which go through my nginx ingress normalises the headers to lowercase

GET /control HTTP/1.1\r\n
Host: [redacted]\r\n
X-Request-ID: baffc0232f142f36142d06547a413f89\r\n
X-Real-IP: [redacted]\r\n
X-Forwarded-For: [redacted]\r\n
X-Forwarded-Host: [redacted]\r\n
X-Forwarded-Port: 443\r\n
X-Forwarded-Proto: https\r\n
X-Scheme: https\r\n
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0\r\n
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n
accept-language: en-US,en;q=0.5\r\n
accept-encoding: gzip, deflate, br\r\n
upgrade-insecure-requests: 1\r\n
pragma: no-cache\r\n
cache-control: no-cache\r\n
cookie: cockpit=dj0yO2s9NGNhMDU4OGU1MDFkZDgzMWI5Yjc5ZDU1M2FkODlmN2M2OWE5ZmY3Y2UxZDZhYTIxNjU0ZTcwM2RjYjA1NTI1OQ==\r\n
\r\n

And results in protocol-error; the direct connection uses Title-Case for all the headers.

[Alion548]

If you didn't have anything else running on nginx, then you should post your logs here journalctl -f for me it looked like this

cockpit-bridge[27435]: received invalid HTTP request headers
cockpit-bridge[27435]: 0:1: http://packages/shell/index.html: received truncated HTTP response
cockpit-ws[27362]: /shell/index.html: external channel failed: protocol-error

Also make sure you have your origins configured in /etc/cockpit/cockpit.conf.
Is your nginx running on the same server you're trying to proxy to?
For me it is working now, but for some reason I had to give it access to exclusive port and disable HTTP/2 in nginx.

journalctl -f
10月 25 18:46:25 localhost cockpit-ws[11330]: cockpit-ws: Failed to open certificate file /run/cockpit/tls/e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855: No such file or directory
10月 25 18:46:25 localhost audit[14226]: USER_AUTH pid=14226 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:authentication grantors=pam_unix acct="root" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success'
10月 25 18:46:25 localhost audit[14226]: USER_ACCT pid=14226 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="root" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success'
10月 25 18:46:25 localhost audit[14226]: CRED_ACQ pid=14226 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_unix acct="root" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success'
10月 25 18:46:25 localhost cockpit-session[14226]: pam_ssh_add: Identity added: /root/.ssh/id_rsa ([email protected])
10月 25 18:46:25 localhost systemd-logind[850]: [🡕] New session 21 of user root.
10月 25 18:46:25 localhost systemd[1]: Started Session 21 of user root.
10月 25 18:46:25 localhost cockpit-session[14226]: pam_unix(cockpit:session): session opened for user root(uid=0) by (uid=0)
10月 25 18:46:25 localhost audit[14226]: USER_START pid=14226 uid=0 auid=0 ses=21 msg='op=PAM:session_open grantors=pam_selinux,pam_loginuid,pam_selinux,pam_keyinit,pam_ssh_add,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_umask,pam_lastlog acct="root" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success'
10月 25 18:46:25 localhost audit[14226]: CRED_REFR pid=14226 uid=0 auid=0 ses=21 msg='op=PAM:setcred grantors=pam_unix acct="root" exe="/usr/libexec/cockpit-session" hostname=? addr=? terminal=? res=success'
10月 25 18:46:25 localhost cockpit-bridge[14231]: received invalid HTTP request headers
10月 25 18:46:25 localhost cockpit-bridge[14231]: 0:1: http://packages/shell/index.html: received truncated HTTP response
10月 25 18:46:25 localhost cockpit-ws[11330]: /shell/index.html: external channel failed: protocol-error

server {
    listen 443 ssl http2;

    # SSL
    ssl_certificate     /usr/etc/cert/fullchain.cer;
    ssl_certificate_key /usr/etc/cert/website.key;


    location / {
        # Required to proxy the connection to Cockpit
        proxy_pass https://127.0.0.1:9090/;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Required for web sockets to function
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Pass ETag header from Cockpit to clients.
        # See: https://github.com/cockpit-project/cockpit/issues/5239
        gzip off;
    }
}

[nos1609]

Same here on ArchLinux right after update

[sriio]

Same problem also with version 230.

Temporary downgrade to version 228 to regain acces to my server.

[Alion548]

Suspect it's a problem with the 229 version of the shell functionality.
https://cockpit-project.org/blog/cockpit-229.html

[saivert]

The bad commit is: 462eda51dec9de4661d07ea275fd14f62372ff59
(ws: If injecting a element, ensure non-gzipped content)

And commenting out json_object_set_string_member (heads, "Accept-Encoding", "identity"); fixes the issue with not supporting http2 reverse proxy in front of it.

[Yoshi2889]

I doubt this is _the_ solution, but @saivert 's research got me thinking.

After adding the following line to my Nginx proxy config:

proxy_set_header Accept-Encoding identity;

The interface loads up fine again on my subdomain. proxy_pass_header doesn't seem to work.

[saivert]

What I find interesting is if I instead of commenting out that line change it to lowercase like so:
json_object_set_string_member (heads, "accept-encoding", "identity");

It also works.

Since http2 mandates all lowercase header names this must be investigated further. Either there is an issue with how nginx handles this or cockpit isn't understanding lowercase headers.

I see pervasive use of hash lookups in the codebase but aren't these case sensitive?

Btw. I'm not using proxy_pass_header at all in my nginx configuration. I'm using the "officially" recommended config stanza from the cockpit project for reverse proxying which is:

    location /cockpitadm/ {
        # Required to proxy the connection to Cockpit
        proxy_pass https://127.0.0.1:9090;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Required for web sockets to function
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Pass ETag header from Cockpit to clients.
        # See: https://github.com/cockpit-project/cockpit/issues/5239
        gzip off;
    }