Okd: OKD 4.4: Installation is complete, but "console" failed to start

[danielyeap]

Hi All,
I had successfully installed OKD4.4 on a local cluster.

[root@okd4-services ~]# oc version
Client Version: 4.4.0-0.okd-2020-03-28-092308
Server Version: 4.4.0-0.okd-2020-03-28-092308
Kubernetes Version: v1.17.1
[root@okd4-services ~]# oc get nodes
NAME STATUS ROLES AGE VERSION
okd4-compute-1.lab.xyz.com Ready worker 150m v1.17.1
okd4-compute-2.lab.xyz.com Ready worker 150m v1.17.1
okd4-compute-3.lab.xyz.com Ready worker 144m v1.17.1
okd4-control-plane-1.lab.xyz.com Ready master,worker 3h1m v1.17.1
okd4-control-plane-2.lab.xyz.com Ready master,worker 3h v1.17.1
okd4-control-plane-3.lab.xyz.com Ready master,worker 173m v1.17.1
[root@okd4-services ~]# oc get pods --all-namespaces | egrep 'oauth|console'
openshift-authentication oauth-openshift-867d4dd94b-8lkz2 1/1 Running 0 91m
openshift-authentication oauth-openshift-867d4dd94b-zrh2f 1/1 Running 0 91m
openshift-console-operator console-operator-8499d75d4-zl5lk 1/1 Running 0 107m
openshift-console console-5444f4b554-ztqll 0/1 Running 22 116m
openshift-console console-55d6dd846f-nv7kn 0/1 Running 30 165m
openshift-console console-7f957cb8c7-lq2hf 0/1 CrashLoopBackOff 31 172m
openshift-console downloads-6c6b797c84-77mjz 1/1 Running 0 173m
openshift-console downloads-6c6b797c84-wnhm8 1/1 Running 0 173m
[root@okd4-services ~]#

However, the "console" refused to startup with error:
2020-04-12T09:41:46Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.lab.xyz.com/oauth/token failed: Head https://oauth-openshift.apps.lab.xyz.com: EOF
2020-04-12T09:41:56Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.lab.xyz.com/oauth/token failed: Head https://oauth-openshift.apps.lab.xyz.com: EOF
2020-04-12T09:42:06Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.lab.xyz.com/oauth/token failed: Head https://oauth-openshift.apps.lab.xyz.com: EOF

Routes info:
[root@okd4-services ~]# oc get routes -n openshift-console
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
console console-openshift-console.apps.lab.xyz.com console https reencrypt/Redirect None
downloads downloads-openshift-console.apps.lab.xyz.com downloads http edge/Redirect None
[root@okd4-services ~]# oc describe routes/console -n openshift-console
Name: console
Namespace: openshift-console
Created: 3 hours ago
Labels: app=console
Annotations: openshift.io/host.generated=true
Requested Host: console-openshift-console.apps.lab.xyz.com
exposed on router default (host apps.lab.xyz.com) 3 hours ago
Path:
TLS Termination: reencrypt
Insecure Policy: Redirect
Endpoint Port: https

Service: console
Weight: 100 (100%)
Endpoints:
[root@okd4-services ~]#

I can resolve the "console" and "outh" addresses any of my "contral-place" nodes without any problem:
[core@okd4-control-plane-1 ~]$ dig console-openshift-console.apps.lab.xyz.com

; <<>> DiG 9.11.14-RedHat-9.11.14-2.fc31 <<>> console-openshift-console.apps.lab.xyz.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14390
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: e727abaf4b9e46e239e7ff055e92e41bb0fc0ca68347c3c4 (good)
;; QUESTION SECTION:
;console-openshift-console.apps.lab.xyz.com. IN A

;; ANSWER SECTION:
console-openshift-console.apps.lab.xyz.com. 604800 IN A 192.168.0.208

;; AUTHORITY SECTION:
xyz.com. 604800 IN NS okd4-services.xyz.com.

;; ADDITIONAL SECTION:
okd4-services.xyz.com. 604800 IN A 192.168.0.208

;; Query time: 1 msec
;; SERVER: 192.168.0.208#53(192.168.0.208)
;; WHEN: Sun Apr 12 09:49:15 UTC 2020
;; MSG SIZE rcvd: 163

[core@okd4-control-plane-1 ~]$
[core@okd4-control-plane-1 ~]$ dig oauth-openshift.apps.lab.xyz.com

; <<>> DiG 9.11.14-RedHat-9.11.14-2.fc31 <<>> oauth-openshift.apps.lab.xyz.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48638
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 8b1dbd82ded79d0a5df7c07c5e92e4b2aaff4974e31ca513 (good)
;; QUESTION SECTION:
;oauth-openshift.apps.lab.xyz.com. IN A

;; ANSWER SECTION:
oauth-openshift.apps.lab.xyz.com. 604800 IN A 192.168.0.208

;; AUTHORITY SECTION:
xyz.com. 604800 IN NS okd4-services.xyz.com.

;; ADDITIONAL SECTION:
okd4-services.xyz.com. 604800 IN A 192.168.0.208

;; Query time: 1 msec
;; SERVER: 192.168.0.208#53(192.168.0.208)
;; WHEN: Sun Apr 12 09:51:46 UTC 2020
;; MSG SIZE rcvd: 153

[core@okd4-control-plane-1 ~]$

Excerpt of haproxy.conf that I believe is related to this issue:
frontend okd4_k8s_api_fe
bind :6443
default_backend okd4_k8s_api_be
mode tcp
option tcplog

backend okd4_k8s_api_be
balance source
mode tcp
# server okd4-bootstrap 192.168.0.200:6443 check
server okd4-control-plane-1 192.168.0.201:6443 check
server okd4-control-plane-2 192.168.0.202:6443 check
server okd4-control-plane-3 192.168.0.203:6443 check

Any idea what's wrong here?
Is it my haproxy?

Thanks.

[cgruver]

Paste the rest of your HA Proxy config. I believe that 443 is the relevant port here.

Is firewalld running on your router?

If so, did you open the necessary ports:

firewall-cmd --add-port=80/tcp --permanent
firewall-cmd --add-port=443/tcp --permanent
firewall-cmd --add-port=6443/tcp --permanent
firewall-cmd --add-port=22623/tcp --permanent
firewall-cmd --reload

[danielyeap]

Hi @cgruver
What "router" are you referring to?
I am using my home network router to route traffic.

The Openshift/Kubernetes routers are as follow:
[root@okd4-services journals]# oc get pods --all-namespaces -o wide | grep router
openshift-ingress router-default-5584f58f4f-dcrh8 1/1 Running 0 5h6m 192.168.0.202 okd4-control-plane-2.lab.xyz.com
openshift-ingress router-default-5584f58f4f-sd964 1/1 Running 0 5h6m 192.168.0.203 okd4-control-plane-3.lab.xyz.com
[root@okd4-services journals]#

There are no "firewalld" running on the control-plane nodes (192.168.0.202 and 192.168.0.203).

The "okd4-services" node which hosts the DNS server does not have firewalld running:
[root@okd4-services journals]# hostname
okd4-services.xyz.com
[root@okd4-services journals]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
[root@okd4-services journals]#

Here's my complete haproxy.conf =>

[root@okd4-services journals]# cat /etc/haproxy/haproxy.cfg

Global settings

---------------------------------------------------------------------

global
maxconn 20000
log /dev/log local0 info
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
user haproxy
group haproxy
daemon

# turn on stats unix socket
stats socket /var/lib/haproxy/stats

---------------------------------------------------------------------

common defaults that all the 'listen' and 'backend' sections will

use if not designated in their block

---------------------------------------------------------------------

defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 300s
timeout server 300s
timeout http-keep-alive 10s
timeout check 10s
maxconn 20000

listen stats
bind :9000
mode http
stats enable
stats uri /

frontend okd4_k8s_api_fe
bind :6443
default_backend okd4_k8s_api_be
mode tcp
option tcplog

backend okd4_k8s_api_be
balance source
mode tcp
# server okd4-bootstrap 192.168.0.200:6443 check
server okd4-control-plane-1 192.168.0.201:6443 check
server okd4-control-plane-2 192.168.0.202:6443 check
server okd4-control-plane-3 192.168.0.203:6443 check

frontend okd4_machine_config_server_fe
bind :22623
default_backend okd4_machine_config_server_be
mode tcp
option tcplog

backend okd4_machine_config_server_be
balance source
mode tcp
# server okd4-bootstrap 192.168.0.200:22623 check
server okd4-control-plane-1 192.168.0.201:22623 check
server okd4-control-plane-2 192.168.0.202:22623 check
server okd4-control-plane-3 192.168.0.203:22623 check

frontend okd4_http_ingress_traffic_fe
bind :80
default_backend okd4_http_ingress_traffic_be
mode tcp
option tcplog

backend okd4_http_ingress_traffic_be
balance source
mode tcp
server okd4-compute-1 192.168.0.204:80 check
server okd4-compute-2 192.168.0.205:80 check
server okd4-compute-3 192.168.0.206:80 check

frontend okd4_https_ingress_traffic_fe
bind *:443
default_backend okd4_https_ingress_traffic_be
mode tcp
option tcplog

backend okd4_https_ingress_traffic_be
balance source
mode tcp
server okd4-compute-1 192.168.0.204:443 check
server okd4-compute-2 192.168.0.205:443 check
server okd4-compute-3 192.168.0.206:443 check

[cgruver]

Apologies, I meant the server where you are running HA Proxy.

If firewalls is running there, make sure the rules are in place for 80 and 443.

The EOF you are getting could be indicative of a connection refused...

[danielyeap]

"okd4-services" is the node where haproxy is running and its firewalld is disabled.

Anything wrong with my haproxy configuration and how can I check whether haproxy is working properly?

I tried to google for haproxy troubleshooting, but still unsure how so far.

Thanks.

[danielyeap]

Here's the route details for "oauth":
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
oauth-openshift oauth-openshift.apps.lab.xyz.com oauth-openshift 6443 passthrough/Redirect None
[root@okd4-services /]#

You can see that the "console" pods are running on nodes "okd4-control-plane-1" and "okd4-control-plane-3". It seems to work fine (except of course the certificate issue, but definitely not EOF problem):
openshift-console console-5444f4b554-ztqll 0/1 CrashLoopBackOff 169 16h 10.128.0.54 okd4-control-plane-1.lab.xyz.com
openshift-console console-55d6dd846f-nv7kn 0/1 CrashLoopBackOff 177 17h 10.131.0.39 okd4-control-plane-3.lab.xyz.com
openshift-console console-7f957cb8c7-lq2hf 0/1 Running 179 17h 10.131.0.12 okd4-control-plane-3.lab.xyz.com

To test, on "okd4-control-plane-1", I did a manual curl command to the "okd4-services" at port 6443 node where haproxy is:
[core@okd4-control-plane-1 ~]$ curl -vvv -k https://192.168.0.208:6443

• Trying 192.168.0.208:6443...
• TCP_NODELAY set
• Connected to 192.168.0.208 (192.168.0.208) port 6443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• successfully set certificate verify locations:
• CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• TLSv1.3 (IN), TLS handshake, Server hello (2):
• TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
• TLSv1.3 (IN), TLS handshake, Request CERT (13):
• TLSv1.3 (IN), TLS handshake, Certificate (11):
• TLSv1.3 (IN), TLS handshake, CERT verify (15):
• TLSv1.3 (IN), TLS handshake, Finished (20):
• TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.3 (OUT), TLS handshake, Certificate (11):
• TLSv1.3 (OUT), TLS handshake, Finished (20):
• SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
• ALPN, server accepted to use h2
• Server certificate:
• subject: CN=172.30.0.1
• start date: Apr 12 06:41:17 2020 GMT
• expire date: May 12 06:41:18 2020 GMT
• issuer: OU=openshift; CN=kube-apiserver-service-network-signer
• SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
• Using HTTP2, server supports multi-use
• Connection state changed (HTTP/2 confirmed)
• Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
• Using Stream ID: 1 (easy handle 0x561316a75200)

GET / HTTP/2
Host: 192.168.0.208:6443
User-Agent: curl/7.66.0
Accept: /

• TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
• Connection state changed (MAX_CONCURRENT_STREAMS == 2000)!
  < HTTP/2 403
  < audit-id: 7ed5e393-6859-4ffa-a1fd-4bb4efabc202
  < content-type: application/json
  < x-content-type-options: nosniff
  < content-length: 233
  < date: Mon, 13 Apr 2020 00:08:59 GMT
  <
  {
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {

},
"code": 403

• Connection #0 to host 192.168.0.208 left intact
  }[core@okd4-control-plane-1 ~]$

==========================

Any clue what could be wrong with the "console" pods?

I am really running out of ideas :(

[cgruver]

Have you looked at the logs to see why the console pods are crash-looping?

Try: oc logs console-7f957cb8c7-lq2hf -n openshift-console replacing console-7f957cb8c7-lq2hf with the actual name of one of the pods attempting to run.

[danielyeap]

Hi @cgruver
That was one of the first things I checked :)

Here you go =>

[root@okd4-services okd4]# oc logs console-5444f4b554-ztqll -n openshift-console
2020-04-13T23:20:36Z cmd/main: cookies are secure!
2020-04-13T23:20:36Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.lab.xyz.com/oauth/token failed: Head https://oauth-openshift.apps.lab.xyz.com: EOF
2020-04-13T23:20:46Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.lab.xyz.com/oauth/token failed: Head https://oauth-openshift.apps.lab.xyz.com: EOF
2020-04-13T23:20:56Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.lab.xyz.com/oauth/token failed: Head https://oauth-openshift.apps.lab.xyz.com: EOF

Thanks.

[cgruver]

What do you see in the logs for pods in openshift-authentication-operator and openshift-authentication? I wonder if either the operator, or the auth pods themselves are unhealthy.

Are you still using the temporary kubeadmin user, or did you provision an auth provider?

[vrutkovs]

Instead of looking into particular pods we'd rather have a look at all available info.

Please collect the info using oc adm must-gather and attach it to the issue

[cgruver]

@vrutkovs Thanks for stepping in.

[danielyeap]

Hi @cgruver @vrutkovs
I had executed the "oc adm must-gather" command, but there are some errors:

[root@okd4-services openshift]# oc adm must-gather
[must-gather ] OUT Using must-gather plugin-in image: registry.svc.ci.openshift.org/origin/4.4-2020-03-28-092308@sha256:a273f5ac7f1ad8f7ffab45205ac36c8dff92d9107ef3ae429eeb135fa8057b8b
[must-gather ] OUT namespace/openshift-must-gather-98drx created
[must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-qhd5j created
[must-gather ] OUT pod for plug-in image registry.svc.ci.openshift.org/origin/4.4-2020-03-28-092308@sha256:a273f5ac7f1ad8f7ffab45205ac36c8dff92d9107ef3ae429eeb135fa8057b8b created
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Gathering data for ns/openshift-cluster-version...
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Gathering data for ns/openshift-config...
[must-gather-2f2bb] POD Gathering data for ns/openshift-config-managed...
[must-gather-2f2bb] POD Gathering data for ns/openshift-authentication...
[must-gather-2f2bb] POD Gathering data for ns/openshift-authentication-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-ingress...
[must-gather-2f2bb] POD Gathering data for ns/openshift-cloud-credential-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-machine-api...
[must-gather-2f2bb] POD Gathering data for ns/openshift-console-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-console...
[must-gather-2f2bb] POD Gathering data for ns/openshift-cluster-storage-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-dns-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-dns...
[must-gather-2f2bb] POD Gathering data for ns/openshift-etcd-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-etcd...
[must-gather-2f2bb] POD Gathering data for ns/openshift-image-registry...
[must-gather-2f2bb] POD Gathering data for ns/openshift-ingress-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-insights...
[must-gather-2f2bb] POD Gathering data for ns/openshift-kube-apiserver-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-kube-apiserver...
[must-gather-2f2bb] POD Gathering data for ns/openshift-kube-controller-manager...
[must-gather-2f2bb] POD Gathering data for ns/openshift-kube-controller-manager-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-kube-scheduler...
[must-gather-2f2bb] POD Gathering data for ns/openshift-kube-scheduler-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-kube-storage-version-migrator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-kube-storage-version-migrator-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-machine-config-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-marketplace...
[must-gather-2f2bb] POD Gathering data for ns/openshift-monitoring...
[must-gather-2f2bb] POD Gathering data for ns/openshift-multus...
[must-gather-2f2bb] POD Gathering data for ns/openshift-sdn...
[must-gather-2f2bb] POD Gathering data for ns/openshift-network-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-cluster-node-tuning-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-apiserver-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-apiserver...
[must-gather-2f2bb] POD Gathering data for ns/openshift-controller-manager-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-controller-manager...
[must-gather-2f2bb] POD Gathering data for ns/openshift-cluster-samples-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift...
[must-gather-2f2bb] POD Gathering data for ns/openshift-operator-lifecycle-manager...
[must-gather-2f2bb] POD Gathering data for ns/openshift-service-ca-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-service-ca...
[must-gather-2f2bb] POD Gathering data for ns/openshift-service-catalog-apiserver-operator...
[must-gather-2f2bb] POD Gathering data for ns/openshift-service-catalog-controller-manager-operator...
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD error: errors ocurred while gathering data:
[must-gather-2f2bb] POD [skipping gathering routes.route.openshift.io/oauth-openshift due to error: a resource cannot be retrieved by name across all namespaces, skipping gathering services/oauth-openshift due to error: a resource cannot be retrieved by name across all namespaces, skipping gathering machineautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "machineautoscalers", skipping gathering clusterautoscalers.machine.openshift.io due to error: the server doesn't have a resource type "clusterautoscalers", skipping gathering clusterroles.rbac.authorization.k8s.io/system:registry due to error: clusterroles.rbac.authorization.k8s.io "system:registry" not found, skipping gathering clusterrolebindings.rbac.authorization.k8s.io/registry-registry-role due to error: clusterrolebindings.rbac.authorization.k8s.io "registry-registry-role" not found, skipping gathering serviceaccounts/registry due to error: serviceaccounts "registry" not found, skipping gathering configmaps/serviceca due to error: configmaps "serviceca" not found, skipping gathering configmaps/image-registry-certificates due to error: configmaps "image-registry-certificates" not found, skipping gathering secrets/image-registry-private-configuration due to error: secrets "image-registry-private-configuration" not found, skipping gathering daemonsets.apps/node-ca due to error: daemonsets.apps "node-ca" not found, skipping gathering services/image-registry due to error: services "image-registry" not found, skipping gathering deployments.apps/image-registry due to error: deployments.apps "image-registry" not found, skipping gathering roles.rbac.authorization.k8s.io/cloud-provider-config-reader due to error: roles.rbac.authorization.k8s.io "cloud-provider-config-reader" not found, skipping gathering all/openshift-monitoring due to error: the server doesn't have a resource type "all", skipping gathering namespaces/openshift-service-catalog-apiserver due to error: namespaces "openshift-service-catalog-apiserver" not found, skipping gathering apiservices.apiregistration.k8s.io/v1beta1.servicecatalog.k8s.io due to error: apiservices.apiregistration.k8s.io "v1beta1.servicecatalog.k8s.io" not found, skipping gathering namespaces/openshift-service-catalog-controller-manager due to error: namespaces "openshift-service-catalog-controller-manager" not found]
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Gathering data for ns/default...
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Gathering data for ns/openshift...
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Gathering data for ns/kube-system...
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Gathering data for ns/openshift-etcd...
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD Wrote inspect data to must-gather.
[must-gather-2f2bb] POD WARNING: Collecting one or more audit logs on ALL masters in your cluster. This could take a large amount of time.
[must-gather-2f2bb] POD /usr/bin/oc adm node-logs okd4-control-plane-1.lab.okd4test.com --path=openshift-apiserver/audit.log
[must-gather-2f2bb] POD /usr/bin/oc adm node-logs okd4-control-plane-2.lab.okd4test.com --path=openshift-apiserver/audit.log
[must-gather-2f2bb] POD /usr/bin/oc adm node-logs okd4-control-plane-3.lab.okd4test.com --path=openshift-apiserver/audit.log
[must-gather-2f2bb] POD INFO: Audit logs for openshift-apiserver collected.
[must-gather-2f2bb] POD WARNING: Collecting one or more audit logs on ALL masters in your cluster. This could take a large amount of time.
[must-gather-2f2bb] POD /usr/bin/oc adm node-logs okd4-control-plane-1.lab.okd4test.com --path=kube-apiserver/audit-2020-04-14T14-26-01.440.log
[must-gather-2f2bb] POD /usr/bin/oc adm node-logs okd4-control-plane-1.lab.okd4test.com --path=kube-apiserver/audit.log
[must-gather-2f2bb] POD /usr/bin/oc adm node-logs okd4-control-plane-2.lab.okd4test.com --path=kube-apiserver/audit.log
[must-gather-2f2bb] POD /usr/bin/oc adm node-logs okd4-control-plane-3.lab.okd4test.com --path=kube-apiserver/audit.log
[must-gather-2f2bb] POD INFO: Audit logs for kube-apiserver collected.
[must-gather-2f2bb] POD WARNING: Collecting one or more service logs on ALL master nodes in your cluster. This could take a large amount of time.
[must-gather-2f2bb] POD INFO: Collecting host service logs for kubelet
[must-gather-2f2bb] POD INFO: Collecting host service logs for crio
[must-gather-2f2bb] POD INFO: Waiting for worker host service log collection to complete ...
[must-gather-2f2bb] POD INFO: Worker host service log collection to complete.
[must-gather-2f2bb] OUT waiting for gather to complete
[must-gather-2f2bb] OUT downloading gather output
[must-gather-2f2bb] OUT gather output not downloaded: mkdir must-gather.local.5189690871901124113: no such file or directory
[must-gather-2f2bb] OUT
[must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-qhd5j deleted
[must-gather ] OUT namespace/openshift-must-gather-98drx deleted
error: unable to download output from pod must-gather-2f2bb: mkdir must-gather.local.5189690871901124113: no such file or directory

Must-gather file can be downloaded from =>
https://drive.google.com/file/d/1cmC2So2hGXnm3h3tu0q1VnOwbmSgeMnl/view?usp=sharing

Thank you so much!!

[danielyeap]

Here's the logs of the openshift-authentication-operator (just in case) =>

https://drive.google.com/file/d/19hpfW5z0KNka2L2WE-3UZvCDVIsYVrx-/view?usp=sharing

[danielyeap]

Logs of one of the "oauth" pods =>

[root@okd4-services tmp]# oc logs oauth-openshift-b48c56756-h74rm -n openshift-authentication
Copying system trust bundle
W0414 13:59:38.071223 1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0414 13:59:38.071539 1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" due to: configmap "extension-apiserver-authentication" not found
I0414 13:59:38.104833 1 secure_serving.go:51] Forcing use of http/1.1 only
I0414 13:59:38.105127 1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0414 13:59:38.105179 1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0414 13:59:38.105240 1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0414 13:59:38.105276 1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0414 13:59:38.105918 1 tlsconfig.go:179] loaded serving cert ["serving-cert::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.crt::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.key"]: "oauth-openshift.openshift-authentication.svc" [serving] validServingFor=[oauth-openshift.openshift-authentication.svc,oauth-openshift.openshift-authentication.svc.cluster.local] issuer="openshift-service-serving-signer@1586872102" (2020-04-14 13:58:52 +0000 UTC to 2022-04-14 13:58:53 +0000 UTC (now=2020-04-14 13:59:38.105869929 +0000 UTC))
I0414 13:59:38.106272 1 dynamic_serving_content.go:129] Starting serving-cert::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.crt::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.key
I0414 13:59:38.106273 1 named_certificates.go:52] loaded SNI cert [1/"sni-serving-cert::/var/config/system/secrets/v4-0-config-system-router-certs/apps.lab.okd4test.com::/var/config/system/secrets/v4-0-config-system-router-certs/apps.lab.okd4test.com"]: ".apps.lab.okd4test.com" [serving] validServingFor=[.apps.lab.okd4test.com] issuer="ingress-operator@1586872563" (2020-04-14 13:56:05 +0000 UTC to 2022-04-14 13:56:06 +0000 UTC (now=2020-04-14 13:59:38.106261804 +0000 UTC))
I0414 13:59:38.106534 1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1586872778" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1586872778" (2020-04-14 12:59:37 +0000 UTC to 2021-04-14 12:59:37 +0000 UTC (now=2020-04-14 13:59:38.10648884 +0000 UTC))
I0414 13:59:38.106564 1 secure_serving.go:178] Serving securely on [::]:6443
I0414 13:59:38.106585 1 tlsconfig.go:219] Starting DynamicServingCertificateController
I0414 13:59:38.106586 1 dynamic_serving_content.go:129] Starting sni-serving-cert::/var/config/system/secrets/v4-0-config-system-router-certs/apps.lab.okd4test.com::/var/config/system/secrets/v4-0-config-system-router-certs/apps.lab.okd4test.com
I0414 13:59:38.205490 1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0414 13:59:38.205490 1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0414 13:59:38.206222 1 tlsconfig.go:157] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "aggregator-signer" [] issuer="" (2020-04-14 13:24:15 +0000 UTC to 2020-04-15 13:24:15 +0000 UTC (now=2020-04-14 13:59:38.206192207 +0000 UTC))
I0414 13:59:38.206651 1 tlsconfig.go:179] loaded serving cert ["serving-cert::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.crt::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.key"]: "oauth-openshift.openshift-authentication.svc" [serving] validServingFor=[oauth-openshift.openshift-authentication.svc,oauth-openshift.openshift-authentication.svc.cluster.local] issuer="openshift-service-serving-signer@1586872102" (2020-04-14 13:58:52 +0000 UTC to 2022-04-14 13:58:53 +0000 UTC (now=2020-04-14 13:59:38.206630488 +0000 UTC))
I0414 13:59:38.207169 1 named_certificates.go:52] loaded SNI cert [1/"sni-serving-cert::/var/config/system/secrets/v4-0-config-system-router-certs/apps.lab.okd4test.com::/var/config/system/secrets/v4-0-config-system-router-certs/apps.lab.okd4test.com"]: ".apps.lab.okd4test.com" [serving] validServingFor=[.apps.lab.okd4test.com] issuer="ingress-operator@1586872563" (2020-04-14 13:56:05 +0000 UTC to 2022-04-14 13:56:06 +0000 UTC (now=2020-04-14 13:59:38.207151946 +0000 UTC))
I0414 13:59:38.207468 1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1586872778" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1586872778" (2020-04-14 12:59:37 +0000 UTC to 2021-04-14 12:59:37 +0000 UTC (now=2020-04-14 13:59:38.207453582 +0000 UTC))
I0414 13:59:38.208114 1 tlsconfig.go:157] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "admin-kubeconfig-signer" [] issuer="" (2020-04-14 13:24:11 +0000 UTC to 2030-04-12 13:24:11 +0000 UTC (now=2020-04-14 13:59:38.20809795 +0000 UTC))
I0414 13:59:38.208181 1 tlsconfig.go:157] loaded client CA [1/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubelet-signer" [] issuer="" (2020-04-14 13:24:16 +0000 UTC to 2020-04-15 13:24:16 +0000 UTC (now=2020-04-14 13:59:38.20816726 +0000 UTC))
I0414 13:59:38.208204 1 tlsconfig.go:157] loaded client CA [2/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-control-plane-signer" [] issuer="" (2020-04-14 13:24:17 +0000 UTC to 2021-04-14 13:24:17 +0000 UTC (now=2020-04-14 13:59:38.208194171 +0000 UTC))
I0414 13:59:38.208224 1 tlsconfig.go:157] loaded client CA [3/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-apiserver-to-kubelet-signer" [] issuer="" (2020-04-14 13:24:17 +0000 UTC to 2021-04-14 13:24:17 +0000 UTC (now=2020-04-14 13:59:38.208214118 +0000 UTC))
I0414 13:59:38.208244 1 tlsconfig.go:157] loaded client CA [4/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubelet-bootstrap-kubeconfig-signer" [] issuer="" (2020-04-14 13:24:13 +0000 UTC to 2030-04-12 13:24:13 +0000 UTC (now=2020-04-14 13:59:38.208234406 +0000 UTC))
I0414 13:59:38.208266 1 tlsconfig.go:157] loaded client CA [5/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kube-csr-signer_@1586872100" [] issuer="kubelet-signer" (2020-04-14 13:48:19 +0000 UTC to 2020-04-15 13:24:16 +0000 UTC (now=2020-04-14 13:59:38.208254343 +0000 UTC))
I0414 13:59:38.208285 1 tlsconfig.go:157] loaded client CA [6/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "aggregator-signer" [] issuer="" (2020-04-14 13:24:15 +0000 UTC to 2020-04-15 13:24:15 +0000 UTC (now=2020-04-14 13:59:38.208275864 +0000 UTC))
I0414 13:59:38.208562 1 tlsconfig.go:179] loaded serving cert ["serving-cert::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.crt::/var/config/system/secrets/v4-0-config-system-serving-cert/tls.key"]: "oauth-openshift.openshift-authentication.svc" [serving] validServingFor=[oauth-openshift.openshift-authentication.svc,oauth-openshift.openshift-authentication.svc.cluster.local] issuer="openshift-service-serving-signer@1586872102" (2020-04-14 13:58:52 +0000 UTC to 2022-04-14 13:58:53 +0000 UTC (now=2020-04-14 13:59:38.208550028 +0000 UTC))
I0414 13:59:38.208906 1 named_certificates.go:52] loaded SNI cert [1/"sni-serving-cert::/var/config/system/secrets/v4-0-config-system-router-certs/apps.lab.okd4test.com::/var/config/system/secrets/v4-0-config-system-router-certs/apps.lab.okd4test.com"]: ".apps.lab.okd4test.com" [serving] validServingFor=[.apps.lab.okd4test.com] issuer="ingress-operator@1586872563" (2020-04-14 13:56:05 +0000 UTC to 2022-04-14 13:56:06 +0000 UTC (now=2020-04-14 13:59:38.208891829 +0000 UTC))
I0414 13:59:38.209193 1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1586872778" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1586872778" (2020-04-14 12:59:37 +0000 UTC to 2021-04-14 12:59:37 +0000 UTC (now=2020-04-14 13:59:38.209182043 +0000 UTC))
[root@okd4-services tmp]#

[danielyeap]

What do you see in the logs for pods in openshift-authentication-operator and openshift-authentication? I wonder if either the operator, or the auth pods themselves are unhealthy.

Are you still using the temporary kubeadmin user, or did you provision an auth provider?

I did not provision any auth provider. The cluster was just up and I wanted to try to access the web console.

I started from scratch again (except the services node) and deployed a total new installation/deployment and the problem remains.

[danielyeap]

Hi @cgruver @vrutkovs
Any tips on how to fix this problem? Thanks.

[cgruver]

Clarification: In your posts above, did you intentionally redact your cluster domain to apps.lab.xyz.com rather than apps.lab.okd4test.com? Your certs are signed for apps.lab.okd4test.com

[danielyeap]

Clarification: In your posts above, did you intentionally redact your cluster domain to apps.lab.xyz.com rather than apps.lab.okd4test.com? Your certs are signed for apps.lab.okd4test.com

Hi @cgruver, I rebuilt the cluster and it's now "okd4test.com".
Both clusters (the old and new) exhibited same issues (same error) with the web console.
Hope that clarifies.

[cgruver]

Hey, I just noticed something that I missed before in your posts. From the output of your oc get nodes your master nodes are also workers.

okd4-compute-1.lab.xyz.com Ready worker 150m v1.17.1
okd4-compute-2.lab.xyz.com Ready worker 150m v1.17.1
okd4-compute-3.lab.xyz.com Ready worker 144m v1.17.1
okd4-control-plane-1.lab.xyz.com Ready master,worker 3h1m v1.17.1
okd4-control-plane-2.lab.xyz.com Ready master,worker 3h v1.17.1
okd4-control-plane-3.lab.xyz.com Ready master,worker 173m v1.17.1

Your HA Proxy is set up to route port 443 only to the worker nodes.

I bet your Ingress operator is running on Master nodes! You can test this by just adding your Master nodes to the HA Proxy config for ports 443 and 80.

Or, remove the worker designation from your Master nodes like this:

oc patch scheduler cluster --patch '{"spec":{"mastersSchedulable":false}}' --type=merge

If you do that, you will need to make sure that the Ingress pods are evicted from the master and move to the workers.

[cgruver]

I took a slightly different approach with my cluster. I am letting the Masters also be my Infrastructure nodes, since they are fairly fat, and my cluster is lightly loaded.

To do that, you will want to reconfigure HAProxy to direct all traffic to your Master nodes.

Configure the nodes like this:

Label the master nodes as Infra nodes:

for i in 1 2 3
do
  oc label nodes okd4-control-plane-${i}.lab.xyz.com node-role.kubernetes.io/infra=""
done

Patch the Ingress operator to use Infra nodes for pod placement:

oc patch -n openshift-ingress-operator ingresscontroller default --patch '{"spec":{"nodePlacement":{"nodeSelector":{"matchLabels":{"node-role.kubernetes.io/infra":""}}}}}' --type=merge

Check status of Ingress operator managed pods:

 oc get pod -n openshift-ingress -o wide

Remove worker from masters:

oc patch scheduler cluster --patch '{"spec":{"mastersSchedulable":false}}' --type=merge

[danielyeap]

Hi @cgruver
Thanks for your feedback!!! My cluster is fully working now (at least I can login to web console).

I have a few questions:
(a) Why did the 3 master nodes started with worker role too?
I installed with the following manifest =>
[cluster-scheduler-02-config.yml]
apiVersion: config.openshift.io/v1
kind: Scheduler
metadata:
creationTimestamp: null
name: cluster
spec:
mastersSchedulable: false
policy:
name: ""
status: {}

(b) As for your workaround, I guess the ingress controller will no longer worker should they be restarted because you turn off scheduling on the master nodes which are now also the infra node needed by the ingress controllers. Right?

Thanks so much for your help!!!

[cgruver]

The approach that I took with my cluster isn't really a work around so much as an intentional design of my cluster. The challenge is that if you have very many worker nodes, then your external load balancer needs to be aware of all of them for it to find the Ingress pods. Every time you add a new worker you have to modify your load balancer accordingly. The solution for this in an OCP cluster is to create 2 or 3 nodes that are designated as infra nodes and create a scheduling policy for the Ingress pods, registry pods, etc... so that they are provisioned on the infra nodes.

In my case, my master nodes have plenty of compute and RAM resources available, so I just designate them as infra nodes.

My master nodes are schedulable during install because I deploy them first with zero workers. The OpenShift installer will automatically make masters shcedulable if you do not have any worker nodes designated in your install-config.yaml. That may be what happened in your case.