Okd: [AWS UPI] OKD 4.4 console fails to start with connection refused

[andygeorge]

OKD: 4.4.0-0.okd-2020-04-21-163702-beta4 (and older betas/releases)
Initial image: stable 31.20200407.3.0 (AWS AMI ami-07159149bd9fd748f)
100% reproducible: have tried a number of other OKD 4.4 releases, all results in the same issue
oc adm must-gather archive: https://keybase.pub/andygeorge/openshift/must-gather.local.8590515517328253782.tar.gz

Using the AWS UPI instructions, we're able to get a 3 master/3 worker cluster up, but install-complete eventually fails due to the console (and its pods) never coming up successfully:

[core@ip-10-127-12-110 ~]$ oc get pods -n openshift-console
NAME                         READY   STATUS    RESTARTS   AGE
console-5f7566d6b6-cfxsj     0/1     Running   3          11m
console-74ddfdc585-vzgp6     0/1     Running   3          11m
console-7c777c67c6-ts64q     0/1     Running   1          4m52s
downloads-7db88fb486-l77m9   1/1     Running   0          14m
downloads-7db88fb486-rsvgg   1/1     Running   0          14m

The behavior is similar to what was seen in #141, but instead of EOFs returned by oauth-openshift, we see refused connections - I assume nothing is binding to tcp/443:

[core@ip-10-127-12-110 ~]$ oc describe -n openshift-console pod/console-5f7566d6b6-cfxsj
Name:                 console-5f7566d6b6-cfxsj
Namespace:            openshift-console
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 ip-10-127-11-58.us-east-2.compute.internal/10.127.11.58
Start Time:           Fri, 24 Apr 2020 15:58:58 +0000
Labels:               app=console
                      component=ui
                      pod-template-hash=5f7566d6b6
Annotations:          console.openshift.io/console-config-version: 16007
                      console.openshift.io/default-ingress-cert-config-version: 12754
                      console.openshift.io/image: quay.io/openshift/okd-content@sha256:ae96918edcbe07424f38f9976132c8ebe694050ffb46cf526df3a0da68be7b8d
                      console.openshift.io/oauth-secret-version: 16027
                      console.openshift.io/proxy-config-version: 443
                      console.openshift.io/service-ca-config-version: 16014
                      console.openshift.io/trusted-ca-config-version: 16016
                      k8s.v1.cni.cncf.io/networks-status:
                        [{
                            "name": "openshift-sdn",
                            "interface": "eth0",
                            "ips": [
                                "10.128.0.40"
                            ],
                            "dns": {},
                            "default-route": [
                                "10.128.0.1"
                            ]
                        }]
                      openshift.io/scc: restricted
                      operator.openshift.io/pull-spec: quay.io/openshift/okd-content@sha256:ae96918edcbe07424f38f9976132c8ebe694050ffb46cf526df3a0da68be7b8d
Status:               Running
IP:                   10.128.0.40
IPs:
  IP:           10.128.0.40
Controlled By:  ReplicaSet/console-5f7566d6b6
Containers:
  console:
    Container ID:  cri-o://e070f86e644677eaf9313d36e58aa4a95a7f1b90f568f5ae6a272c0e992df861
    Image:         quay.io/openshift/okd-content@sha256:ae96918edcbe07424f38f9976132c8ebe694050ffb46cf526df3a0da68be7b8d
    Image ID:      quay.io/openshift/okd-content@sha256:ae96918edcbe07424f38f9976132c8ebe694050ffb46cf526df3a0da68be7b8d
    Port:          443/TCP
    Host Port:     0/TCP
    Command:
      /opt/bridge/bin/bridge
      --public-dir=/opt/bridge/static
      --config=/var/console-config/console-config.yaml
      --service-ca-file=/var/service-ca/service-ca.crt
    State:       Running
      Started:   Fri, 24 Apr 2020 16:09:07 +0000
    Last State:  Terminated
      Reason:    Error
      Message:   ec.com/oauth/token failed: Head https://oauth-openshift.apps.os-test.test.us-east-2.aws.int.kennasec.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-04-24T16:08:07Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.os-test.test.us-east-2.aws.int.kennasec.com/oauth/token failed: Head https://oauth-openshift.apps.os-test.test.us-east-2.aws.int.kennasec.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-04-24T16:08:22Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.os-test.test.us-east-2.aws.int.kennasec.com/oauth/token failed: Head https://oauth-openshift.apps.os-test.test.us-east-2.aws.int.kennasec.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-04-24T16:08:37Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.os-test.test.us-east-2.aws.int.kennasec.com/oauth/token failed: Head https://oauth-openshift.apps.os-test.test.us-east-2.aws.int.kennasec.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-04-24T16:08:47Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.os-test.test.us-east-2.aws.int.kennasec.com/oauth/token failed: Head https://oauth-openshift.apps.os-test.test.us-east-2.aws.int.kennasec.com: dial tcp 10.127.12.247:443: connect: connection refused

We also wish to run our master nodes as infra nodes, so I followed some instructions in #141, and got those to come up correctly, but that doesn't affect the openshift-console pods.

Not sure if this is actually a bug, and feels like we're just missing something simple in our configuration - any thoughts or ideas of what to try next?

[cgruver]

@andygeorge

Hey, I think I gave some incorrect advice in #141. If your masters were schedulable before, and you followed the instructions that I gave for making them into infra pods, I have since discovered that they still need to be schedulable... I killed one of the ingress pods to see if it would move to another master node. It did not... It was stuck pending for a node that was schedulable. my bad.

Do this to make the masters schedulable again if your Ingress pods are stuck pending.

oc patch scheduler cluster --patch '{"spec":{"mastersSchedulable":true}}' --type=merge

Of course, your load balancer will have to be sending http/s traffic to the master nodes, not the worker nodes.

[cgruver]

There is probably a "right" way to put the ingress pods on the master nodes using taints and tolerations, yet not label them as worker nodes. I'll see if I can figure it out and post here.

@vrutkovs Vadim, do you know how to do that? Should one do that? (philosophical question.)

[vrutkovs]

The right way to place ingress pods is via configuration: https://docs.okd.io/latest/networking/ingress-operator.html#nw-installation-ingress-config-asset_configuring-ingress.

Router pods have been created, but LB failed to be created:
Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB in events of openshift-ingress namespace.

I'm not entirely sure why is that happening - missing permissions / some AWS error? Could you have a look on AWS console?

[andygeorge]

@vrutkovs thank you so much, that got me pointed in the right direction!! I didn't have my tagging set correctly on our VPC's subnets. Works now!

For anyone else out there who finds this, I needed to make sure my VPC's subnets had the following Tags:

On both public and private subnets:
"kubernetes.io/cluster/$CLUSTERNAME" = "owned"

On public subnets:
"kubernetes.io/role/elb = "1"

On private subnets:
"kubernetes.io/role/internal-elb" = "1"

[fenggolang]

@andfasano ,hello,I also encountered a similar problem, mine is bare-metal deploy ocp4.4, can you provide some ideas, thank you very much

[root@cloud-com-1 registry]# oc describe pod console-6b899d75d6-5k9qn -n openshift-console
Name: console-6b899d75d6-5k9qn
Namespace: openshift-console
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: master1/192.168.122.202
Start Time: Tue, 19 May 2020 12:02:09 +0000
Labels: app=console
component=ui
pod-template-hash=6b899d75d6
Annotations: console.openshift.io/console-config-version: 132198
console.openshift.io/default-ingress-cert-config-version: 131913
console.openshift.io/image:
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52465443a0c0f8e1d9dd8f2e3c7ec0e5d32dc438f181adf467ab883a6b283c26
console.openshift.io/oauth-secret-version: 132207
console.openshift.io/proxy-config-version: 733
console.openshift.io/service-ca-config-version: 132714
console.openshift.io/trusted-ca-config-version: 132202
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.130.0.125"
],
"dns": {},
"default-route": [
"10.130.0.1"
]
}]
openshift.io/scc: restricted
operator.openshift.io/pull-spec:
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52465443a0c0f8e1d9dd8f2e3c7ec0e5d32dc438f181adf467ab883a6b283c26
Status: Running
IP: 10.130.0.125
IPs:
IP: 10.130.0.125
Controlled By: ReplicaSet/console-6b899d75d6
Containers:
console:
Container ID: cri-o://bf6538ff52cd90498263209a63663b13888679da6f34946a141d3a100a307f9e
Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52465443a0c0f8e1d9dd8f2e3c7ec0e5d32dc438f181adf467ab883a6b283c26
Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52465443a0c0f8e1d9dd8f2e3c7ec0e5d32dc438f181adf467ab883a6b283c26
Port: 443/TCP
Host Port: 0/TCP
Command:
/opt/bridge/bin/bridge
--public-dir=/opt/bridge/static
--config=/var/console-config/console-config.yaml
--service-ca-file=/var/service-ca/service-ca.crt
State: Running
Started: Tue, 19 May 2020 12:16:16 +0000
Last State: Terminated
Reason: Error
Message: T12:14:48Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.okd4-cluster.okd4.com/oauth/token failed: Head https://oauth-openshift.apps.okd4-cluster.okd4.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-05-19T12:15:07Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.okd4-cluster.okd4.com/oauth/token failed: Head https://oauth-openshift.apps.okd4-cluster.okd4.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-05-19T12:15:22Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.okd4-cluster.okd4.com/oauth/token failed: Head https://oauth-openshift.apps.okd4-cluster.okd4.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-05-19T12:15:37Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.okd4-cluster.okd4.com/oauth/token failed: Head https://oauth-openshift.apps.okd4-cluster.okd4.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-05-19T12:15:52Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.okd4-cluster.okd4.com/oauth/token failed: Head https://oauth-openshift.apps.okd4-cluster.okd4.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-05-19T12:16:07Z auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.okd4-cluster.okd4.com/oauth/token failed: Head https://oauth-openshift.apps.okd4-cluster.okd4.com: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

  Exit Code:    2
  Started:      Tue, 19 May 2020 12:12:56 +0000
  Finished:     Tue, 19 May 2020 12:16:15 +0000
Ready:          False
Restart Count:  4
Requests:
  cpu:        10m
  memory:     100Mi
Liveness:     http-get https://:8443/health delay=150s timeout=1s period=10s #success=1 #failure=3
Readiness:    http-get https://:8443/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:  <none>
Mounts:
  /etc/pki/ca-trust/extracted/pem from trusted-ca-bundle (ro)
  /var/console-config from console-config (ro)
  /var/default-ingress-cert from default-ingress-cert (ro)
  /var/oauth-config from console-oauth-config (ro)
  /var/run/secrets/kubernetes.io/serviceaccount from console-token-2dwlb (ro)
  /var/service-ca from service-ca (ro)
  /var/serving-cert from console-serving-cert (ro)

Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
console-serving-cert:
Type: Secret (a volume populated by a Secret)
SecretName: console-serving-cert
Optional: false
console-oauth-config:
Type: Secret (a volume populated by a Secret)
SecretName: console-oauth-config
Optional: false
console-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: console-config
Optional: false
service-ca:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: service-ca
Optional: false
default-ingress-cert:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: default-ingress-cert
Optional: false
trusted-ca-bundle:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: trusted-ca-bundle
Optional: false
console-token-2dwlb:
Type: Secret (a volume populated by a Secret)
SecretName: console-token-2dwlb
Optional: false
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/master=
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-reachable:NoExecute for 120s
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 120s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled default-scheduler Successfully assigned openshift-console/console-6b899d75d6-5k9qn to master1
Normal Pulled 15m kubelet, master1 Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52465443a0c0f8e1d9dd8f2e3c7ec0e5d32dc438f181adf467ab883a6b283c26" already present on machine
Normal Created 15m kubelet, master1 Created container console
Normal Started 15m kubelet, master1 Started container console
Warning Unhealthy 12m (x3 over 13m) kubelet, master1 Liveness probe failed: Get https://10.130.0.125:8443/health: dial tcp 10.130.0.125:8443: connect: connection refused
Normal Killing 12m kubelet, master1 Container console failed liveness probe, will be restarted
Warning Unhealthy 47s (x89 over 15m) kubelet, master1 Readiness probe failed: Get https://10.130.0.125:8443/health: dial tcp 10.130.0.125:8443: connect: connection refused
[root@cloud-com-1 registry]#

[cgruver]

@fenggolang

This issue has been closed for a while. You should probably open a new issue. You'll get more visibility.

I noticed that it looks like things are trying to connect to port 8443 instead of 6443 or 443. Unless you explicitly changed the configuration, OKD 4.4 does not use port 8443 for any connections. external API traffic is on 6443, machine-config uses 22623, and all other traffic is either HTTP or HTTPs on 80 and 443 respectively.

[cgruver]

Nevermind, those are internal cluster connections. Those pods may very well be using 8443 internally...