Openshift-ansible: Docker Registry and Router failed scheduling

[anhnguyenbk]

I am using openshift-ansible to set up a cluster has 1 master and 1 node. The ansible-playbook command result was ok but I found docker registry pod and router pod failed scheduling. Here is my cluster info.

Inventory file:

[OSEv3:children]
masters
nodes

[OSEv3:vars]
ansible_ssh_user=root
deployment_type=origin

[masters]
ose3-master.example.com

[nodes]
ose3-master.example.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
ose3-node1.example.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"

$ oc get nodes

NAME                      STATUS                     AGE
ose3-master.example.com   Ready,SchedulingDisabled   1h
ose3-node1.example.com    Ready                      1h

$ oc get pods

NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-deploy   0/1       Pending   0          42m
router-1-deploy            0/1       Pending   0          43m

$ oc describe pod docker-registry-1-deploy

Name:       docker-registry-1-deploy
Namespace:  default
Node:       /
Labels:     openshift.io/deployer-pod-for.name=docker-registry-1
Status:     Pending
IP:     
Controllers:    <none>
Containers:
  deployment:
    Image:  openshift/origin-deployer:v1.2.1
    Port:   
    QoS Tier:
      cpu:  BestEffort
      memory:   BestEffort
    Environment Variables:
      KUBERNETES_MASTER:    https://ose3-master.example.com:8443
      OPENSHIFT_MASTER:     https://ose3-master.example.com:8443
      BEARER_TOKEN_FILE:    /var/run/secrets/kubernetes.io/serviceaccount/token
      OPENSHIFT_CA_DATA:    -----BEGIN CERTIFICATE-----
MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu
c2hpZnQtc2lnbmVyQDE0Njk0MjI5MTYwHhcNMTYwNzI1MDUwMTU1WhcNMjEwNzI0
MDUwMTU2WjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE0Njk0MjI5MTYw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC2A6mYuwBmObp2PRv8xIKE
o/+RSr8a/My05nQvMg+PGeQUVCSUmEbEehgpwOYvpJK4nnFBjyOirVc+b4r1m/oH
FvRrq6jjDy0ZIW0SXAR46jgS4S+AWSuCP5AROqQwJhWEFAOyNSMok+eM+8IQsWph
usvVGU7rayAsDx7V88Mfj8mPceS+6zeda7G7DEU4Jr1vqSHq27bNmwZhUso2X2Ws
8G3LvLf3iRPHQ8tpiuVSuLBGYnkuQfs4tJu4FrQdWh951a3dXrRGb94wO4PZgf58
CaqWGpX5VtZYQ1/+/9g2hfBWAXVfn/m4K63b362Bpd8T+CbgNijlwqiu0/aHd4Lj
AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
SIb3DQEBCwUAA4IBAQAPpfaebgQjQewsq+tq+4ndIko5yabouPkLOnAIz2d3VDql
eYKPrqadscb0n2GvswENImRKFzpR8owkhjUvArDHmsx5MgmQx5a1pVdCTBXaYGcI
XJt4EVxSZVI2E7fHcZ76Mna0YRl9aBYnGVErJQVsOi4N5TZ6zQXPA6kHROgKweLU
3RXXiMfBf1XMVeh7iqI0OvLot06Wrh+hP7w8aSjIhjjF6h5cuECnHIEmzqWlXFPp
COaHEO0lVG//4wGGdBMpGD7+/cVqDleWwv9M30IrDBTf+HZ7OqvFlhFgxvw3XW70
W7+yceytbe9cUWyB0R7/MqgKF4FwcKr8Ggj7SFvT
-----END CERTIFICATE-----

      OPENSHIFT_DEPLOYMENT_NAME:    docker-registry-1
      OPENSHIFT_DEPLOYMENT_NAMESPACE:   default
Volumes:
  deployer-token-np92j:
    Type:   Secret (a volume populated by a Secret)
    SecretName: deployer-token-np92j
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  55m       7s      92  {default-scheduler }            Warning     FailedScheduling    pod (docker-registry-1-deploy) failed to fit in any node
fit failure on node (ose3-node1.example.com): MatchNodeSelector

  55m   3s  101 {default-scheduler }        Warning FailedScheduling    pod (docker-registry-1-deploy) failed to fit in any node
fit failure on node (ose3-node1.example.com): Region

[rushabh268]

Seeing the same issue. How was this fixed?

[sdodson]

@rushabh268 It's fixed by adding openshift_schedulable=true to your node that's also a master or post install oadm manage-node ose3-master.example.com --schedulable=true

The documentation has recently been updated to reflect that, by default, masters are marked unschedulable. So if your only nodes marked as region=infra are also masters then you won't get successful router and registry deployment.

[rushabh268]

@sdodson I have a OSE HA setup and I don't even see the master nodes when I do oc get nodes.

[sdodson]

@rushabh268 Can you open a new issue and provide your inventory file so we can debug there?

[jsandova]

The --schedulable=true setting on my master node fix the issue for me.

[pakala27]

This setting i have applied but issue was not resolved. Do we need to restart anything? Changed server name.

oc adm manage-node master.example.com --schedulable=true

NAME STATUS ROLES AGE VERSION
master.example.com Ready master 4d v1.10.0+b81c8f8

[root@master ~]# oc get po
NAME READY STATUS RESTARTS AGE
docker-registry-3-deploy 0/1 Pending 0 1m
registry-console-1-s26bq 1/1 Running 0 4d
router-5-deploy 0/1 Pending 0 1m
router1-1-rcbpp 1/1 Running 0 2m
Please advise

[pakala27]

I just wanted to update you that my issue was resolved after labeling master node as infra node too. This i missed in ansible inventory file. So you can update ansible inventory file too.

oc label node master.example.com node-role.kubernetes.io/infra=true

update inventory as below if you have single master and wanted to it for infra node as well.

[nodes]
master.example.com openshift_node_group_name='node-config-master-infra'
node1.example.com openshift_node_group_name='node-config-compute'
node2.example.com openshift_node_group_name='node-config-compute'

I hope it resolve the issue.