Origin: docker-registry and origin-haproxy-router pending deployment

[mibaboo]

I followed the official openshift origin ansible install which completed successfully. However when I run oc status I see the following messages.

https://docker-registry-default.router.default.svc.cluster.local (passthrough) (svc/docker-registry)
dc/docker-registry deploys docker.io/openshift/origin-docker-registry:v3.6.0
deployment #1 pending 2 days ago

svc/kubernetes - 172.30.0.1 ports 443->8443, 53->8053, 53->8053

https://registry-console-default.router.default.svc.cluster.local (passthrough) (svc/registry-console)
dc/registry-console deploys docker.io/cockpit/kubernetes:latest
deployment #1 failed 2 days ago: config change

svc/router - 172.30.161.169 ports 80, 443, 1936
dc/router deploys docker.io/openshift/origin-haproxy-router:v3.6.0
deployment #12 pending 18 hours ago
deployment #11 failed 18 hours ago: newer deployment was found running
deployment #10 failed 18 hours ago: newer deployment was found running

Errors:

• route/docker-registry is routing traffic to svc/docker-registry, but either the administrator has not installed a router or the router is not selecting this route.
• route/registry-console is routing traffic to svc/registry-console, but either the administrator has not installed a router or the router is not selecting this route.

Version

openshift v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

oc version
oc v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://servername:8443
openshift v3.6.0+c4dd4cf
kubernetes v1.6.1+5115d708d7

Steps To Reproduce

1. [step 1]
   Installed openshift origin with ansible
2. [step 2]
   run oc status

Any idea what could be wrong?

[pweil-]

@mibaboo what do the deployment logs say? Are you getting the same error on both the registry and router deployment or are there different issues? If you can post the deployment logs that would be helpful in assigning this issue to the right folks to help you debug. Thanks

[mibaboo]

@pweil- yes I am getting the same error got registry and router.

oc logs dc/docker-registry-2-deploy

Error from server (NotFound): deploymentconfigs.apps.openshift.io "docker-registry-2-deploy" not found

oc logs dc/router-12-deploy

Error from server (NotFound): deploymentconfigs.apps.openshift.io "router-12-deploy" not found

Hope the above is what you looking for, I am new to OpenShift.

I also see the following when I check get events

oc get events

LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
1m 8h 1746 docker-registry-2-deploy Pod Warning FailedScheduling default-scheduler No nodes are available that match all of the following predicates:: CheckServiceAffinity (2), MatchNodeSelector (2).
1m 8h 1816 router-12-deploy Pod Warning FailedScheduling default-scheduler No nodes are available that match all of the following predicates:: CheckServiceAffinity (2), MatchNodeSelector (2).

[pweil-]

@mibaboo that is saying that the deployment doesn't exist, not that the deployment has an error. Please check that you have existing deployments and if you have failed deployments please post the logs.

Thanks

[mibaboo]

@pweil- When I check the history it shows

Error from server (NotFound): deploymentconfigs.apps.openshift.io "*" not found

So why would the deployment not exist? I was under the impression the ansible playbook deploys the registry as well?

could this issue not be related to output of oc get events ?

No nodes are available that match all of the following predicates:: CheckServiceAffinity (2), MatchNodeSelector (2).

[pweil-]

@mibaboo that seems to contradict the initial description where there were at least a couple of DCs out there.

oc get dc/router returns nothing?

[mibaboo]

oc get dc/router
NAME REVISION DESIRED CURRENT TRIGGERED BY
router 12 1 0 config

[pweil-]

ok, so the dc is there. Let's try to deploy it and get the logs. Try this:

oc deploy --latest dc/router
oc logs -f dc/router

When that is complete post the logs and also the output of oc get events

[mibaboo]

oc rollout latest dc/router

error: #12 is already in progress (Pending).

oc logs -f dc/router

Error from server (BadRequest): failed to run deployer pod router-12-deploy: timed out waiting for the condition

Forgot to post oc get events

LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
1m 9h 1939 docker-registry-2-deploy Pod Warning FailedScheduling default-scheduler No nodes are available that match all of the following predicates:: CheckServiceAffinity (2), MatchNodeSelector (2).
1m 9h 2005 router-12-deploy Pod Warning FailedScheduling default-scheduler No nodes are available that match all of the following predicates:: CheckServiceAffinity (2), MatchNodeSelector (2).
1m 9h 934 sti-ruby BuildConfig Warning BuildConfigInstantiateFailed buildconfig-controller error instantiating Build from BuildConfig default/sti-ruby (0): Error resolving ImageStreamTag ruby:latest in namespace openshift: unable to find latest tagged image

[pweil-]

@mibaboo thanks. Yes, it looks like this is a scheduling issue based on the events. Can you check that oc get nodes shows nodes in a schedulable state? Can you do an oc get on that deployment and show us what the full json looks like?

[mibaboo]

@pweil- Here is is the output for oc get nodes

NAME STATUS AGE VERSION
master Ready 4d v1.6.1+5115d708d7
node Ready 3d v1.6.1+5115d708d7

oc get dc/docker-registry

NAME REVISION DESIRED CURRENT TRIGGERED BY
docker-registry 2 0 0 config

for the full json, should I run oc describe dc/docker-registry ?

[pweil-]

oc get dc/docker-registry -o json

[mibaboo]

{
"apiVersion": "v1",
"kind": "DeploymentConfig",
"metadata": {
"creationTimestamp": "2017-09-07T14:17:10Z",
"generation": 3,
"labels": {
"docker-registry": "default"
},
"name": "docker-registry",
"namespace": "default",
"resourceVersion": "258330",
"selfLink": "/oapi/v1/namespaces/default/deploymentconfigs/docker-registry",
"uid": "3c8688cc-93d7-11e7-83e4-005056873b8c"
},
"spec": {
"replicas": 0,
"selector": {
"docker-registry": "default"
},
"strategy": {
"activeDeadlineSeconds": 21600,
"resources": {},
"rollingParams": {
"intervalSeconds": 1,
"maxSurge": "25%",
"maxUnavailable": "25%",
"timeoutSeconds": 600,
"updatePeriodSeconds": 1
},
"type": "Rolling"
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"docker-registry": "default"
}
},
"spec": {
"containers": [
{
"env": [
{
"name": "REGISTRY_HTTP_ADDR",
"value": ":5000"
},
{
"name": "REGISTRY_HTTP_NET",
"value": "tcp"
},
{
"name": "REGISTRY_HTTP_SECRET",
"value": "RctZ9aHBhqHfC3HD8yhacx0MkFZ0XnlD5T6/Y96NXoY="
},
{
"name": "REGISTRY_MIDDLEWARE_REPOSITORY_OPENSHIFT_ENFORCEQUOTA",
"value": "false"
},
{
"name": "OPENSHIFT_DEFAULT_REGISTRY",
"value": "docker-registry.default.svc:5000"
},
{
"name": "REGISTRY_HTTP_TLS_KEY",
"value": "/etc/secrets/registry.key"
},
{
"name": "REGISTRY_HTTP_TLS_CERTIFICATE",
"value": "/etc/secrets/registry.crt"
}
],
"image": "openshift/origin-docker-registry:v3.6.0",
"imagePullPolicy": "IfNotPresent",
"livenessProbe": {
"failureThreshold": 3,
"httpGet": {
"path": "/healthz",
"port": 5000,
"scheme": "HTTPS"
},
"initialDelaySeconds": 10,
"periodSeconds": 10,
"successThreshold": 1,
"timeoutSeconds": 5
},
"name": "registry",
"ports": [
{
"containerPort": 5000,
"protocol": "TCP"
}
],
"readinessProbe": {
"failureThreshold": 3,
"httpGet": {
"path": "/healthz",
"port": 5000,
"scheme": "HTTPS"
},
"periodSeconds": 10,
"successThreshold": 1,
"timeoutSeconds": 5
},
"resources": {
"requests": {
"cpu": "100m",
"memory": "256Mi"
}
},
"securityContext": {
"privileged": false
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/registry",
"name": "registry-storage"
},
{
"mountPath": "/etc/secrets",
"name": "registry-certificates"
}
]
}
],
"dnsPolicy": "ClusterFirst",
"nodeSelector": {
"region": "infra"
},
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"serviceAccount": "registry",
"serviceAccountName": "registry",
"terminationGracePeriodSeconds": 30,
"volumes": [
{
"emptyDir": {},
"name": "registry-storage"
},
{
"name": "registry-certificates",
"secret": {
"defaultMode": 420,
"secretName": "registry-certificates"
}
}
]
}
},
"test": false,
"triggers": [
{
"type": "ConfigChange"
}
]
},
"status": {
"availableReplicas": 0,
"conditions": [
{
"lastTransitionTime": "2017-09-07T14:17:10Z",
"lastUpdateTime": "2017-09-07T14:17:10Z",
"message": "Deployment config does not have minimum availability.",
"status": "False",
"type": "Available"
},
{
"lastTransitionTime": "2017-09-11T10:15:52Z",
"lastUpdateTime": "2017-09-11T10:15:52Z",
"message": "replication controller \"docker-registry-2\" is waiting for pod \"docker-registry-2-deploy\" to run",
"status": "Unknown",
"type": "Progressing"
}
],
"details": {
"causes": [
{
"type": "Manual"
}
],
"message": "manual change"
},
"latestVersion": 2,
"observedGeneration": 3,
"replicas": 0,
"unavailableReplicas": 0,
"updatedReplicas": 0
}
}

[pweil-]

Thanks

"nodeSelector": {
"region": "infra"
},

Please double check the labels on your node and ensure that is has a region label set to infra. This is probably what is causing the scheduling issue and could cause the MatchNodeSelector predicate to fail.

[mibaboo]

@pweil- Thanks that worked :+1:
I really appreciate the time you spent helping me.

[borissenkobogdan]

Thanks too )

oc label node/node1.openshift.origin.ua region=infra zone=east --overwrite

oc label node/node2.openshift.origin.ua region=infra zone=west --overwrite

[singhcl]

@pweil- can you please confirm if dc/docker-registry need not to be run as a pod since the config has desired pods as 0.

sudo oc get dc/docker-registry

NAME REVISION DESIRED CURRENT TRIGGERED BY
docker-registry 3 0 0 config

[bparees]

@singhcl yes the registry runs as a pod. sounds like someone/something scaled your deployment to zero replicas.