Openshift-ansible: Openshift cannot pull the image from the internal registry

[rosaldogarcia]

Openshift cannot pull the image from the internal registry

[origin@ctrl ~]$ oc get pods
NAME READY STATUS RESTARTS AGE
deployment-example-1-d6xmh 1/1 Running 2 2d
hello-openshift-nfs-pod 1/1 Running 2 1d
nginx-1-6kgcp 1/1 Running 2 1d
nginx-1-hjpq8 0/1 ImagePullBackOff 0 5m

[origin@ctrl ~]$ oc get pod nginx-1-hjpq8
NAME READY STATUS RESTARTS AGE
nginx-1-hjpq8 0/1 ImagePullBackOff 0 7m
[origin@ctrl ~]$ oc describe pod nginx-1-hjpq8
Name: nginx-1-hjpq8
Namespace: test-project
Node: node1.xxx.xxx.xx/10.200.4.101
Start Time: Thu, 27 Sep 2018 21:12:04 +0800
Labels: app=nginx
deployment=nginx-1
deploymentconfig=nginx
Annotations: openshift.io/deployment-config.latest-version=1
openshift.io/deployment-config.name=nginx
openshift.io/deployment.name=nginx-1
openshift.io/generated-by=OpenShiftWebConsole
openshift.io/scc=anyuid
Status: Pending
IP: 10.129.0.175
Controlled By: ReplicationController/nginx-1
Containers:
nginx:
Container ID:
Image: docker-registry.default.svc:5000/test-project/nginx@sha256:0b5c73966ec996a05672c4aea0a0d1910c6d7495147805ef88205bff51e119f3
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-k29qm (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-k29qm:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-k29qm
Optional: false
QoS Class: BestEffort
Node-Selectors: node-role.kubernetes.io/compute=true
Tolerations:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m default-scheduler Successfully assigned nginx-1-hjpq8 to node1.xxx.xxx.xx
Normal Pulling 7m (x2 over 7m) kubelet, node1.xxx.xxx.xx pulling image "docker-registry.default.svc:5000/test-project/nginx@sha256:0b5c73966ec996a05672c4aea0a0d1910c6d7495147805ef88205bff51e119f3"
Warning Failed 7m (x2 over 7m) kubelet, node1.xxx.xxx.xx Failed to pull image "docker-registry.default.svc:5000/test-project/nginx@sha256:0b5c73966ec996a05672c4aea0a0d1910c6d7495147805ef88205bff51e119f3": [rpc error: code = Unknown desc = Get http://docker-registry.default.svc:5000/v2/: dial tcp 172.30.240.233:5000: getsockopt: no route to host, rpc error: code = Unknown desc = Get http://docker-registry.default.svc:5000/v2/: dial tcp 172.30.240.233:5000: getsockopt: no route to host]
Warning Failed 7m (x2 over 7m) kubelet, node1.xxx.xxx.xx Error: ErrImagePull
Normal SandboxChanged 7m (x7 over 7m) kubelet, node1.xxx.xxx.xx Pod sandbox changed, it will be killed and re-created.
Normal BackOff 7m (x6 over 7m) kubelet, node1.xxx.xxx.xx Back-off pulling image "docker-registry.default.svc:5000/test-project/nginx@sha256:0b5c73966ec996a05672c4aea0a0d1910c6d7495147805ef88205bff51e119f3"
Warning Failed 2m (x34 over 7m) kubelet, node1.xxx.xxx.xx Error: ImagePullBackOff

VERSION:

[origin@ctrl ~]$ oc version
oc v3.10.0+0c4577e-1
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://10.200.4.108:8443
openshift v3.10.0+fd501dd-48
kubernetes v1.10.0+b81c8f8

openshift v3.10.0+0c4577e-1

OTHER INFO:

[origin@ctrl ~]$ sudo cat /etc/sysconfig/docker
INSECURE_REGISTRY='--insecure-registry 172.30.0.0/16 --insecure-registry 172.30.48.87:5000'

[glennodickson]

Hi I'm receiving the exact same error. The versions I'm using is:

Server https://192.168.1.50:8443
openshift v3.10.0+fd501dd-48
kubernetes v1.10.0+b81c8f8

I had issues pulling Jenkins using the;
add to project -> browse catalog -> Jenkins (Ephemeral)

Error:
Failed to pull image "docker-registry.default.svc:5000/openshift/jenkins@sha256:26d9f54ff135d9a28c5e49a431328c9c49af5235c952ce2b9cb4afafdc336fa7": rpc error: code = Unknown desc = Get https://docker-registry.default.svc:5000/v2/: dial tcp: lookup docker-registry.default.svc: device or resource busy

Assistance resolving this would be much appreciated.

[little-fish]

The same happens with java as well.

Pulling image "docker-registry.default.svc:5000/openshift/java@sha256:78af15475eac13d2ff439b33a9c3bdd39147858a824c420e8042fd5f35adce15" ...
pulling image error : Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
error: build error: unable to get docker-registry.default.svc:5000/openshift/java@sha256:78af15475eac13d2ff439b33a9c3bdd39147858a824c420e8042fd5f35adce15

[svenvogel]

we have the same problem. Some new about this issue? We use 3.11

[glennodickson]

Using OCP 3.10
I solved this by adding the following as the first nameserver in my /etc/resolve.conf
nameserver 172.30.0.1

which is the Kubernetes service as it listens on port 53

[ameagol]

As ROOT let resolv.conf editable.
chattr -ai /etc/resolv.conf

Edit it
vi /etc/resolv.conf

Add those on search entry
search cluster.local default.svc.cluster.local svc.cluster.local cluster.local default.svc

Activate it
systemctl restart NetworkManager

[sgremyachikh]

As ROOT let resolv.conf editable.
chattr -ai /etc/resolv.conf

Edit it
vi /etc/resolv.conf

Add those on search entry
search cluster.local default.svc.cluster.local svc.cluster.local cluster.local default.svc

Activate it
systemctl restart NetworkManager

Unfortunaly it doesn't work, because after systemctl restart NetworkManager your resolv.conf would be restored

To change search domain you should use NetworkManager:
nmcli con mod eth0 ipv4.dns-search "your.domain cluster.local default.svc.cluster.local svc.cluster.local cluster.local default.svc"
where eth0 is your ethernet interface and your.domain is your FQDN domain

Next you should check if your ethernet is configured with DNS:
cat /etc/sysconfig/network-scripts/ifcfg-eth0 where eth0 is your ethernet interface
if you didn't find there DNS1 and DNS2 records, then you need to do it by yourself

Finaly you must restart NetworkManager:
Systemctl restart NetworkManager

Thats how you gets propertly rendered resolv.conf.

But our mistake Failed to pull image "docker-registry.default.svc:5000/openshift/jenkins still exist

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.