Openshift-ansible: cockpit/kubernetes-image no longer available from dockerhub

[pgress]

Description

On a fresh cluster-installation the playbooks fail at the docker_image_availability-step. Reason for this issue is, that the cockpit/kubernetes:latest image is deleted. It looks like the work on this image ist discontinued.
See also https://github.com/cockpit-project/cockpit/commit/37aa31e58f3375feafdfa87d66ac8a494148b418

Version

* Your ansible version per `ansible --version`
ansible 2.9.4
  config file = None
  configured module search path = ['/home/ansible/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.7.5 (default, Oct 17 2019, 12:25:15) [GCC 8.3.0]

* The output of `git describe`
openshift-ansible-3.11.174-1

Steps To Reproduce

1. Just follow the instruction to install any sort of cluster

Expected Results

A working Cluster in initial state

Observed Results

     One or more checks failed
      check "docker_image_availability":
                One or more required container images are not available:
                    docker.io/cockpit/kubernetes:latest
                Checked with: skopeo inspect [--tls-verify=false] [--creds=<user>:<pass>] docker://<registry>/<image>

[js-max]

got errors also deploying 3.11 here, same msg

[js-max]

Checking the code if I'm wrong please correct me

defaults/main.yml file will have

openshift_health_check_required_images: "{{ [] + l_required_node_images + l_cockpit_images + l_master_required_images + l_etcd_required_images + l_atomic_node_required_images }}"

l_cockpit_images var is evaluated two lines before with a ternary

l_cockpit_images: "{{ (openshift_hosted_manage_registry_console | bool) | ternary([openshift_cockpit_deployer_image], []) }}"

so as a quick workaround openshift_hosted_manage_registry_console=false at inventory file is enough for a basic installation

different configs should need to be checked accordingly, tested and worked as expected on my side but with that approach openshift install will continue to check everything apart cockpit

really not aware that this should be handled like a var setup at inventory or the code should be changed

[ZrytyADHD]

I have the same error today installing cluster several times. I write on cockpit git and also try to pull image to my private repo from other clusterr.

[timbrd]

I've uploaded a recent cockpit/kubernetes image to my personal namespace on dockerhub.

With the following line in your inventory, you can deploy a cluster with a working registry-console:

openshift_cockpit_deployer_image='docker.io/timbordemann/cockpit-kubernetes:latest'

Btw: This seems to be the most recent source, if you want to build the image yourself: https://github.com/cockpit-project/cockpit/tree/195.6/containers/kubernetes

[thdevelthreesixtiy]

Thx very much to timbrd for the copy.
You made my day!

[aroute]

Is there a way to fix this on an already-running OKD 3.11 cluster? My registry-console pod has ImagePullBackOff error due to this issue.

[pgress]

@aroute Yes, you can use the image from @timbrd and put it in your deployment for the registry-console

[aroute]

okay, that worked. I built the image, which also worked. But, this must not be a permanent solution? Is anyone looking to resolve this so we don't have to apply our own work around?

[timbrd]

okay, that worked. I built the image, which also worked. But, this must not be a permanent solution? Is anyone looking to resolve this so we don't have to apply our own work around?

@ZrytyADHD has opened an issue on the cockpit project. Hopefully they reupload the image to the official namespace again soon.

[magick93]

I have tried the following:

• docker pull cockpit/kubernetes:latest
• docker pull registry.access.redhat.com/openshift3/registry-console:latest
• docker pull timbordemann/kubernetes:latest
• docker pull openshift3/registry-console
• oc import-image openshift3/registry-console --from=registry.access.redhat.com/openshift3/registry-console --confirm

None have worked.

I'm using Docker version 1.13.1, build cccb291/1.13.1 for Okd 3.11.

[magick93]

This playbook will pull the image (tested on Centos 7)

- name: Fix missing registry console
  hosts: masters
  gather_facts: True

  tasks:

    - name: ensure /etc/rhsm/ca/redhat-uep.pem exists
      copy:
        content: ""
        dest: /etc/rhsm/ca/redhat-uep.pem
        owner: root
      become: yes


    - name: pull registry.access.redhat.com/openshift3/registry-console:v3.11.170-5
      docker_image:
        name: registry.access.redhat.com/openshift3/registry-console:v3.11.170-5
        source: pull

[zilmarr]

Thanks @timbrd ! I owe you a beer.

[harrisiftikhar]

I've uploaded a recent cockpit/kubernetes image to my personal namespace on dockerhub.

With the following line in your inventory, you can deploy a cluster with a working registry-console:

openshift_cockpit_deployer_image='docker.io/timbordemann/cockpit-kubernetes:latest'

Btw: This seems to be the most recent source, if you want to build the image yourself: https://github.com/cockpit-project/cockpit/tree/195.6/containers/kubernetes

Thanks for the response. I have added openshift_cockpit_deployer_image=docker.io/timbordemann/cockpit-kubernetes in /etc/ansible/hosts file
and I am able to pull the image
[root@master openshift-ansible]# docker pull docker.io/timbordemann/cockpit-kubernetes:latest
Trying to pull repository docker.io/timbordemann/cockpit-kubernetes ...
latest: Pulling from docker.io/timbordemann/cockpit-kubernetes

I am getting the following error
TASK [Gathering Facts] **********************************
Tuesday 24 March 2020 14:11:44 -0300 (0:00:00.120) 0:00:02.321 **
fatal: [openshift_cockpit_deployer_image=docker.io/timbordemann/cockpit-kubernetes]: UNREACHABLE! => {"changed": false, "msg": "SSH Error: data could not be sent to remote host "openshift_cockpit_deployer_image=docker.io/timbordemann/cockpit-kubernetes". Make sure this host can be reached over ssh", "unreachable": true}
ok: [master.iftikhar.us]

NO MORE HOSTS LEFT ***********************************

PLAY RECAP ***************************************
localhost : ok=11 changed=0 unreachable=0 failed=0
master.iftikhar.us : ok=2 changed=0 unreachable=0 failed=0
openshift_cockpit_deployer_image=docker.io/timbordemann/cockpit-kubernetes : ok=0 changed=0 unreachable=1 failed=0

[zilmarr]

@harrisiftikhar I think you're adding that line in the wrong section.
It seems you added to the [hosts] section, you have to add it to the vars section (the end of your inventory file)

[harrisiftikhar]

Thanks! it worked after moving it to the variable section.

[harrisiftikhar]

I am still seeing the following errors.
Failed to pull image "cockpit/kubernetes:latest": rpc error: code = Unknown desc = manifest for docker.io/cockpit/kubernetes:latest not found

I have restarted the ansible deployment after adding openshift_cockpit_deployer_image=docker.io/timbordemann/cockpit-kubernetes:latest
ansible-playbook playbooks/deploy_cluster.yml

[harrisiftikhar]

3m 13m 1
registry-console-2-deploy.15ff5aae0b91ab4d Pod
spec.containers{deployment} Normal Pulled
kubelet, compute Successfully pulled image
"openshift/origin-deployer:v3.9.0"

13m 13m 1
registry-console-2-deploy.15ff5aae32132009 Pod
spec.containers{deployment} Normal Created
kubelet, compute Created container

13m 13m 1
registry-console-2-deploy.15ff5aae4dbbb3bf Pod
spec.containers{deployment} Normal Started
kubelet, compute Started container

13m 13m 1
registry-console-2-lxwkl.15ff5aaed032b0c6 Pod
Normal Scheduled
default-scheduler Successfully assigned
registry-console-2-lxwkl to compute

13m 13m 1
registry-console-2-lxwkl.15ff5aaedeb24a43 Pod
Normal SuccessfulMountVolume
kubelet, compute MountVolume.SetUp succeeded for volume
"default-token-lvxgg"

13m 13m 2
registry-console-2-lxwkl.15ff5aaf95818b38 Pod
spec.containers{registry-console} Normal Pulling
kubelet, compute pulling image
"cockpit/kubernetes:latest"

13m 13m 2
registry-console-2-lxwkl.15ff5aafaf521441 Pod
spec.containers{registry-console} Warning Failed
kubelet, compute Failed to pull image
"cockpit/kubernetes:latest": rpc error: code = Unknown desc = manifest for
docker.io/cockpit/kubernetes:latest not found

13m 13m 2
registry-console-2-lxwkl.15ff5aafaf52cd98 Pod
spec.containers{registry-console} Warning Failed
kubelet, compute Error: ErrImagePull

13m 13m 7
registry-console-2-lxwkl.15ff5aaff246d9d6 Pod
Normal SandboxChanged
kubelet, compute Pod sandbox changed, it will be killed
and re-created.

13m 13m 5
registry-console-2-lxwkl.15ff5ab08ae11606 Pod
spec.containers{registry-console} Normal BackOff
kubelet, compute Back-off pulling image
"cockpit/kubernetes:latest"

8m 13m 76
registry-console-2-lxwkl.15ff5ab08ae18e22 Pod
spec.containers{registry-console} Warning Failed
kubelet, compute Error: ImagePullBackOff

13m 13m 1 registry-console-2.15ff5aaecfc0d4fb
ReplicationController Normal
SuccessfulCreate replication-controller Created
pod: registry-console-2-lxwkl

3m 3m 1 registry-console-2.15ff5b3aa86b9ce7
ReplicationController Normal
SuccessfulDelete replication-controller Deleted
pod: registry-console-2-lxwkl

14m 14m 1 registry-console.15ff5aa5cda74d81
DeploymentConfig Normal
DeploymentCreated deploymentconfig-controller Created
new replication controller "registry-console-2" for version 2

3m 3m 1 registry-console.15ff5b3aa27929f6
DeploymentConfig Normal
ReplicationControllerScaled deploymentconfig-controller Scaled
replication controller "registry-console-2" from 1 to 0

9m 2h 7 template-service-broker.15ff54db60d3a5f1
ClusterServiceBroker Normal
FetchedCatalog service-catalog-controller-manager
Successfully fetched catalog entries from broker.

On Tue, Mar 24, 2020 at 1:17 PM Zilmar de Souza Junior <
[email protected]> wrote:

@harrisiftikhar https://github.com/harrisiftikhar I think you're adding
that line in the wrong section.
It seems you added to the [hosts] section, you have to add it to the vars
section (the end of your inventory file)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/openshift-ansible/issues/12115#issuecomment-603382125,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AO5WX67KWFBE3L5NMORJZ7TRJDTKBANCNFSM4LBHUD2Q
.

[harrisiftikhar]

I was able to manually delete the deployment config and recreate it with the timbrd's image (Thanks a lot @timbrd). It'd be nice to figure out if I can understand why the initial deployment was pointing to the wrong image.

"Failed to pull image "cockpit/kubernetes:latest": rpc error: code = Unknown desc = manifest for docker.io/cockpit/kubernetes:latest not found"

Following is my /etc/ansible/hosts file:

[OSEv3:children]
masters
nodes
etcd

[OSEv3:vars]
openshift_deployment_type=origin
os_firewall_use_firewalld=True
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_pkg_version='-3.9.0'
openshift_master_default_subdomain=origin.iftikhar.us
openshift_disable_check=disk_avalability,memory_availability

openshift_disable_check=disk_avalability,memory_availability,docker_storage

ansible_ssh_user=root
ansible_user=root
openshift_cockpit_deployer_image='docker.io/timbordemann/cockpit-kubernetes:latest'

template_service_broker_install=false

.....

[tinu73]

Solution is described here

[khodayard]

No need to edit ansible host file and redeploy the whole cluster, I've just edited this deployment yaml file and replaced not-working (not-found) image with this:

docker.io/timbordemann/cockpit-kubernetes:latest

It was mentioned in two places.
Register console came up as soon as I've saved the yaml file.
Thanks to @timbrd

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[pgress]

/remove-lifecycle stale

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[mtnbikenc]

Fixed in #12278