Openshift-ansible: openshift/origin-node:v3.9 not found

[dougbtv]

Description

Using an inventory based on the provided inventory/hosts.example fails during a deployment. With the openshift_node : Check status of node image pre-pull play failing, with a complaint that the image openshift/origin-node:v3.9 not found.

This is being deployed to CentOS atomic hosts.

Version

$ ansible --version
ansible 2.5.3
  config file = /root/openshift-ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Apr 11 2018, 07:36:10) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

$ git describe
openshift-ansible-3.10.0-0.56.0-6-g2722c90

Steps To Reproduce

1. Spin up CentOS atomic hosts
2. Use inventory provided below.
3. Run the playbook with:

$ ansible-playbook -i inventory/doug.inventory playbooks/prerequisites.yml playbooks/deploy_cluster.yml 

Expected Results

Playbook completes without failure.

Observed Results

Full log available in this paste.

Running with this command:

$ ansible-playbook -i inventory/doug.inventory playbooks/prerequisites.yml playbooks/deploy_cluster.yml 

Errors out with:

TASK [openshift_node : Check status of node image pre-pull] *********************************************************************************************************************************************************************************
Friday 01 June 2018  11:31:00 +0000 (0:00:01.582)       0:01:53.209 *********** 
fatal: [ose3-master.test.example.com]: FAILED! => {"ansible_job_id": "554524704637.16931", "attempts": 1, "changed": false, "finished": 1, "msg": "Error pulling docker.io/openshift/origin-node - code: None message: manifest for docker.io/openshift/origin-node:v3.9 not found"}
fatal: [ose3-node1.test.example.com]: FAILED! => {"ansible_job_id": "286554451804.16763", "attempts": 1, "changed": false, "finished": 1, "msg": "Error pulling docker.io/openshift/origin-node - code: None message: manifest for docker.io/openshift/origin-node:v3.9 not found"}
fatal: [ose3-infra.test.example.com]: FAILED! => {"ansible_job_id": "838496782697.16866", "attempts": 1, "changed": false, "finished": 1, "msg": "Error pulling docker.io/openshift/origin-node - code: None message: manifest for docker.io/openshift/origin-node:v3.9 not found"}

PLAY RECAP **********************************************************************************************************************************************************************************************************************************

Inventory file

Inventory file based on ./inventory/hosts.example/ inventory as provided in this repo.

ose3-master.test.example.com ansible_host=192.168.1.247
ose3-infra.test.example.com ansible_host=192.168.1.121
ose3-node1.test.example.com ansible_host=192.168.1.56
ose3-lb.test.example.com ansible_host=192.168.1.150

[masters]
ose3-master.test.example.com

[etcd]
ose3-master.test.example.com

[nodes]
ose3-master.test.example.com
ose3-infra.test.example.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
ose3-node1.test.example.com openshift_node_labels="{'region': 'primary', 'zone': 'default'}"

[nfs]
ose3-master.test.example.com

[lb]
ose3-lb.test.example.com

[OSEv3:children]
masters
nodes
etcd
lb
nfs

[OSEv3:vars]
openshift_deployment_type=origin
openshift_release="3.9"
openshift_master_default_subdomain=apps.test.example.com
openshift_master_cluster_hostname=ose3-lb.test.example.com
openshift_disable_check=disk_availability,memory_availability #,docker_image_availability
debug_level=2
ansible_ssh_user=centos
ansible_become=yes
debug_level=2
ansible_ssh_private_key_file=/path/to/id_vm_rsa

Notably: Inclusion or exclusion of docker_image_availability didn't appear to make a difference.

Additional Information

Provide any additional information which may help us diagnose the
issue.

• Your operating system and version, ie: RHEL 7.2, Fedora 23 ($ cat /etc/redhat-release)

[centos@ose3-master ~]$ cat /etc/redhat-release 
CentOS Linux release 7.5.1804 (Core) 
[centos@ose3-master ~]$ uname -a
Linux ose3-master.test.example.com 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[rabem00]

Also add the openshift_image_tag in the inventory that way it can find the right container image version:
openshift_release=3.9
openshift_image_tag=v3.9.0

[dougbtv]

@rabem00 -- still get a similar error, this time with a more specific tag:

fatal: [ose3-node1.test.example.com]: FAILED! => {"ansible_job_id": "131115064205.13028", "attempts": 1, "changed": false, "finished": 1, "msg": "Error pulling docker.io/openshift/origin-node - code: None message: manifest for docker.io/openshift/origin-node:v3.9.0 not found"}

Here's the modified inventory including the added openshift_image_tag=v3.9.0.

[midekra]

+1
I get the same error. Seems like the images do not exists on docker hub

[dougbtv]

For what it's worth, this is a recently introduced issue, I had a friend who's been having successful deploys from a forked openshift-ansible, and I took the basis commit of that fork, and I can have a success if I roll back to that commit, using the same inventory... It's fairly old (late April), but, this is the version I used and had a successful deploy:

$ git describe
openshift-ansible-3.9.24-1-22-g56fdecf
$ git rev-parse HEAD
56fdecfbf6fe411455b91789bec0bab0abd91845

[dougbtv]

Taking a further look at the docker images on the master of this successful deploy, it looks like a change in the naming convention for the node image name:

$ docker images | grep -iP "node|repo"
REPOSITORY                                   TAG                 IMAGE ID            CREATED             SIZE
docker.io/openshift/node                     v3.9.0              37e5a77820a6        44 hours ago        1.52 GB

Whereas with the current master@HEAD openshift-ansible, it's looking for another name:

openshift/origin-node:v3.9

[JamesDrummond]

Same issue. Please fix soon :)

[dusansusic]

Similar thing here but closed too fast without any reason. https://github.com/openshift/openshift-ansible/issues/8582 I am using clean RH 7.5 AMIs. I really cannot understand all these commits on release-3.9 without real testing changes.
Even examples aren't working.

[vrutkovs]

3.10 playbooks are using static pods to install the cluster, its not applicable to 3.9.
/close

[dusansusic]

@vrutkovs but what is the solution here? You're just closing issues without answer about solution.

[dougbtv]

+1, is there a recipe for how to specify that you'd like to install 3.9 with openshift-ansible in your inventory?

[midekra]

Master branch should reflect the correct version in the hosts.example inventory file. It is configured for v3.9 (instead of v.3.10). So it doesn't work out-of-the-box apparently.

I got it working by just simply switching to the release-3.9 branch and run the appropriate playbooks, e.g.: prerequisites.yml and deploy_cluster.yml

[JamesDrummond]

Why was this issue closed? @dougbtv are you able to open it again. @vrutkovs Can you open this item back up? I cannot use openshift ansible until there is a solution.

[dougbtv]

Unfortunately I don't have the permissions to do so @JamesDrummond -- did you try a deployment from the release-3.9 branch yet? ...I personally haven't yet either.

[dusansusic]

I tried deployment from release-3.9 branch - it doesn't work at all.

Hosts: 1,2
Play: Ensure the requested version packages are available.
Task: openshift_version : fail
Message: Package 'origin-.9.0*' not found

[abhaynayak24]

I'm facing the same issue, why is this thread closed without any solution?

I've was able to run openshift 3.9 in my lab previously, but had re install everything due to some issue. I didn't face this issue previously but now i'm having this trouble. Any suggestions on how to overcome this issue? Im not sure but the fix seems to be in the inventory file

following is my inventory file from the previous working setup.

[OSEv3:children]
nodes
nfs
masters
etcd

[OSEv3:vars]
openshift_master_identity_providers=[{'name': 'allow_all_auth','login': 'true', 'challenge': 'true','kind': 'AllowAllPasswordIdentityProvider'}]
#openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
ansible_ssh_user=root
enable_excluders=False
enable_docker_excluder=False
ansible_service_broker_install=False
#openshift_enable_origin_repo=true
#openshift_repos_enable_testing=true
deployment_type=origin
openshift_deployment_type=origin
#os_sdn_network_plugin_name='redhat/openshift-ovs-multitenant'
#openshift_web_console_nodeselector={'region':'infra'}
openshift_pkg_version=-3.9.0
openshift_version=v3.9.0
openshift_release=v3.9.0
openshift_image_tag=v3.9.0
openshift_service_catalog_image_version=v3.9.0
openshift_service_broker_image_version=v3.9.0
openshift_metrics_image_version="v3.9"
#osm_use_cockpit=true
#osm_cockpit_plugins=['cockpit-kubernetes']
openshift_disable_check=disk_availability,docker_storage,memory_availability
#openshift_logging_image_version="v3.9"
#openshift_logging_elasticsearch_proxy_image_version="v1.0.0"
openshift_web_console_install=true
openshift_metrics_install_metrics=true
#openshift_logging_install_logging=true
openshift_metrics_hawkular_hostname=hawkular-metrics.example.com
openshift_master_metrics_public_url=https://hawkular-metrics.example.com/hawkular/metrics
openshift_metrics_cassandra_storage_type=emptydir
#openshift_metrics_cassandra_storage_type=dynamic
#openshift_metrics_cassandra_pvc_size=10Gi
#openshift_metrics_start_cluster=true
#openshift_public_hostname=master
#openshift_master_default_subdomain=apps.master
#openshift_master_api_port=443
#openshift_master_console_port=443
[nodes]
192.168.1.10  openshift_public_ip=master openshift_ip=192.168.1.10 openshift_public_hostname=master openshift_hostname=192.168.1.10 connect_to=192.168.1.10 containerized=false openshift_node_labels="{'region': 'infra'}" openshift_schedulable=True
#192.168.1.11  openshift_public_ip=node1 openshift_ip=192.168.1.11 openshift_public_hostname=192.168.1.11 openshift_hostname=192.168.1.11 containerized=True connect_to=192.168.1.11 openshift_node_labels="{'region': 'infra'}" openshift_schedulable=True
#192.168.1.8  openshift_public_ip=node2 openshift_ip=192.168.1.8 openshift_public_hostname=192.168.1.8 openshift_hostname=192.168.1.8  connect_to=192.168.1.8 containerized=True openshift_node_labels="{'region': 'infra'}" openshift_schedulable=True

[nfs]
192.168.1.10  openshift_public_ip=master openshift_ip=192.168.1.10 openshift_public_hostname=192.168.1.10 openshift_hostname=192.168.1.10 connect_to=192.168.1.10

[masters]
192.168.1.10  openshift_public_ip=master openshift_ip=192.168.1.10 openshift_public_hostname=192.168.1.10 openshift_hostname=master connect_to=192.168.1.10

[etcd]
192.168.1.10  openshift_public_ip=master openshift_ip=192.168.1.10 openshift_public_hostname=192.168.1.10 openshift_hostname=192.168.1.10 connect_to=192.168.1.10

[abhaynayak24]

I also noticed this one update from noarch 0:3.9.27 to 0:3.9.30. Not sure if this should affect the process but still I would try to install 3.9.27 using rpms if its possible. just need to know if it actually makes a difference or not

Package atomic-openshift-utils.noarch 0:3.9.27-1.git.0.52e35b5.el7 will be updated
---> Package atomic-openshift-utils.noarch 0:3.9.30-1.git.7.46f8678.el7 will be an update
--> Processing Dependency: openshift-ansible-playbooks = 3.9.30-1.git.7.46f8678.el7 for package: atomic-openshift-utils-3.9.30-1.git.7.46f8678.el7.noarch
--> Running transaction check
---> Package openshift-ansible-playbooks.noarch 0:3.9.27-1.git.0.52e35b5.el7 will be updated
--> Processing Dependency: openshift-ansible-playbooks = 3.9.27-1.git.0.52e35b5.el7 for package: openshift-ansible-3.9.27-1.git.0.52e35b5.el7.noarch
---> Package openshift-ansible-playbooks.noarch 0:3.9.30-1.git.7.46f8678.el7 will be an update
--> Processing Dependency: openshift-ansible-roles = 3.9.30-1.git.7.46f8678.el7 for package: openshift-ansible-playbooks-3.9.30-1.git.7.46f8678.el7.noarch
--> Running transaction check
---> Package openshift-ansible.noarch 0:3.9.27-1.git.0.52e35b5.el7 will be updated
--> Processing Dependency: openshift-ansible = 3.9.27-1.git.0.52e35b5.el7 for package: openshift-ansible-docs-3.9.27-1.git.0.52e35b5.el7.noarch
---> Package openshift-ansible.noarch 0:3.9.30-1.git.7.46f8678.el7 will be an update
---> Package openshift-ansible-roles.noarch 0:3.9.27-1.git.0.52e35b5.el7 will be updated
---> Package openshift-ansible-roles.noarch 0:3.9.30-1.git.7.46f8678.el7 will be an update
--> Running transaction check
---> Package openshift-ansible-docs.noarch 0:3.9.27-1.git.0.52e35b5.el7 will be updated
---> Package openshift-ansible-docs.noarch 0:3.9.30-1.git.7.46f8678.el7 will be an update
--> Finished Dependency Resolution

fatal: [xxx:xx:xx:x]: FAILED! => {"changed": false, "failed": true, "msg": "Package 'origin-v3.9.0*' not found"}

[leifmadsen]

I get the same issue on the latest release-3.9, with a previously working release-3.9 branch. This shouldn't be closed.