Openshift-ansible: Playbooks fail with containerized=true for version > 3.7 when looking for RPMs

[dougbtv]

Description

When trying to run the playbooks to deploy a cluster on CentOS machines using a BYO inventory (running the deploy_cluster.yml playbook), if one sets containerized=true and openshift_release=3.9 (for example), the playbooks fail when it goes to run the openshift_version roles. Resulting in a failure stating: "Package origin not found".

My observation is that RPMs are not available for versions greater than 3.7.0, and therefore when using containerized=true, the plays looking for the RPMs should otherwise be skipped.

I believe this was introduced in this line of the ./roles/openshift_version/tasks/masters_and_nodes.yml tasks -- at this commit https://github.com/openshift/openshift-ansible/commit/54a83bf2b898338c70aeb094c9a0b86b8df8e2d2

I will submit a PR with a patch shortly referencing this issue.

This may be related to another reported issue, "lack of 3.7.1 rpm's blocks containarized upgrade" (sic) https://github.com/openshift/openshift-ansible/issues/6788

Version

$ ansible --version
ansible 2.4.2.0
  config file = /root/openshift-ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

$ git describe
openshift-ansible-3.9.0-0.24.0-19-g6c921b0

Steps To Reproduce

1. Clone openshift-ansible @ master HEAD
2. Create an inventory [full inventory below under "additional information"], including containerized=true & openshift_release=3.9
3. Run the rereqs playbook, ansible-playbook -i doug.inventory ./playbooks/prerequisites.yml
4. Run the deploy cluster playbook, ansible-playbook -i doug.inventory ./playbooks/deploy_cluster.yml

Expected Results

Completion without failure, e.g. exits 0, resulting in a OpenShift Origin 3.9.x cluster.

Observed Results

./playbooks/deploy_cluster.yml fails with: Package origin not found.

Full paste of output available in this pasteall snippet.

$ ansible-playbook -i doug.inventory ./playbooks/prerequisites.yml &> /dev/null
$ echo $?
0
$ ansible-playbook -i doug.inventory ./playbooks/deploy_cluster.yml
[ ... snipped ...]
TASK [openshift_version : fail] *************************************************************************************************************************************************************************************************************
Thursday 25 January 2018  20:47:46 +0000 (0:00:02.710)       0:00:25.732 ****** 
fatal: [openshift-minion-1]: FAILED! => {"changed": false, "msg": "Package origin not found"}
fatal: [openshift-minion-2]: FAILED! => {"changed": false, "msg": "Package origin not found"}
[ ... snipped ...]
Failure summary:


  1. Hosts:    openshift-minion-1, openshift-minion-2
     Play:     Ensure the requested version packages are available.
     Task:     openshift_version : fail
     Message:  Package origin not found
$ echo $?
2

Work around

I was able to successfully work-around this issue by making this change:

$ git diff
diff --git a/roles/openshift_version/tasks/masters_and_nodes.yml b/roles/openshift_version/tasks/masters_and_nodes.yml
index eddd5ff..757fa04 100644
--- a/roles/openshift_version/tasks/masters_and_nodes.yml
+++ b/roles/openshift_version/tasks/masters_and_nodes.yml
@@ -13,7 +13,9 @@
     - openshift_version_reinit | default(false)

   # block when
-  when: not openshift_is_atomic | bool
+  when: 
+  - not openshift_is_atomic | bool
+  - not openshift_is_containerized | bool

 # We can't map an openshift_release to full rpm version like we can with containers; make sure
 # the rpm version we looked up matches the release requested and error out if not.

This also required that I additionally set these variables in my inventory for [OSEv3:vars]:

openshift_disable_check=docker_image_availability
enable_excluders=false

With that in place, I could run these commands:

$ ansible-playbook -i doug.inventory ./playbooks/prerequisites.yml
$ ansible-playbook -i doug.inventory ./playbooks/deploy_cluster.yml

This would result in a working cluster with a master and two nodes, and OpenShift version as such:

[root@openshift-master centos]# /usr/local/bin/oc version
oc v3.9.0-alpha.3+4f709b4-198
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://openshift-master.example.local:8443
openshift v3.9.0-alpha.3+4f709b4-198
kubernetes v1.9.1+a0ce1bc657

[root@openshift-master centos]# /usr/local/bin/oc get nodes
NAME                               STATUS    ROLES     AGE       VERSION
openshift-master.example.local     Ready     master    30m       v1.9.1+a0ce1bc657
openshift-minion-1.example.local   Ready     <none>    23m       v1.9.1+a0ce1bc657
openshift-minion-2.example.local   Ready     <none>    23m       v1.9.1+a0ce1bc657

Additional Information

Full inventory:

openshift-master ansible_host=192.168.1.154
openshift-minion-1 ansible_host=192.168.1.249
openshift-minion-2 ansible_host=192.168.1.233

[OSEv3:children]
masters
nodes
etcd

[OSEv3:vars]
ansible_ssh_user=centos
ansible_become=yes
debug_level=2
ansible_ssh_private_key_file=/root/.ssh/id_vm_rsa
openshift_master_unsupported_embedded_etcd=true 
openshift_disable_check=disk_availability,memory_availability,docker_image_availability
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]
openshift_deployment_type=origin 
containerized=true
openshift_release=3.9
openshift_image_tag=latest
enable_excluders=false

[masters]
openshift-master

[etcd]
openshift-master

[nodes]
openshift-master openshift_node_labels="{'region': 'infra', 'zone': 'default'}" openshift_schedulable=true
openshift-minion-[1:2] openshift_node_labels="{'region': 'primary', 'zone': 'default'}"

Using CentOS 7.

[vrutkovs]

See https://github.com/openshift/openshift-ansible/pull/6755#issuecomment-358653089

[dougbtv]

@vrutkovs thanks for the pointer, so in theory it should work given that
PR? Any idea what I'm missing in my playbook or my workflow that might be
causing it to fail?

On Sat, Jan 27, 2018, 7:06 AM Vadim Rutkovsky [e-mail]
wrote:

See #6755 (comment)
https://github.com/openshift/openshift-ansible/pull/6755#issuecomment-358653089

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/openshift/openshift-ansible/issues/6899#issuecomment-360980520,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFN_vaaLHQlgCVYaRD_UgFQ9bE5Of4Kyks5tOxFEgaJpZM4RutfI
.

[vrutkovs]

The containerized installation on non-Atomic OSes is not currently supported, as containerized install expects an Atomic system, which already has some packages pre-installed - yum-utils, PyYAML etc.

You should be able to install containerized version of 3.9 on CentOS Atomic though:

openshift_release=v3.9
openshift_image_tag: "v3.9.0-alpha.3"
# or
# openshift_image_tag: "latest"

thanks for the pointer, so in theory it should work given that PR?

I was able to install a minimal Openshift system on that, but it may fail due to a missing package when additional services are installed

[dougbtv]

Ah ha, thanks @vrutkovs . If that's the case, is it true that releases past v3.7 can only use an Atomic OS? With what you've said, plus that I noticed there aren't RPMs past v3.7 would that be the case? Thanks!

[vrutkovs]

If that's the case, is it true that releases past v3.7 can only use an Atomic OS?

Yes, for future versions I think the only supported case is running containerized etcd on non-Atomic host so far. Note that non-Atomic can run services in system containers though.

With what you've said, plus that I noticed there aren't RPMs past v3.7 would that be the case?

Right, 3.9 (and 3.8) RPMs are not yet released and would also require Atomic hosts for containerized installs

[dougbtv]

Thanks a bunch for that clarification @vrutkovs -- given that is the case then this can be closed on out. Appreciated your input on it.

[g00nix]

I followed the documentation and I didn't see any notification that releases past v3.7 will only work on Atomic hosts. Did I miss it? Is this mentioned in the documentation?

Does this mean that Atomic hosts is required for all node types (master, infra, compute, gluster)?

[vrutkovs]

I didn't see any notification that releases past v3.7 will only work on Atomic hosts

Releases past 3.7 will work on Fedora / CentOS / RHEL hosts as well as Atomic hosts

Does this mean that Atomic hosts is required for all node types (master, infra, compute, gluster)?

No.

This issue is about a configuration where a classic Fedora / CentOS / RHEL hosts is used with containerized = true - this config is not supported as Atomic hosts have necessary packages installed

[g00nix]

This issue is about a configuration where a classic Fedora / CentOS / RHEL hosts is used with containerized = true - this config is not supported as Atomic hosts have necessary packages installed

Is it specified in the documentation that "containerized = true" will only work on atomic hosts?

[ReSearchITEng]

Using the 3.9 on non-atomic and the issue is still there.
Documentation states that it's containerized is officially support, but we fail with the above.
IMHO this issue must be reopen.