Origin: Ensure openshift-ansible installer package deps are installed

[dmage]

The test pull-ci-origin-e2e-gcp currently doesn't work because of a problem with the installer.

attempts: 3
changed: false
rc: 1
item: python-docker

results:

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.advancedhosters.com
 * criotested: mirror.ops.rhcloud.com
 * dockertested: mirror.ops.rhcloud.com
 * epel: mirror.us.leaseweb.net
 * extras: packages.oit.ncsu.edu
 * updates: reflector.westga.edu
Package docker-python is obsoleted by python2-docker, trying to install python2-docker-3.3.0-1.el7.noarch instead
Resolving Dependencies
--> Running transaction check
---> Package python-docker-py.noarch 0:1.10.6-4.el7 will be obsoleted
--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64
---> Package python2-docker.noarch 0:3.3.0-1.el7 will be obsoleting
--> Processing Dependency: python-requests >= 2.14.2 for package: python2-docker-3.3.0-1.el7.noarch
--> Processing Dependency: python-pyOpenSSL for package: python2-docker-3.3.0-1.el7.noarch
--> Processing Dependency: python-idna for package: python2-docker-3.3.0-1.el7.noarch
--> Processing Dependency: python-cryptography for package: python2-docker-3.3.0-1.el7.noarch
--> Running transaction check
---> Package python-docker-py.noarch 0:1.10.6-4.el7 will be obsoleted
--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64
---> Package python-requests.noarch 0:2.6.0-1.el7_1 will be obsoleted
---> Package python2-cryptography.x86_64 0:1.7.2-2.el7 will be installed
--> Processing Dependency: python-cffi >= 1.4.1 for package: python2-cryptography-1.7.2-2.el7.x86_64
--> Processing Dependency: python-enum34 for package: python2-cryptography-1.7.2-2.el7.x86_64
---> Package python2-idna.noarch 0:2.5-1.el7 will be installed
---> Package python2-pyOpenSSL.noarch 0:17.3.0-3.el7 will be installed
---> Package python2-requests.noarch 0:2.19.1-3.el7 will be obsoleting
--> Processing Dependency: python2-urllib3 for package: python2-requests-2.19.1-3.el7.noarch
--> Running transaction check
---> Package python-cffi.x86_64 0:1.6.0-5.el7 will be installed
--> Processing Dependency: python-pycparser for package: python-cffi-1.6.0-5.el7.x86_64
---> Package python-docker-py.noarch 0:1.10.6-4.el7 will be obsoleted
--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64
---> Package python-enum34.noarch 0:1.0.4-1.el7 will be installed
---> Package python-urllib3.noarch 0:1.10.2-5.el7 will be obsoleted
---> Package python2-urllib3.noarch 0:1.16-2.el7 will be obsoleting
--> Processing Dependency: python-pysocks for package: python2-urllib3-1.16-2.el7.noarch
--> Running transaction check
---> Package python-docker-py.noarch 0:1.10.6-4.el7 will be obsoleted
--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64
---> Package python-pycparser.noarch 0:2.14-1.el7 will be installed
--> Processing Dependency: python-ply for package: python-pycparser-2.14-1.el7.noarch
---> Package python2-pysocks.noarch 0:1.6.7-1.el7 will be installed
--> Running transaction check
---> Package python-docker-py.noarch 0:1.10.6-4.el7 will be obsoleted
--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64
---> Package python-ply.noarch 0:3.4-11.el7 will be installed
--> Finished Dependency Resolution
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

msg:

Error: Package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64 (@extras)
           Requires: python-docker-py >= 1.7.2-1
           Removing: python-docker-py-1.10.6-4.el7.noarch (@extras)
               python-docker-py = 1.10.6-4.el7
           Obsoleted By: python2-docker-3.3.0-1.el7.noarch (centos-openshift-origin)
               Not found
           Available: docker-python-1.4.0-115.el7.x86_64 (extras)
               python-docker-py = 1.4.0-115.el7
           Available: python-docker-py-1.10.6-3.el7.noarch (extras)
               python-docker-py = 1.10.6-3.el7

[dmage]

/cc @sdodson

[sdodson]

This is happening because a new version of python2-docker was added to the CentOS PaaS SIG repos and it obsoletes python-docker but it doesn't provide all of the requirements of the atomic package, namely python-docker-py.

@DanyC97 do you know why http://cbs.centos.org/koji/buildinfo?buildID=22923 was tagged into CentOS PaaS SIG repos.

[DanyC97]

@sdodson so the issue was around the errors mentioned by users when they tested 3.10 release - see https://github.com/openshift/openshift-ansible/issues/8399#issuecomment-411727365

as such i went on and added the below pkgs to fix the prob on their side ...

python-docker-3.3.0-1.el7 
pyOpenSSL-17.3.0-3.el7 
python-requests-2.19.1-3.el7 
python-urllib3-1.16-2.el7 
python-idna-2.5-1.el7 
python-pysocks-1.5.6-3.el7

[sdodson]

While https://github.com/openshift/openshift-ansible/pull/9605 attempts to work around the python2-pyOpenSSL problem I can see no way that we can workaround python-docker-3.3 problem short of either rebuilding that package to ensure it meets the requirements of the atomic package or removing it.

[DanyC97]

okay, working to rollback the changes and i'll see how we get then Origin on centOS working again.

[smarterclayton]

Do we have an ETA here?

[DanyC97]

@smarterclayton fyi @bstinsonmhk is already working on this, i've untagged the packages using CBS but seems is more than that to remove the rpms from mirrors.

[DanyC97]

@dmage any chance you can retry and see if it unblocks you?

@bstinsonmhk managed to remove the rpms from the repo metadata.

[sdodson]

Some tests are passing now.
https://openshift-gce-devel.appspot.com/builds/origin-ci-test/pr-logs/directory/pull-ci-openshift-openshift-ansible-e2e-gcp

[sdodson]

Resolved.

[DanyC97]

@markllama if you check https://github.com/openshift/openshift-ansible/issues/8399#issuecomment-413504162 you can see i just tried on a fresh system (including Origin repo from PaaS) myself to see the atomic dependencies and that pulls down python-docker-py 1.10.6-4.el7 for atomic 1.22.1-22.git5a342e3.el7 both coming from extras repo.

so i think we are good, i'll drive the other discussion as i think is been a lot of red herring ...

[nealef]

I am not sure I understand the resolution. Were any changes required? I note that in playbooks/init/base_packages.yml there is

   - "{{ 'python3-docker' if ansible_distribution == 'Fedora' else 'python-docker' }}"

but python-docker-py provides docker-python and the python2-docker package provides python-docker. However, the latter package obsoletes the former so they can't co-exist.

---

Once I got rid of - atomic from the list of packages to install, things went better.

[bilalcaliskan]

I'm still getting this error while installing clean release-3.11 cluster. Python packages version information below. @sdodson could you explain how to work around this issue?

Thanks.

python-2.7.5-69.el7_5
python-libs-2.7.5-69.el7_5.x86_64
ansible-2.7.0-1.el7.noarch

Additionally:
[root@localhost ~]# yum repolist
base/7/x86_64
centos-openshift-origin
centos7basechannel
epel/x86_64
extras/7/x86_64
updates/7/x86_64

[malagant]

Same here with upgrade to 3.11

[jstange]

Also seeing bad behavior here around python-docker here, attempting to install on RHEL 7.5. I've tried a few different versions of openshift-ansible including master, release-3.11, release-3.10, and various tags.

Initially I was getting the error described in https://github.com/openshift/openshift-ansible/issues/10440 ("No package matching 'python-docker' found available"), but I seem to have gotten these machines into a state where I now get the following instead:

fatal: [master-node]: FAILED! => {"attempts": 3, "changed": false, "msg": "Error from repoquery: ['/bin/repoquery', '--show-duplicates', '--plugins', '--quiet', '-c', None, '--disablerepo', '', '--enablerepo', '', '--qf', '%{epoch}:%{name}-%{version}-%{release}.%{arch}', 'python-docker']: Error accessing file for config file:///home/ansibleagent/--disablerepo\nError accessing file for config file:///home/ansibleagent/--disablerepo\n"}

Here's what yum repolist says we have, after an aborted run of the prerequisites playbook:

centos-openshift-origin                         CentOS OpenShift Origin     196+58
epel/x86_64                                     Extra Packages for Enterp 12,741+1
rhel-7-server-extras-rpms/x86_64                Red Hat Enterprise Linux       923
rhel-7-server-optional-rpms/7Server/x86_64      Red Hat Enterprise Linux    15,387
rhel-7-server-rh-common-rpms/7Server/x86_64     Red Hat Enterprise Linux       234
rhel-7-server-rpms/7Server/x86_64               Red Hat Enterprise Linux    21,082
rhel-7-server-supplementary-rpms/7Server/x86_64 Red Hat Enterprise Linux       280
rhel-x86_64-rhev-mgmt-agent-7                   Red Hat Enterprise Virt M      773
rhel-x86_64-server-7                            Red Hat Enterprise Linux    21,082
rhel-x86_64-server-optional-7                   RHEL Server Optional (v.    15,387
rhel-x86_64-server-rh-common-7                  Red Hat Common (for RHEL       234
rhn-tools-rhel-x86_64-server-7                  RHN Tools for RHEL Server      132

EDIT: And I don't see python2-docker on this machine, so that's not in the way of dependencies. If I install python-docker-py manually I still get the same behavior. Also if I uninstall it.

[sst1xx]

I also have the problem with python-docker-py. I am trying install OKD 3.11 on Centos7.5 node-config-all-in-one.
How to fix this issue?

[malagant]

I also have the problem with python-docker-py. I am trying install OKD 3.11 on Centos7.5 node-config-all-in-one.
How to fix this issue?

I had success with uninstalling python2-docker and then running the prerequisites.yml playbook again.
If you had run the playbook before etc. you might have installed a broken version. After uninstall and retry it might work. No guarantee but, give it a try. ;)

[jstange]

FWIW our current hackaround is just to mangle the Ansible to refer to python-docker-py instead of python-docker. No combination of installing/uninstalling the obsoleted packages was helpful.

[DanyC97]

@jstange @malagant @vital-st1x if you check the comment https://github.com/openshift/openshift-ansible/issues/8399#issuecomment-413582800 you can the expectation.

also from the above issue you can see all the problems we had around the python-docker-py & python-docker which broke the CI Origin.

P.S - sorry for late response

[elluvium]

I was able to resolve issue with python-docker-py by setting obsoletes=0 in /etc/yum.conf before running playbook

[mvdelaluz]

For me, these solved my issue:

subscription-manager repos --enable="rhel-7-server-rpms" --enable="rhel-7-server-extras-rpms" --enable="rhel-7-server-ansible-2.6-rpms" --enable="rhel-7-server-ose-3.11-rpms" --enable="rh-gluster-3-client-for-rhel-7-server-rpms";

then; yum update; systemctl reboot; and run again prerequisities.yml

Good Luck!

[Nurlan199206]

@mvdelaluz Thanks!