openshift-ansible 🚀 - Failed to install Openshift 3.10 because of missing packages

@gbenhaim correct the rpms for 3.10 are not yet built, am planning to work on it this week however it won't be published on official repo in the next 2 week. Once will be available as candidate i'll tag you, sorry for delay

DanyC97 on 21 May 2018

@DanyC97 thanks.

gbenhaim on 22 May 2018

👍1

@nellyc

gbenhaim on 5 Jun 2018

@DanyC97 any updates?

gbenhaim on 7 Jun 2018

@gbenhaim sorry no, just got back to work. is on my top priority list, just please bear with me a bit.

DanyC97 on 7 Jun 2018

@gbenhaim : any update or way we could help ?

ticapix on 27 Jul 2018

👍1

@ticapix Is there any official repo for 3.10 rc?

gbenhaim on 31 Jul 2018

👍1

@DanyC97 Repo 3.10 for soon or not yet? Thank's

moula on 2 Aug 2018

😕1

Any news here. Openshift 3.10 has been released by RedHat, however origin is not available

rhnewtron on 2 Aug 2018

👍2

guys i understand the excitement but it doesn't help if everyone is adding a comment on top of another comment.

the reason i haven't released as part of PaaS SIG any origin rpm for v3.10.rc0 is because we couldn't build it due to missing code which was made in OCP but has been ported to Origin but that is now sorted

Once that was done i was waiting for the new tag in Origin repo to be created which was done few hours ago with the official v3.10 release and so i'm working on it to get the rpms out for testing.

DanyC97 on 3 Aug 2018

❤2

in case you haven't seen the notification sent to dev mailing list

Let me know if there are any issues either here or writing to the mailer

DanyC97 on 3 Aug 2018

👍4

The default playbooks/prerequisites.yml and playbooks/deploy_cluster.yml are working on the first try on a fresh Centos7 install with the default inventory/hosts.localhost config.
The only tweak was

pip uninstall ansible || true
yum -y install yum # this will install ansible-2.4
pip install --upgrade ansible  # this will install ansible-2.6

Otherwise if using ansible-2.6 installed from pip and without having the package installed from the distrib package manager, the playbook will force the install of 2.4 version via yum and it messes up the script at some point. (error about environment 'shell' var not being defined)

ticapix on 3 Aug 2018

eh ?

to my knowledge 3.10 is working with 2.4, @sdodson am i missing something here ?
only 3.11 will require 2.6 ansible

You saying that deploying 3.10 with ansible 2.4 doesn't work ?

DanyC97 on 3 Aug 2018

I'm saying that if ansible 2.6 is installed via pip before running the
playbook it fails. I'll try with the default 2.4 version from yum.

Le ven. 3 août 2018 à 20:19, Dani Comnea notifications@github.com a
écrit :

eh ?

to my knowledge 3.10 is working with 2.4, @sdodson
https://github.com/sdodson am i missing something here ?
only 3.11 will require 2.6 ansible

You saying that deploying 3.10 with ansible 2.4 doesn't work ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/openshift-ansible/issues/8399#issuecomment-410336067,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC8k_nAGF8vUiHp0ytocGgoUzKHex2ypks5uNJQsgaJpZM4UCqDY
.

ticapix on 3 Aug 2018

right which is what i expected @ticapix .
As mentioned 2.6 is only for 3.11 which is not released, is work in progress in master(dev) branch.

DanyC97 on 3 Aug 2018

I just installed but still getting error
OS Version: CentOS 7.5

git clone https://github.com/openshift/openshift-ansible.git
cd openshift-ansible && git fetch && git checkout release-3.10 && cd ..
ansible-playbook -i inventory.ini openshift-ansible/playbooks/prerequisites.yml
ansible-playbook -i inventory.ini openshift-ansible/playbooks/deploy_cluster.yml

Following is error

No package matching 'origin-clients-3.10*' found available, installed or updated

imranrazakhan on 6 Aug 2018

@imranrazakhan which yum repo have you used ?

If you look here the rpm is present.

the promotion to the other repos have not taken place due to few people responsible with the CentOS repo traveling.

when promotion will happen i'm going to send another notification.

DanyC97 on 6 Aug 2018

@imranrazakhan production repos won't work yet.

check dev mailing list for testing RPMs: http://lists.openshift.redhat.com/openshift-archives/dev/2018-August/msg00000.html

Anyone to post how to activate them in ansible?

isaldarriaga on 6 Aug 2018

@imranrazakhan : add the testing repo

cat > /etc/yum.repos.d/CentOS-OpenShift-Origin-CBS.repo <<EOF
[centos-openshift-origin-testing-cbs]
name=CentOS OpenShift Origin Testing CBS
baseurl=https://cbs.centos.org/repos/paas7-openshift-origin310-testing/x86_64/os/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS
EOF

ticapix on 6 Aug 2018

👍1

@ticapix and you saying with the above repo things still don't work ?

DanyC97 on 6 Aug 2018

@DanyC97 @imranrazakhan
For my 3.10 installation I used the following baseurl in the .repo file:
https://cbs.centos.org/repos/paas7-openshift-origin310-candidate/x86_64/os/Packages/

The installation of the package origin-node-3.10.0 worked without problems.
I used for the installation the Ansible version 2.6.2. and the openshift-ansible repo with the branch release-3.10.
Only the openshift-infra deployment failed because of image pull errors of version v3.10, it seems not be build already. Version latest worked for my test without problems.

chrisop on 7 Aug 2018

@DanyC97 I was able to upgrade to version 3.10 with the provided rpms. Is there anything I can do for you to help you to get this RPM into production?
@isaldarriaga At the moment it's not possibly to activate this repo with openshift-ansible
@ticapix Thanks for your repo definition. I copied this definition via ansible ad hoc command to my cluster nodes

Reamer on 7 Aug 2018

@Reamer thank you kindly for your help, there is nothing major left now that you confirmed. I just sent out another notification with the next steps.

DanyC97 on 7 Aug 2018

@Reamer FYI i set the repo in all host as per @ticapix instructions and ansible was able to pull the images without problem.

My installation is not working though.

i traced down the error to the api control plane pod:

$ master-logs api $(docker ps | grep api | awk '{print $1}')
json: cannot unmarshal array into Go value of type types.ContainerJSON

the pod's yaml looks fine, but not sure if causing the problem or not:

$ cat /etc/origin/node/pods/apiserver.yaml

https://gist.github.com/isaldarriaga/3bedaf1dfa731485b6783a5f289740d6

docker logs shows nothing.

i couldn't find anything in /var/log for /usr/bin/pod in the api control plane's container either:

$ docker exec -it $(docker ps | grep api | awk '{print $1}') ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
1001 1 0.0 0.0 2764 720 ? Ssl Aug07 0:00 /usr/bin/pod
1001 44 2.0 0.0 51708 1652 ? Rs+ 00:57 0:00 ps aux

$ docker exec -it $(docker ps | grep api | awk '{print $1}') ls /var/log
btmp grubby_prune_debug lastlog tallylog wtmp yum.log

how to increase the verbosity in master-logs / docker container? already is 8 in ansible and control plane looks already set to that.

$ cat /etc/origin/master/master.env

https://gist.github.com/isaldarriaga/d9a836ef6700df0b0f43fa5b9c76b82d

This is the docker's config just in case:

$ cat /etc/sysconfig/docker

https://gist.github.com/isaldarriaga/2fcddaea8b53d74279f64236695ef55c

centos version (master node):
$ uname -a
Linux prod-master-01.example.com 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
https://gist.github.com/isaldarriaga/29397ec40caac0aa1888d9f81730ba08

=========================
Fedora version (remote ansible):
$ uname -a
Linux dev-ops 4.17.9-200.fc28.x86_64 #1 SMP Mon Jul 23 21:41:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Ansible version:
https://gist.github.com/isaldarriaga/84583361107c0eb38e107ab28e3dc39c

Inventory file:
https://gist.github.com/isaldarriaga/4af701c1ed8b4e8a47a752a5e48011ff

ansible log (verbosity = 8)
ansible-2018-08-07T18:32:37.zip

* edit: adding long output to gist *

isaldarriaga on 8 Aug 2018

@isaldarriaga It was very helpful to start the openshift api service manual with openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=${DEBUG_LOGLEVEL:-2}
Then you should be able to look via webconsole into the output log of your failed pods in namespace kube-system. In my case it was an old vsphere config file /etc/vsphere/vsphere.conf - custom stuff...

Reamer on 8 Aug 2018

@Reamer i started the api that way.

$ openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8
https://gist.github.com/isaldarriaga/0f9e0a536ae6c414951609d9c2d8c877

The $DEBUG_LOGLEVEL is empty by default:
$ env
https://gist.github.com/isaldarriaga/9ff9da4c8e842d2f4aabd6f7dc28093d

$ cat /etc/origin/master/master-config.yaml
https://gist.github.com/isaldarriaga/975f86539c12825de643d4944547d625

openshift start master api command's last output is :
Invalid MasterConfig /etc/origin/master/master-config.yaml
etcdClientInfo.ca: Invalid value: "/etc/origin/master/master.etcd-ca.crt": could not read file: stat /etc/origin/master/master.etcd-ca.crt: no such file or directory

$ ls -al /etc/origin/master/
https://gist.github.com/isaldarriaga/6bd94aec66473a15e1315a57519b2ded

Since the api didn't come up, i cannot get anything from oc:

$ oc get pods -n kube-system
The connection to the server prod-master-01.example.com:443 was refused - did you specify the right host or port?

This is what i attempted:

a) $ cp /etc/origin/master/ca-bundle.crt /etc/origin/master/master.etcd-ca.crt && openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8
. . .
F0808 06:07:12.757726 25569 start_api.go:68] dial tcp 192.168.246.111:2379: getsockopt: connection refused

the master-01 (.111) attempts to reach itself over :2379

b) $ cp /etc/origin/master/ca.crt /etc/origin/master/master.etcd-ca.crt && openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8
. . .
F0808 06:13:58.080386 26172 start_api.go:68] dial tcp 192.168.246.112:2379: getsockopt: no route to host

the master-01 (.111) attempts to reach master-02 (.112) over :2379

$ iptables --list-rules
https://gist.github.com/isaldarriaga/eba36ac855ce37844c7f0b10eac8615f

c) $ cp /etc/origin/master/client-ca-bundle.crt /etc/origin/master/master.etcd-ca.crt && openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8
. . .
F0808 05:54:39.059864 24454 start_api.go:68] dial tcp 192.168.246.112:2379: getsockopt: no route to host

the master-01 (.111) attempts to reach master-02 (.112) over :2379

d) $ cp /etc/origin/master/frontproxy-ca.crt /etc/origin/master/master.etcd-ca.crt && openshift start master api --config=/etc/origin/master/master-config.yaml --loglevel=8
. . .
F0808 06:26:58.111176 27244 start_api.go:68] dial tcp 192.168.246.113:2379: getsockopt: no route to host

the master-01 (.111) attempts to reach master-03 (.113) over :2379

so there is a number of problems:

DEBUG_LOGLEVEL env is not populated during installation, affecting openshift start master api verbosity
/etc/origin/master/master.etcd-ca.crt is not generated before starting up api pod. Though other certs are.
master-logs api $(docker ps | grep api | awk '{print $1}') command's output does not correspond to
openshift start master api command's output. maybe caused by 1)
iptables :2379 rule is missing in all masters

isaldarriaga on 8 Aug 2018

/etc/origin/master/master.etcd-ca.crt should be in your folder. My file is quite old. Generated from openshift-ansible during the installation of 3.7. Take a look into your backup if you can recover this file.
On Port 2379 etcd is running. Is your etcd-cluster up?

Reamer on 8 Aug 2018

so i'm a bit confused @Reamer @isaldarriaga ..

running the Ansible deployment of 3.10 what error you got ?

DanyC97 on 8 Aug 2018

@DanyC97 This has nothing to do with missing packages.
@isaldarriaga Please open a new issue for your problem.

Reamer on 8 Aug 2018

okay @Reamer , thank you for confirmation. But can you confirm if you you managed to get a working/ running 3.10 cluster ?

DanyC97 on 8 Aug 2018

I can confirm that i have now a running 3.10 cluster with your provided rpms using the openshift-ansible upgrade playbook.

Reamer on 8 Aug 2018

thank you v much for confirmation @Reamer !
@gbenhaim can you please close this issue that that the rpms been provided ? i'll complete the final step on promoting to mirror.centos in the next few hours and it should get in sync by tmw.

DanyC97 on 8 Aug 2018

@DanyC97, ansible detected one missing rpm (python-docker).

on TASK [Ensure openshift-ansible installer package deps are installed]:
"item": "python-docker",
"msg": "Failure talking to yum: failure: repodata/repomd.xml from centos-openshift-origin-testing-cbs: [Errno 256] No more mirrors to try.\nhttps://cbs.centos.org/repos/paas7-openshift-origin310-testing/x86_64/os/Packages/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found"

there's no such rpm in any of the 3 referenced openshift repos.

but here:
https://rpmfind.net/linux/rpm2html/search.php?query=python-docker-py

I reinstalled having etcd in separate hosts this time.

inventory:
https://gist.github.com/isaldarriaga/43ed5c19a32f4b2ed778a7b59952439a

install logs (verbose = 8)
ansible.zip

isaldarriaga on 9 Aug 2018

@isaldarriaga the error you showing me is related to missing the python-docker rpm but then you are showing me in rpmfind.net the python-docker-py package .

Now looking in the code i can see this so i suspect it was a typo ?

DanyC97 on 9 Aug 2018

and for your info @isaldarriaga i just tagged python-docker-3.3.0-1.el7 into testing repo, any chance you can give it another run ?

DanyC97 on 9 Aug 2018

you're right. i was trying to point out to the existence of the package, but typo.

My lab's hosts is centos7, my client fedora28 (ansible runs here).

Does the code mean if ansible running in fedora, no matter the host's OS, it should install python-docker instead of python3-docker?

now seeing python2-docker-3.3.0-1.el7.noarch.rpm in centos CBS/buildlogs.

thanks, i'm going to retry.

isaldarriaga on 9 Aug 2018

Does the code mean if ansible running in fedora, no matter the host's OS, it should install python-docker instead of python3-docker?

no, what the code does is:

if the host's OS where you are deploying OpenShift (provided by hosts: "{{ l_base_packages_hosts | default('oo_all_hosts') }}" is:

fedora => then python3-docker is installed
everything else => then python-docker is installed

DanyC97 on 9 Aug 2018

ok. Some dependencies are missing when yum -y update:

Error: Package: python2-docker-3.3.0-1.el7.noarch (centos-openshift-origin-testing-cbs)
Requires: python-requests >= 2.14.2
Installed: python-requests-2.6.0-1.el7_1.noarch (@base)
python-requests = 2.6.0-1.el7_1
Error: Package: python2-docker-3.3.0-1.el7.noarch (centos-openshift-origin-testing-cbs)
Requires: python-pyOpenSSL
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest

I'm connected to https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/

Full output here:
https://gist.github.com/isaldarriaga/4203a08fbd5455bc0587c52dfe75205d

isaldarriaga on 9 Aug 2018

It works for me

Thank's

2018-08-09 14:39 GMT+02:00 Ivan Saldarriaga notifications@github.com:

ok. Some dependencies are missing when yum -y update:

Error: Package: python2-docker-3.3.0-1.el7.noarch
(centos-openshift-origin-testing-cbs)
Requires: python-requests >= 2.14.2
Installed: python-requests-2.6.0-1.el7_1.noarch (@base
https://github.com/base)
python-requests = 2.6.0-1.el7_1
Error: Package: python2-docker-3.3.0-1.el7.noarch
(centos-openshift-origin-testing-cbs)
Requires: python-pyOpenSSL
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest

I'm connected to https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/

Full output here:
https://gist.github.com/isaldarriaga/4203a08fbd5455bc0587c52dfe75205d

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/openshift/openshift-ansible/issues/8399#issuecomment-411743000,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABuKD-HRYLAPvdfsvty55FvFWYuDZMm-ks5uPC2XgaJpZM4UCqDY
.

moula on 9 Aug 2018

ok. i'm going to try with a fresh install to confirm.

isaldarriaga on 9 Aug 2018

fyi i just tagged pyOpenSSL-17.3.0-3.el7 & python-requests-2.19.1-3.el7 and it should appear in next 10-15 min in the repo.

Saying that will be good if you can try 2 more fresh installed:

with the packages i just added => let me know the result
then i'll remove the packages => see if you get same like @moula in which case there won't be any need for the last 2 rpms

DanyC97 on 9 Aug 2018

actually ran the installer from fresh install before the new packages to appear in the repo.

failed in the prerequisites.

"item": "python-docker",
"msg": "Error: Package: python2-docker-3.3.0-1.el7.noarch (centos-openshift-origin-testing-cbs)\n Requires: python-requests >= 2.14.2\n Available: python-requests-2.6.0-1.el7_1.noarch (base)\n python-requests = 2.6.0-1.el7_1\nError: Package: python2-docker-3.3.0-1.el7.noarch (centos-openshift-origin-testing-cbs)\n Requires: python-pyOpenSSL\n",

looks like they are actually needed.

i'll wait an retry. many thanks.

isaldarriaga on 9 Aug 2018

@DanyC97 so far python2-pyOpenSSL-17.3.0-3.el7.noarch.rpm and python2-requests-2.19.1-3.el7.noarch.rpm appear in CBS, but not in buildlogs.

if i change the repo's baseurl from https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/ to https://cbs.centos.org/repos/paas7-openshift-origin310-testing/x86_64/os/Packages/ and try to yum -y update i get:

https://cbs.centos.org/repos/paas7-openshift-origin310-testing/x86_64/os/Packages/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found

The same error appear when running ansible's prerequisites.yaml too.

will the rpm appear in buildlogs? or better to add repodata folder in CBS?

isaldarriaga on 9 Aug 2018

update:

python2-idna
python2-urllib3

required/missing

using: https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/

full output (ansible prerequisites):
https://gist.github.com/isaldarriaga/1c3bcc0cb6a470b42d1a48e021b741e3

isaldarriaga on 9 Aug 2018

so far python2-pyOpenSSL-17.3.0-3.el7.noarch.rpm and python2-requests-2.19.1-3.el7.noarch.rpm appear in CBS, but not in buildlogs.

they are now

python2-idna
python2-urllib3

will add them soon

DanyC97 on 9 Aug 2018

@isaldarriaga here we go: python-idna-2.5-1.el7 & python-urllib3-1.16-2.el7 added to the repo, it should appear in next 10-30 min.

any chance you can give it another go and let me know how it goes ? thx

DanyC97 on 10 Aug 2018

yes, this time I'll add the [lb] (haproxy) to see if more packages are needed or not.

isaldarriaga on 10 Aug 2018

@DanyC97, got a new error

{
"attempts": 3,
"changed": false,
"invocation": {
"module_args": {
"allow_downgrade": false,
"bugfix": false,
"conf_file": null,
"disable_gpg_check": false,
"disable_plugin": [],
"disablerepo": null,
"enable_plugin": [],
"enablerepo": null,
"exclude": null,
"install_repoquery": true,
"installroot": "/",
"list": null,
"name": [
"python-docker"
],
"security": false,
"skip_broken": false,
"state": "present",
"update_cache": false,
"update_only": false,
"validate_certs": true
}
},
"item": "python-docker",
"msg": "Error: Package: python2-urllib3-1.16-2.el7.noarch (centos-openshift-origin-testing-cbs)\n Requires: python-pysocks\n",
"rc": 1,
"results": [
"Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: linux.mirrors.es.net\n * extras: centos.s.uw.edu\n * updates: mirror.sjc02.svwh.net\nPackage docker-python is obsoleted by python2-docker, trying to install python2-docker-3.3.0-1.el7.noarch instead\nResolving Dependencies\n--> Running transaction check\n---> Package python2-docker.noarch 0:3.3.0-1.el7 will be installed\n--> Processing Dependency: python-websocket-client >= 0.32.0 for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-six >= 1.4.0 for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-requests >= 2.14.2 for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-docker-pycreds >= 0.2.1 for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-backports-ssl_match_hostname >= 3.5 for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-pyOpenSSL for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-idna for package: python2-docker-3.3.0-1.el7.noarch\n--> Processing Dependency: python-cryptography for package: python2-docker-3.3.0-1.el7.noarch\n--> Running transaction check\n---> Package python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7 will be installed\n--> Processing Dependency: python-backports for package: python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch\n---> Package python-docker-pycreds.noarch 0:1.10.6-4.el7 will be installed\n---> Package python-six.noarch 0:1.9.0-2.el7 will be installed\n---> Package python-websocket-client.noarch 0:0.32.0-116.el7 will be installed\n---> Package python2-cryptography.x86_64 0:1.7.2-2.el7 will be installed\n--> Processing Dependency: python-pyasn1 >= 0.1.8 for package: python2-cryptography-1.7.2-2.el7.x86_64\n--> Processing Dependency: python-cffi >= 1.4.1 for package: python2-cryptography-1.7.2-2.el7.x86_64\n--> Processing Dependency: python-setuptools for package: python2-cryptography-1.7.2-2.el7.x86_64\n--> Processing Dependency: python-enum34 for package: python2-cryptography-1.7.2-2.el7.x86_64\n---> Package python2-idna.noarch 0:2.5-1.el7 will be installed\n---> Package python2-pyOpenSSL.noarch 0:17.3.0-3.el7 will be installed\n---> Package python2-requests.noarch 0:2.19.1-3.el7 will be installed\n--> Processing Dependency: python2-urllib3 for package: python2-requests-2.19.1-3.el7.noarch\n--> Running transaction check\n---> Package python-backports.x86_64 0:1.0-8.el7 will be installed\n---> Package python-cffi.x86_64 0:1.6.0-5.el7 will be installed\n--> Processing Dependency: python-pycparser for package: python-cffi-1.6.0-5.el7.x86_64\n---> Package python-enum34.noarch 0:1.0.4-1.el7 will be installed\n---> Package python-setuptools.noarch 0:0.9.8-7.el7 will be installed\n---> Package python2-pyasn1.noarch 0:0.1.9-7.el7 will be installed\n---> Package python2-urllib3.noarch 0:1.16-2.el7 will be installed\n--> Processing Dependency: python-pysocks for package: python2-urllib3-1.16-2.el7.noarch\n--> Running transaction check\n---> Package python-pycparser.noarch 0:2.14-1.el7 will be installed\n--> Processing Dependency: python-ply for package: python-pycparser-2.14-1.el7.noarch\n---> Package python2-urllib3.noarch 0:1.16-2.el7 will be installed\n--> Processing Dependency: python-pysocks for package: python2-urllib3-1.16-2.el7.noarch\n--> Running transaction check\n---> Package python-ply.noarch 0:3.4-11.el7 will be installed\n---> Package python2-urllib3.noarch 0:1.16-2.el7 will be installed\n--> Processing Dependency: python-pysocks for package: python2-urllib3-1.16-2.el7.noarch\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"
]
}

isaldarriaga on 10 Aug 2018

@isaldarriaga sorted, python-pysocks-1.5.6-3.el7 added, give it 10-15 min pls to show up

DanyC97 on 10 Aug 2018

prerequisites passed. running playbooks/deploy_cluster.yml now.

isaldarriaga on 10 Aug 2018

🎉1

@isaldarriaga thank you v much for not giving up and work with me on finding all missing dependencies

once you finish then i'll tag all the rpm into the public repo

DanyC97 on 10 Aug 2018

My pleasure @DanyC97

This is another one when running deploy_cluster.yml

{
"attempts": 3,
"changed": false,
"invocation": {
"module_args": {
"allow_downgrade": false,
"bugfix": false,
"conf_file": null,
"disable_gpg_check": false,
"disable_plugin": [],
"disablerepo": null,
"enable_plugin": [],
"enablerepo": null,
"exclude": null,
"install_repoquery": true,
"installroot": "/",
"list": null,
"name": [
"atomic"
],
"security": false,
"skip_broken": false,
"state": "present",
"update_cache": false,
"update_only": false,
"validate_certs": true
}
},
"item": "atomic",
"msg": "Error: Package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64 (extras)\n Requires: python-docker-py >= 1.7.2-1\n Available: docker-python-1.4.0-115.el7.x86_64 (extras)\n python-docker-py = 1.4.0-115.el7\n Available: python-docker-py-1.10.6-3.el7.noarch (extras)\n python-docker-py = 1.10.6-3.el7\n Available: python-docker-py-1.10.6-4.el7.noarch (extras)\n python-docker-py = 1.10.6-4.el7\n",
"rc": 1,
"results": [
"Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: centos.mirror.lstn.net\n * extras: mirror.hostduplex.com\n * updates: centos.mirror.lstn.net\nResolving Dependencies\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n--> Processing Dependency: runc for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Processing Dependency: python-dateutil for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Processing Dependency: gomtree for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n---> Package gomtree.x86_64 0:0.5.0-0.2.git16da0f8.el7 will be installed\n---> Package python-dateutil.noarch 0:1.5-7.el7 will be installed\n---> Package runc.x86_64 0:1.0.0-27.rc5.dev.git4bb1fe4.el7 will be installed\n--> Processing Dependency: criu for package: runc-1.0.0-27.rc5.dev.git4bb1fe4.el7.x86_64\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n---> Package criu.x86_64 0:3.5-4.el7 will be installed\n--> Processing Dependency: libprotobuf-c.so.1(LIBPROTOBUF_C_1.0.0)(64bit) for package: criu-3.5-4.el7.x86_64\n--> Processing Dependency: libprotobuf-c.so.1()(64bit) for package: criu-3.5-4.el7.x86_64\n--> Processing Dependency: libnet.so.1()(64bit) for package: criu-3.5-4.el7.x86_64\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n---> Package libnet.x86_64 0:1.1.6-7.el7 will be installed\n---> Package protobuf-c.x86_64 0:1.0.2-3.el7 will be installed\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"
]
}

isaldarriaga on 10 Aug 2018

👍1

@isaldarriaga now this one is confusing me ...
On which operating system are you deploying OpenShift ? i assumed is centOS in which case atomic pkg should not be installed.

Can you pls confirm ?
Also please share your inventory file and confirm your openshift-ansible release version (i hope is the release openshift-ansible-3.10.27-1 and not the tip of 3.10 branch ? )

DanyC97 on 11 Aug 2018

@DanyC97 I'm deploying over CentOS Linux release 7.5.1804 (Core) / proxmox VMs.

I was checking-out the 3.10 branch without tags :|, and using the following line in the inventory:
openshift_release=v3.10

inventory.ini

==============

so i've switched over the tag:

$ VERSION="v3.10" && TAG="openshift-ansible-3.10.27-1" && \
mkdir --parents $TAG && \
git clone https://github.com/openshift/openshift-ansible $TAG && \
cd $TAG && \
git fetch --all --tags --prune && \
git checkout tags/$TAG -b release-$VERSION
echo && echo "Current tag is:" && \
git describe --tags && \
cd ..

Current tag is:
openshift-ansible-3.10.27-1

i did alter the inventory too:
openshift_release=v3.10
openshift_pkg_version=openshift-ansible-3.10.27-1

I ran prerequisites.yaml over a fresh install:

"msg": "No package matching 'origin-docker-excluder-openshift-ansible-3.10.27-1*' found available, installed or updated",

I think this is the one that prevent yum to manually update docker, there's another one for openshift (#7794).

We already have these RPMs in the repo:

origin-docker-excluder-3.10.0-1.el7.git.0.0c4577e.noarch.rpm
origin-excluder-3.10.0-1.el7.git.0.0c4577e.noarch.rpm

probably what we need is to push *-3.10.27-1 instead of whatever we have in the repo as *-3.10.0-1 and test again.

ansible.log

isaldarriaga on 11 Aug 2018

@isaldarriaga so few things:

1) you don't need to switch to tags, there is an rpm for ansible openshift-ansible-3.10.27-1.git.0.d5723a3.el7.noarch.rpm here
2) thank you for pointing to the other issue, the prob i see is that http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin/ has not been updated except http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin310/ -> i'll fix that.

please note that all the other missing python-* packages were pushed to mirror.centos.org and that will appear Monday morning after 09:00 UTC (a scheduled job needs to run to sign and publish the packages)

DanyC97 on 12 Aug 2018

@isaldarriaga can you please confirm if your yum repo is setup like i mentioned here ?

i just want to see if you get same output like @danscarf

DanyC97 on 12 Aug 2018

@DanyC97 not exactly as you expected

i was initially using:

$ cat /etc/yum.repos.d/CentOS-OpenShift-Origin-CBS.repo

[centos-openshift-origin-testing-cbs]
name=CentOS OpenShift Origin Testing CBS
baseurl=https://cbs.centos.org/repos/paas7-openshift-origin310-testing/x86_64/os/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS

(as posted in this thread)

then moved and changed the file to be:

$ cat /etc/yum.repos.d/CentOS-OpenShift-Origin-buildlogs.repo

[centos-openshift-origin-testing-buildlogs]
name=CentOS OpenShift Origin Testing buildlogs
baseurl=https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS

(upon your notification)

i noticed Ansible creates this file during installation:

$ cat /etc/yum.repos.d/CentOS-OpenShift-Origin.repo

[centos-openshift-origin]
name=CentOS OpenShift Origin
baseurl=http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-PaaS

[centos-openshift-origin-testing]
name=CentOS OpenShift Origin Testing
baseurl=http://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin/
enabled=0
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS

[centos-openshift-origin-debuginfo]
name=CentOS OpenShift Origin DebugInfo
baseurl=http://debuginfo.centos.org/centos/7/paas/x86_64/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS

[centos-openshift-origin-source]
name=CentOS OpenShift Origin Source
baseurl=http://vault.centos.org/centos/7/paas/Source/openshift-origin/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS

which looks pretty similar to what you posted in 7794

My thinking is ansible should be putting the version correctly according to openshift_release=v3.10 in the inventory file.

If ok, what i'm going to do is to script the modification of /etc/yum.repos.d/CentOS-OpenShift-Origin.repo file over all the hosts before installation, so it everything is pulled from the 3.10 folder in the repo. and run a fresh install as soon as the RPMs become available on monday. I'll remove the switching to the openshift-ansible-3.10.27-1 tag too.

thanks.

isaldarriaga on 12 Aug 2018

i noticed Ansible creates this file during installation:

oh !! thank you for this vital info, i'll fire a PR and fix this too in addition to what i mentioned in my previous comment

_update_ - PR created

DanyC97 on 12 Aug 2018

🎉1

Hi,

I am afraid that this patch is not enough to resolve this problem.

1 ) https://github.com/openshift/openshift-ansible/blob/release-3.10/roles/openshift_repos/tasks/centos_repos.yml#L22 - The file with name CentOS-OpenShift-Origin31.repo.j2 (not Origin310) should be taken.

2) But because {{ openshift_version }} is not defined in this line the file CentOS-OpenShift-Origin.repo.j2 will be used. I suppose the role version should be before role repo
https://github.com/openshift/openshift-ansible/blob/release-3.10/playbooks/init/main.yml#L27-L35

Please correct me if I'm wrong
Thanks

sst1xx on 13 Aug 2018

@vital-st1x please comment directly on the PR and not on this issue.

Trying to keep the history of all the comments close to the initial problem of missing rpms.
I'll look at your comments and respond on the PR itself, thanks !

DanyC97 on 13 Aug 2018

👍1

@DanyC97 i patched my local openshift-ansible repo with the changes that you performed in the PR.
I also removed any origin's .repo file from the servers (fresh machines setup).

I can tell the new code worked! i.e. the right /etc/yum.repos.d/CentOS-OpenShift-Origin310.repo file was created and pre-requisites ran correctly.

looking forward they become available in the branch after approval/tests/commit.

now I'm running deploy_cluster.yaml.

so far so good. I'll keep you posted of the final results on this part.

isaldarriaga on 13 Aug 2018

🎉1

@DanyC97 i got this:

TASK [openshift_node : install needed rpm(s)] *************************************
task path: /home/ivan/convergence.cloud/dev-ops/kube-prod/origin/openshift-ansible-v3.10/roles/openshift_node/tasks/install_rpms.yml:2

...

{
"attempts": 3,
"changed": false,
"invocation": {
"module_args": {
"allow_downgrade": false,
"bugfix": false,
"conf_file": null,
"disable_gpg_check": false,
"disable_plugin": [],
"disablerepo": null,
"enable_plugin": [],
"enablerepo": null,
"exclude": null,
"install_repoquery": true,
"installroot": "/",
"list": null,
"name": [
"atomic"
],
"security": false,
"skip_broken": false,
"state": "present",
"update_cache": false,
"update_only": false,
"validate_certs": true
}
},
"item": "atomic",
"msg": "Error: Package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64 (extras)\n Requires: python-docker-py >= 1.7.2-1\n Available: docker-python-1.4.0-115.el7.x86_64 (extras)\n python-docker-py = 1.4.0-115.el7\n Available: python-docker-py-1.10.6-3.el7.noarch (extras)\n python-docker-py = 1.10.6-3.el7\n Available: python-docker-py-1.10.6-4.el7.noarch (extras)\n python-docker-py = 1.10.6-4.el7\n",
"rc": 1,
"results": [
"Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: mirrors.xmission.com\n * extras: repos-lax.psychz.net\n * updates: repos.lax.quadranet.com\nResolving Dependencies\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n--> Processing Dependency: runc for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Processing Dependency: python-dateutil for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Processing Dependency: gomtree for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n---> Package gomtree.x86_64 0:0.5.0-0.2.git16da0f8.el7 will be installed\n---> Package python-dateutil.noarch 0:1.5-7.el7 will be installed\n---> Package runc.x86_64 0:1.0.0-27.rc5.dev.git4bb1fe4.el7 will be installed\n--> Processing Dependency: criu for package: runc-1.0.0-27.rc5.dev.git4bb1fe4.el7.x86_64\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n---> Package criu.x86_64 0:3.5-4.el7 will be installed\n--> Processing Dependency: libprotobuf-c.so.1(LIBPROTOBUF_C_1.0.0)(64bit) for package: criu-3.5-4.el7.x86_64\n--> Processing Dependency: libprotobuf-c.so.1()(64bit) for package: criu-3.5-4.el7.x86_64\n--> Processing Dependency: libnet.so.1()(64bit) for package: criu-3.5-4.el7.x86_64\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n---> Package libnet.x86_64 0:1.1.6-7.el7 will be installed\n---> Package protobuf-c.x86_64 0:1.0.2-3.el7 will be installed\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"
]
}

Seems like the atomic dependency is not handled correctly at this point because i explicitly refused its use in the inventory file:
containerized=false as node and [OSEv3:vars] attributes

and as far as i know it's deprecated in 3.10 too.

i ran it in fresh install twice, getting the same error.
Also made sure the git branch was correct (release-3.10, no tags)

by the other hand the inventory includes:

openshift_release=v3.10
openshift_repos_enable_testing=true

which generates:

$ cat /etc/yum.repos.d/CentOS-OpenShift-Origin310.repo

[centos-openshift-origin310]
name=CentOS OpenShift Origin
baseurl=http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin310/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-PaaS

[centos-openshift-origin310-testing]
name=CentOS OpenShift Origin Testing
baseurl=http://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin310/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-PaaS

[centos-openshift-origin310-debuginfo]
name=CentOS OpenShift Origin DebugInfo
baseurl=http://debuginfo.centos.org/centos/7/paas/x86_64/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-PaaS

[centos-openshift-origin310-source]
name=CentOS OpenShift Origin Source
baseurl=http://vault.centos.org/centos/7/paas/Source/openshift-origin310/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-PaaS[ivan@asus-rog openshift-ansible-v3.10]

i attached the output at verbosity 8:

ansible.log

I'll take a look to this output tomorrow. if you find something let me know.

thanks.

isaldarriaga on 14 Aug 2018

@isaldarriaga this was reported by someone else too on the dev mailer. I would suggest open a new issue as this is very likely a bug (atomic should not be installed)

DanyC97 on 14 Aug 2018

After checking out openshift-ansible-3.10.27-1 tag this atomic issue didn't happen.

aramalipoor on 14 Aug 2018

👍2

@DanyC97 Thank you for publishing the origin 3.10 rpm also under openshift-origin
My colleague and I updated our production cluster a few minutes ago.
We doens't use the openshift-ansible rpm. To get the openshift-ansible we are using git with branch release-3.10.

Reamer on 14 Aug 2018

@Reamer my pleasure ;) . I still need to publish some rpms into common sub dir but we hit a minor issue which i'm working on with CentOS infra folks.

regarding using or not the rpm, is up to you however i'd suggest if you using git to at least stick to the tag and not tip of the branch. Things are fluid as you can see so you might not always get the same result

DanyC97 on 14 Aug 2018

@Reamer @sebile @isaldarriaga fyi the last piece of the puzzle has been sorted, the job run this morning updated the common repo

Also tomorrow morning after 09:00 UTC you should see a new rpm openshift-ansible-3.10.28-1.git.0.9242c73 which includes the @vrutkovs 's fix around openvswitch rpm which is no longer required.

DanyC97 on 15 Aug 2018

After tomorrow's job we should all be aligned and have no rpm issues anymore, hence we could close this issue.

Thank you for your patience while sailing through.

DanyC97 on 15 Aug 2018

@DanyC97 What's the difference between openshift-origin/common and openshift-origin?

Reamer on 15 Aug 2018

FYI, I've asked @DanyC97 to untag those packages from the CentOS PaaS SIG repos because they introduce package dependency issues. I'll work on reading up on this issue and proposing an alternate solution and/or set of correct packages to tag into those repos.

sdodson on 15 Aug 2018

i just tried again

OS Version: CentOS 7.5

git clone https://github.com/openshift/openshift-ansible.git
cd openshift-ansible && git fetch && git checkout release-3.10 && cd ..
ansible-playbook -i inventory.ini openshift-ansible/playbooks/prerequisites.yml
ansible-playbook -i inventory.ini openshift-ansible/playbooks/deploy_cluster.yml

Following is error

{  
   "attempts":3,
   "changed":false,
   "item":"atomic",
   "msg":"Error: Package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64 (extras)\n           Requi 
res: python-docker-py >= 1.7.2-1\n           Available: docker-python-1.4.0-115.el7.x86_64 (extras)\n               python-docker-py = 1.4.0-115.el7\n",
   "rc":1,
   "results":[  
      "Loade 
d plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: ftp.fau.de\n * epel: mirrors.dotsrc.org\n * extras: mirror2.hs-esslingen.de\n * updates: mirror2.hs-e 
sslingen.de\nResolving Dependencies\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: skopeo >= 1: 
0.1.29-3 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Pr 
ocessing Dependency: runc for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Processing Dependency: python-dateutil for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n- 
-> Processing Dependency: ostree for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Processing Dependency: gomtree for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n-- 
> Running transaction check\n---> Package gomtree.x86_64 0:0.5.0-0.2.git16da0f8.el7 will be installed\n---> Package ostree.x86_64 0:2018.5-1.el7 will be installed\n--> Processing D 
ependency: libarchive.so.13()(64bit) for package: ostree-2018.5-1.el7.x86_64\n---> Package python-dateutil.noarch 0:1.5-7.el7 will be installed\n---> Package python-docker-py.noarc 
h 0:1.10.6-4.el7 will be installed\n---> Package runc.x86_64 0:1.0.0-27.rc5.dev.git4bb1fe4.el7 will be installed\n--> Processing Dependency: criu for package: runc-1.0.0-27.rc5.dev 
.git4bb1fe4.el7.x86_64\n---> Package skopeo.x86_64 1:0.1.31-1.dev.gitae64ff7.el7.centos will be installed\n--> Running transaction check\n---> Package criu.x86_64 0:3.5-4.el7 will 
be installed\n--> Processing Dependency: libprotobuf-c.so.1(LIBPROTOBUF_C_1.0.0)(64bit) for package: criu-3.5-4.el7.x86_64\n--> Processing Dependency: libprotobuf-c.so.1()(64bit) f 
or package: criu-3.5-4.el7.x86_64\n--> Processing Dependency: libnet.so.1()(64bit) for package: criu-3.5-4.el7.x86_64\n---> Package libarchive.x86_64 0:3.1.2-10.el7_2 will be insta 
lled\n--> Running transaction check\n---> Package libnet.x86_64 0:1.1.6-7.el7 will be installed\n---> Package protobuf-c.x86_64 0:1.0.2-3.el7 will be installed\nRemoving python-doc 
ker-py.noarch 0:1.10.6-4.el7 - u due to obsoletes from installed python2-docker-3.3.0-1.el7.noarch\n--> Restarting Dependency Resolution with new changes.\n--> Running transaction 
check\n---> Package python-docker-py.noarch 0:1.10.6-4.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7. 
x86_64\n--> Running transaction check\n---> Package python-docker-py.noarch 0:1.10.6-3.el7 will be installed\nRemoving python-docker-py.noarch 0:1.10.6-3.el7 - u due to obsoletes f 
rom installed python2-docker-3.3.0-1.el7.noarch\n--> Restarting Dependency Resolution with new changes.\n--> Running transaction check\n---> Package python-docker-py.noarch 0:1.10. 
6-3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Finished Dependency Resolution\n You co 
uld try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"
   ]
}

imranrazakhan on 16 Aug 2018

@imranrazakhan a quick fix for that issue is to remove - atomic from the default_r_openshift_node_image_prep_packages variable in roles/openshift_node/defaults/main.yml

joshainglis on 16 Aug 2018

@imranrazakhan sorry for the issue you got, we are basically back to square one after we rollback the changes i made.

If you don't mind asking, any chance you can post here or in a gist the cat out of all the files you have in your /etc/yum.repos.d/ dir?

DanyC97 on 16 Aug 2018

List of files under /etc/yum.repos.d/

 epel-testing.repo
 epel.repo
 CentOS-Vault.repo
 CentOS-Sources.repo
 CentOS-Media.repo
 CentOS-fasttrack.repo
 CentOS-Debuginfo.repo
 CentOS-CR.repo
 CentOS-Base.repo
 CentOS-OpenShift-Origin.repo

Following is output of all files

[vagrant@c1-ocp yum.repos.d]$ cat *
# CentOS-Base.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#

[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates
[updates]
name=CentOS-$releasever - Updates
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

# CentOS-CR.repo
#
# The Continuous Release ( CR )  repository contains rpms that are due in the next
# release for a specific CentOS Version ( eg. next release in CentOS-7 ); these rpms
# are far less tested, with no integration checking or update path testing having
# taken place. They are still built from the upstream sources, but might not map
# to an exact upstream distro release.
#
# These packages are made available soon after they are built, for people willing
# to test their environments, provide feedback on content for the next release, and
# for people looking for early-access to next release content.
#
# The CR repo is shipped in a disabled state by default; its important that users
# understand the implications of turning this on.
#
# NOTE: We do not use a mirrorlist for the CR repos, to ensure content is available
#       to everyone as soon as possible, and not need to wait for the external
#       mirror network to seed first. However, many local mirrors will carry CR repos
#       and if desired you can use one of these local mirrors by editing the baseurl
#       line in the repo config below.
#

[cr]
name=CentOS-$releasever - cr
baseurl=http://mirror.centos.org/centos/$releasever/cr/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0
# CentOS-Debug.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#

# All debug packages from all the various CentOS-7 releases
# are merged into a single repo, split by BaseArch
#
# Note: packages in the debuginfo repo are currently not signed
#

[base-debuginfo]
name=CentOS-7 - Debuginfo
baseurl=http://debuginfo.centos.org/7/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-Debug-7
enabled=0
#
#CentOS-fasttrack.repo

[fasttrack]
name=CentOS-7 - fasttrack
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=fasttrack&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/fasttrack/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

# CentOS-Media.repo
#
#  This repo can be used with mounted DVD media, verify the mount point for
#  CentOS-7.  You can use this repo and yum to install items directly off the
#  DVD ISO that we release.
#
# To use this repo, put in your DVD and use it with the other repos too:
#  yum --enablerepo=c7-media [command]
#
# or for ONLY the media repo, do this:
#
#  yum --disablerepo=\* --enablerepo=c7-media [command]

[c7-media]
name=CentOS-$releasever - Media
baseurl=file:///media/CentOS/
        file:///media/cdrom/
        file:///media/cdrecorder/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[centos-openshift-origin]
name=CentOS OpenShift Origin
baseurl=http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-PaaS

[centos-openshift-origin-testing]
name=CentOS OpenShift Origin Testing
baseurl=http://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin/
enabled=0
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS

[centos-openshift-origin-debuginfo]
name=CentOS OpenShift Origin DebugInfo
baseurl=http://debuginfo.centos.org/centos/7/paas/x86_64/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS

[centos-openshift-origin-source]
name=CentOS OpenShift Origin Source
baseurl=http://vault.centos.org/centos/7/paas/Source/openshift-origin/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/openshift-ansible-CentOS-SIG-PaaS
# CentOS-Sources.repo
#
# The mirror system uses the connecting IP address of the client and the
# update status of each mirror to pick mirrors that are updated to and
# geographically close to the client.  You should use this for CentOS updates
# unless you are manually picking other mirrors.
#
# If the mirrorlist= does not work for you, as a fall back you can try the
# remarked out baseurl= line instead.
#
#

[base-source]
name=CentOS-$releasever - Base Sources
baseurl=http://vault.centos.org/centos/$releasever/os/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates
[updates-source]
name=CentOS-$releasever - Updates Sources
baseurl=http://vault.centos.org/centos/$releasever/updates/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras-source]
name=CentOS-$releasever - Extras Sources
baseurl=http://vault.centos.org/centos/$releasever/extras/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that extend functionality of existing packages
[centosplus-source]
name=CentOS-$releasever - Plus Sources
baseurl=http://vault.centos.org/centos/$releasever/centosplus/Source/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
# CentOS Vault contains rpms from older releases in the CentOS-7
# tree.

#c7.0.1406
[C7.0.1406-base]
name=CentOS-7.0.1406 - Base
baseurl=http://vault.centos.org/7.0.1406/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.0.1406-updates]
name=CentOS-7.0.1406 - Updates
baseurl=http://vault.centos.org/7.0.1406/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.0.1406-extras]
name=CentOS-7.0.1406 - Extras
baseurl=http://vault.centos.org/7.0.1406/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.0.1406-centosplus]
name=CentOS-7.0.1406 - CentOSPlus
baseurl=http://vault.centos.org/7.0.1406/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.0.1406-fasttrack]
name=CentOS-7.0.1406 - CentOSPlus
baseurl=http://vault.centos.org/7.0.1406/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

# C7.1.1503
[C7.1.1503-base]
name=CentOS-7.1.1503 - Base
baseurl=http://vault.centos.org/7.1.1503/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.1.1503-updates]
name=CentOS-7.1.1503 - Updates
baseurl=http://vault.centos.org/7.1.1503/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.1.1503-extras]
name=CentOS-7.1.1503 - Extras
baseurl=http://vault.centos.org/7.1.1503/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.1.1503-centosplus]
name=CentOS-7.1.1503 - CentOSPlus
baseurl=http://vault.centos.org/7.1.1503/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.1.1503-fasttrack]
name=CentOS-7.1.1503 - CentOSPlus
baseurl=http://vault.centos.org/7.1.1503/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

# C7.2.1511
[C7.2.1511-base]
name=CentOS-7.2.1511 - Base
baseurl=http://vault.centos.org/7.2.1511/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.2.1511-updates]
name=CentOS-7.2.1511 - Updates
baseurl=http://vault.centos.org/7.2.1511/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.2.1511-extras]
name=CentOS-7.2.1511 - Extras
baseurl=http://vault.centos.org/7.2.1511/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.2.1511-centosplus]
name=CentOS-7.2.1511 - CentOSPlus
baseurl=http://vault.centos.org/7.2.1511/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.2.1511-fasttrack]
name=CentOS-7.2.1511 - CentOSPlus
baseurl=http://vault.centos.org/7.2.1511/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

# C7.3.1611
[C7.3.1611-base]
name=CentOS-7.3.1611 - Base
baseurl=http://vault.centos.org/7.3.1611/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.3.1611-updates]
name=CentOS-7.3.1611 - Updates
baseurl=http://vault.centos.org/7.3.1611/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.3.1611-extras]
name=CentOS-7.3.1611 - Extras
baseurl=http://vault.centos.org/7.3.1611/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.3.1611-centosplus]
name=CentOS-7.3.1611 - CentOSPlus
baseurl=http://vault.centos.org/7.3.1611/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.3.1611-fasttrack]
name=CentOS-7.3.1611 - CentOSPlus
baseurl=http://vault.centos.org/7.3.1611/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

# C7.4.1708
[C7.4.1708-base]
name=CentOS-7.4.1708 - Base
baseurl=http://vault.centos.org/7.4.1708/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.4.1708-updates]
name=CentOS-7.4.1708 - Updates
baseurl=http://vault.centos.org/7.4.1708/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.4.1708-extras]
name=CentOS-7.4.1708 - Extras
baseurl=http://vault.centos.org/7.4.1708/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.4.1708-centosplus]
name=CentOS-7.4.1708 - CentOSPlus
baseurl=http://vault.centos.org/7.4.1708/centosplus/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0

[C7.4.1708-fasttrack]
name=CentOS-7.4.1708 - CentOSPlus
baseurl=http://vault.centos.org/7.4.1708/fasttrack/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
enabled=0
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch/debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/7/SRPMS
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1
[epel-testing]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

[epel-testing-debuginfo]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/$basearch/debug
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-debug-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=1

[epel-testing-source]
name=Extra Packages for Enterprise Linux 7 - Testing - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/testing/7/SRPMS
metalink=https://mirrors.fedoraproject.org/metalink?repo=testing-source-epel7&arch=$basearch
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

imranrazakhan on 16 Aug 2018

👍1

@imranrazakhan in your output i can see

Package python-docker-py.noarch 0:1.10.6-4.el7 will be installed
--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64
--> Running transaction check
---> Package python-docker-py.noarch 0:1.10.6-3.el7 will be installed\nRemoving python-docker-py.noarch 0:1.10.6-3.el7 - u due to obsoletes from installed python2-docker-3.3.0-1.el7.noarch
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package python-docker-py.noarch 0:1.10.6-3.el7 will be installed
--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64
--> Finished Dependency Resolution
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest"

Now i just done myself a quick test to see the dependecies for atomic pkg and so see http://pastebin.centos.org/1504251/

As such suggest you start from a clean env, or uninstall the the previous pkg and try again.

@isaldarriaga @Reamer you guys able to help and try a fresh deployment on a clean env and let me know the result ?

DanyC97 on 16 Aug 2018

@imranrazakhan check if you have python2-docker installed, then remove it.

70 | remove python2-docker    | 2018-08-14 16:31 | Erase          |    1

Atomic has dependence python-docker-py >= 1.7.2-1, in my env it was blocked by this package.

alv91 on 16 Aug 2018

@alv91 indeed atomic depends on python-docker-py however is not pinned to a specific version. As shown in my previous comment, it pulls down the latest python-docker-py version if there is nothing present.

DanyC97 on 16 Aug 2018

@DanyC97 Yes, but i had to remove this package to allow reinstallation atomic package. I cant find also any installation package like python2-docker in openshift-ansible repository.

# rpm -i python2-docker-3.3.0-1.el7.noarch.rpm 
error: Failed dependencies:
    python-docker-py < 2 is obsoleted by python2-docker-3.3.0-1.el7.noarch

alv91 on 16 Aug 2018

@alv91 correct because we had to remove the package yesterday evening. And so the expectation is that everyone who already have a non pristine deployment or an env where the python2-docker-3.3.0-1.el7.noarch.rpm & python-docker-py >= 1.7.2-1 are present to _uninstall it_

DanyC97 on 16 Aug 2018

👍2

@DanyC97

fresh install, this time:

manually copied /etc/yum.repos.d/CentOS-OpenShift-Origin310.repo to all hosts (looks like the MERGE code of PR9548 is not doing this correctly. could you please verify this?)
not using sdn but flannel (errors during deploy prevent etcd, infra, and worker nodes to start/join openshift cluster with empty /etc/cni/net.d/)
not using firewalld but iptables (firewall rule application errors detected possibly causing sdn not to start)
removed firewalld service from hosts (service in failed stated)
disabled firewall and rules at the virtualization level (just in case)
made sure the hosts have the following packages pre-installed: wget git net-tools bind-utils yum-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct
not using dedicated etcd hosts ('cause noticed masters are forced to include etcd no matter they are not part of [etcd], so had x6 etcd during install. only 3 needed)
synchronized server clocks
made sure no system services are failing (BTW, kdump / kexec-tools requires 2GB RAM as minimum for this service to be up)
increased all retries: n and delay: m to 100 so i have room to restart NetworkManager and dnsmasq services during installation (master to master connectivity is lost during httpd-tools rpm setup)
testing both 3.10 branch and 3.10.28-1 tag

test results:

$ BRANCH="3.10" && \
mkdir --parents "openshift-ansible-v$BRANCH" && \
git clone https://github.com/openshift/openshift-ansible "openshift-ansible-v$BRANCH" && \
cd "openshift-ansible-v$BRANCH" && \
git checkout release-$BRANCH && \
echo && echo "Current version is:" && \
git describe && \
cd ..

Cloning into 'openshift-ansible-v3.10'...
remote: Counting objects: 119843, done.
remote: Total 119843 (delta 0), reused 0 (delta 0), pack-reused 119842
Receiving objects: 100% (119843/119843), 31.00 MiB | 1.84 MiB/s, done.
Resolving deltas: 100% (75124/75124), done.
Branch 'release-3.10' set up to track remote branch 'release-3.10' from 'origin'.
Switched to a new branch 'release-3.10'

Current version is:
openshift-ansible-3.10.29-1-2-g8e9cd4a1b

ansible-playbook -vvv -i /path/to/inventory.ini /path/to/playbooks/prerequisites.yml

TASK [openshift_node : install needed rpm(s)] 
task path: /home/ivan/convergence.cloud/dev-ops/kube-prod/origin/openshift-ansible-v3.10/roles/openshift_node/tasks/install_rpms.yml:2
...
FAILED - RETRYING: install needed rpm(s) (1 retries left).Result was:
...
"item": "atomic", 
    "msg": "Error: Package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64 (extras)\n           Requires: python-docker-py >= 1.7.2-1\n           Available: docker-python-1.4.0-115.el7.x86_64 (extras)\n               python-docker-py = 1.4.0-115.el7\n           Available: python-docker-py-1.10.6-3.el7.noarch (extras)\n               python-docker-py = 1.10.6-3.el7\n           Available: python-docker-py-1.10.6-4.el7.noarch (extras)\n               python-docker-py = 1.10.6-4.el7\n"

I'll checkout 3.10.28-1 tag and post results later.

isaldarriaga on 16 Aug 2018

getting tag code:

```BRANCH="3.10" && TAG="3.10.28-1" && \
mkdir --parents "openshift-ansible-v$TAG" && \
git clone https://github.com/openshift/openshift-ansible "openshift-ansible-v$TAG" && \
cd "openshift-ansible-v$TAG" && \
git fetch --all --tags --prune && \
git checkout tags/openshift-ansible-$TAG -b release-$BRANCH
echo && echo "Current version is:" && \
git describe --tags && \
cd ..

`ansible-playbook -vvv -i /path/to/inventory.ini /path/to/playbooks/prerequisites.yml`

exact same output on a fresh install:

{
"attempts": 3,
"changed": false,
"invocation": {
"module_args": {
"allow_downgrade": false,
"bugfix": false,
"conf_file": null,
"disable_gpg_check": false,
"disable_plugin": [],
"disablerepo": null,
"enable_plugin": [],
"enablerepo": null,
"exclude": null,
"install_repoquery": true,
"installroot": "/",
"list": null,
"name": [
"atomic"
],
"security": false,
"skip_broken": false,
"state": "present",
"update_cache": false,
"update_only": false,
"validate_certs": true
}
},
"item": "atomic",
"msg": "Error: Package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64 (extras)\n Requires: python-docker-py >= 1.7.2-1\n Available: docker-python-1.4.0-115.el7.x86_64 (extras)\n python-docker-py = 1.4.0-115.el7\n Available: python-docker-py-1.10.6-3.el7.noarch (extras)\n python-docker-py = 1.10.6-3.el7\n Available: python-docker-py-1.10.6-4.el7.noarch (extras)\n python-docker-py = 1.10.6-4.el7\n",
"rc": 1,
"results": [
"Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: repos.lax.quadranet.com\n * extras: mirrors.xmission.com\n * updates: centos.sonn.com\nResolving Dependencies\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n--> Processing Dependency: runc for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Processing Dependency: python-dateutil for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Processing Dependency: gomtree for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n---> Package gomtree.x86_64 0:0.5.0-0.2.git16da0f8.el7 will be installed\n---> Package python-dateutil.noarch 0:1.5-7.el7 will be installed\n---> Package runc.x86_64 0:1.0.0-27.rc5.dev.git4bb1fe4.el7 will be installed\n--> Processing Dependency: criu for package: runc-1.0.0-27.rc5.dev.git4bb1fe4.el7.x86_64\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n---> Package criu.x86_64 0:3.5-4.el7 will be installed\n--> Processing Dependency: libprotobuf-c.so.1(LIBPROTOBUF_C_1.0.0)(64bit) for package: criu-3.5-4.el7.x86_64\n--> Processing Dependency: libprotobuf-c.so.1()(64bit) for package: criu-3.5-4.el7.x86_64\n--> Processing Dependency: libnet.so.1()(64bit) for package: criu-3.5-4.el7.x86_64\n--> Running transaction check\n---> Package atomic.x86_64 1:1.22.1-22.git5a342e3.el7 will be installed\n--> Processing Dependency: python-docker-py >= 1.7.2-1 for package: 1:atomic-1.22.1-22.git5a342e3.el7.x86_64\nPackage python-docker-py-1.10.6-4.el7.noarch is obsoleted by python2-docker-3.3.0-1.el7.noarch which is already installed\n---> Package libnet.x86_64 0:1.1.6-7.el7 will be installed\n---> Package protobuf-c.x86_64 0:1.0.2-3.el7 will be installed\n--> Finished Dependency Resolution\n You could try using --skip-broken to work around the problem\n You could try running: rpm -Va --nofiles --nodigest\n"
]
}

```

isaldarriaga on 16 Aug 2018

@DanyC97 Can we tag this into our repos? This is the same version that's shipped in OCP and it fulfills all of the package requirements of atomic.
http://cbs.centos.org/koji/buildinfo?buildID=17580

sdodson on 16 Aug 2018

@isaldarriaga If you install the package from http://cbs.centos.org/koji/buildinfo?buildID=17580 are all the dependencies of atomic fulfilled?

sdodson on 16 Aug 2018

@sdodson I haven't tried that option yet but could in a next run.

i already did what @joshainglis suggested, to comment/remove "atomic" from default_r_openshift_node_image_prep_packages.

so far so good, this approach let the prerequisites continue to further steps.

I'll keep you posted. waiting for results yet.

isaldarriaga on 16 Aug 2018

👍1

@sdodson i've tagged the rpm into

candidate repo => rpm present
testing repo => rpm present

Before tagging into -release i'll wait for @isaldarriaga to confirm if it worked or not as i don't want to go again through the process of removing rpms.

@isaldarriaga regarding

manually copied /etc/yum.repos.d/CentOS-OpenShift-Origin310.repo to all hosts (looks like the MERGE code of PR9548 is not doing this correctly. could you please verify this?)

can you share more info why you think is not doing the right thing ?

DanyC97 on 17 Aug 2018

@sdodson
just getting a couple of non related errors to this issue.

Hosts: prod-lb-01.example.com
Play: Initialize cluster facts
Task: Gather Cluster facts
Message: MODULE FAILURE
1. Hosts: prod-master-01.example.com
  Play: Create Hosted Resources - router
  Task: Get the certificate contents for router
  Message: |lib_utils_oo_select_keys failed expects to filter on a dict or object

I'll work in a solution, retry and let you know.

@DanyC97

can you share more info why you think is not doing the right thing ?

sure, no matter i've set openshift_release=v3.10 and openshift_repos_enable_testing=true in the inventory, the /etc/yum.repos.d/CentOS-OpenShift-Origin310.repo file does not appear in the server during prerequisites nor cluster_deploy steps, thence there's no valid source to pull the rpms from. This happen in the current v3.10 branch ($ git describe => openshift-ansible-3.10.29-1-2-g8e9cd4a1b) which already has the PR changes merged.

I was able to successfully test this part before, i think in 3.10-27-1, by locally patching roles/openshift_repos/tasks/centos_repos.yml and adding the roles/openshift_repos/templates/CentOS-OpenShift-Origin310.repo.j2 file. With the patch, ansible did copy the CentOS-OpenShift-Origin310.repo to the servers. see links with its respective content.

Maybe here there's a clue: I noticed centos_repos.yml is no longer part of the PR nor the merge, which initially included this change: CentOS-OpenShift-Origin{{ ((openshift_release | default('')).split('.') | join(''))[0:3] }}.repo.j2, but even if i patch the current branch with that sentence i cannot get the CentOS-OpenShift-Origin310.repo generated in the servers either. so that's the reason i decided to manually copy the CentOS-OpenShift-Origin310.repo file to the servers before the test.

hope this help to clarify.

isaldarriaga on 17 Aug 2018

👍1

@joshainglis and @isaldarriaga by remove "atomic" from default_r_openshift_node_image_prep_packages. I bypassed previous error but now it fail on below

TASK [openshift_web_console : Verify that the console is running] ***************************************************************************************************
FAILED - RETRYING: Verify that the console is running (60 retries left) ...

{  
   "attempts":60,
   "changed":false,
   "results":{  
      "cmd":"/usr/bin/oc get deployment webconsole -o json -n openshift-web-console",
      "results":[  
         {  
            "apiVers 
ion":"extensions/v1beta1",
            "kind":"Deployment",
            "metadata":{  
               "annotations":{  
                  "deployment.kubernetes.io/revision":"1",
                  "kubectl.kubernetes.io/last-applied-configuration":"{\"api 
Version\":\"apps/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"openshift-web-console\",\"webconsole\":\"true\"},\"name\":\"webconsole\",\ 
"                  namespace\":\"openshift-web-console\"
               },
               \"spec\":{  
                  \"replicas\":1,
                  \"strategy\":{  
                     \"rollingUpdate\":{  
                        \"maxUnavailable\":\"100%\"
                     },
                     \"type\":\"RollingUpdate\"
                  },
                  \"template\":{  
                     \"metadata\ 
":{  
                        \"labels\":{  
                           \"app\":\"openshift-web-console\",
                           \"webconsole\":\"true\"
                        },
                        \"name\":\"webconsole\"
                     },
                     \"spec\":{  
                        \"containers\":[  
                           {  
                              \"command\":[  
                                 \"/usr/bin/origin-web-console\",
                                 \"--audit 
-log-path=-\",
                                 \"-v=0\",
                                 \"--config=/var/webconsole-config/webconsole-config.yaml\"
                              ],
                              \"image\":                              \"docker.io/openshift/origin-web-console:v3.10\",
                              \"imagePullPolicy\":\"IfNotPresent\",
                              \ 
"livenessProbe\":{  
                                 \"exec\":{  
                                    \"command\":[  
                                       \"/bin/sh\",
                                       \"-c\",
                                       \"if                                       [  
                                          [  
                                             ! -f /tmp/webconsole-config.hash
                                          ]
                                       ]                                       ; then \\\\\\n  md5sum /var/webconsole-config/webconsole-config.yaml \\u003e 
/tmp/webconsole-config.hash; \\\\\\nelif                                       [  
                                          [  
                                             $(md5sum /var/webconsole-config/webconsole-config.yaml) != $(cat /tmp/webconsole-config.hash)
                                          ]
                                       ]                                       ; then \\\\\\n  echo 'webconsole-config. 
yaml has changed.'; \\\\\\n  exit 1; \\\\\\nfi \\u0026\\u0026 curl -k -f https:                                       //0.0.0.0:8443                                       /console/\"
                                    ]
                                 }
                              },
                              \"name\":\"webconsole\",
                              \"ports\":[  
                                 {  
                                    \"containerPort\":8443
                                 }
                              ],
                              \"readiness 
Probe\":{  
                                 \"httpGet\":{  
                                    \"path\":\"/healthz\",
                                    \"port\":8443,
                                    \"scheme\":\"HTTPS\"
                                 }
                              },
                              \"resources\":{  
                                 \"requests\":{  
                                    \"cpu\":\"100m\",
                                    \"memory\":\"100Mi\"
                                 }
                              },
                              \"volumeMounts\":[  
                                 {  
                                    \"mountPath 
\":\"/var/serving-cert\",
                                    \"name\":\"serving-cert\"
                                 },
                                 {  
                                    \"mountPath\":\"/var/webconsole-config\",
                                    \"name\":\"webconsole-config\"
                                 }
                              ]
                           }
                        ],
                        \"nodeSelector\":{  
                           \"node-role.kubernetes.io/master\ 
":\"true\"
                        },
                        \"serviceAccountName\":\"webconsole\",
                        \"volumes\":[  
                           {  
                              \"name\":\"serving-cert\",
                              \"secret\":{  
                                 \"defaultMode\":288,
                                 \"secretName\":\"webconsole-serving-cert\"
                              }
                           },
                           {  
                              \"configMap\ 
":{  
                                 \"defaultMode\":288,
                                 \"name\":\"webconsole-config\"
                              },
                              \"name\":\"webconsole-config\"
                           }
                        ]
                     }
                  }
               }
            }            \n"
         },
         "creationTimestamp":"2018-08-17T09:31:34Z",
         "generation":1,
         "labels":{  
            "app":"op 
enshift-web-console",
            "webconsole":"true"
         },
         "name":"webconsole",
         "namespace":"openshift-web-console",
         "resourceVersion":"2719",
         "selfLink":"/apis/extensions/v1beta1/namespaces 
/openshift-web-console/deployments/webconsole",
         "uid":"54c0c507-a200-11e8-bb03-0800279b8177"
      },
      "spec":{  
         "progressDeadlineSeconds":600,
         "replicas":1,
         "revisionHistoryLimit":2,
         " 
selector":{  
            "matchLabels":{  
               "app":"openshift-web-console",
               "webconsole":"true"
            }
         },
         "strategy":{  
            "rollingUpdate":{  
               "maxSurge":"25%",
               "maxUnavailable":"100%"
            },
            "type":"RollingUpd 
ate"
         },
         "template":{  
            "metadata":{  
               "creationTimestamp":null,
               "labels":{  
                  "app":"openshift-web-console",
                  "webconsole":"true"
               },
               "name":"webconsole"
            },
            "spec":{  
               "containers":[  
                  {  
                     "comma 
nd":[  
                        "/usr/bin/origin-web-console",
                        "--audit-log-path=-",
                        "-v=0",
                        "--config=/var/webconsole-config/webconsole-config.yaml"
                     ],
                     "image":"docker.io/openshift/origin-web-console:v3.10 
",
                     "imagePullPolicy":"IfNotPresent",
                     "livenessProbe":{  
                        "exec":{  
                           "command":[  
                              "/bin/sh",
                              "-c",
                              "if [[ ! -f /tmp/webconsole-config.hash ]]; then \\\n  md5sum /var/webconsole-config/w 
ebconsole-config.yaml > /tmp/webconsole-config.hash; \\\nelif [[ $(md5sum /var/webconsole-config/webconsole-config.yaml) != $(cat /tmp/webconsole-config.hash) ]]; then \\\n  echo ' 
webconsole-config.yaml has changed.'; \\\n  exit 1; \\\nfi && curl -k -f https://0.0.0.0:8443/console/"
                           ]
                        },
                        "failureThreshold":3,
                        "periodSeconds":10,
                        "successThreshold":1,
                        "timeo 
utSeconds":1
                     },
                     "name":"webconsole",
                     "ports":[  
                        {  
                           "containerPort":8443,
                           "protocol":"TCP"
                        }
                     ],
                     "readinessProbe":{  
                        "failureThreshold":3,
                        "httpGet":{  
                           "path":"/healthz",
                           "port":8443,
                           "scheme":"HTTPS"
                        },
                        "periodSeconds":10,
                        "successThreshold":1,
                        "timeoutSeconds":1
                     },
                     "resources":{  
                        "requests":{  
                           "cpu":"100m",
                           "memory":"100Mi"
                        }
                     },
                     "terminationMessagePath":"/de 
v/termination-log",
                     "terminationMessagePolicy":"File",
                     "volumeMounts":[  
                        {  
                           "mountPath":"/var/serving-cert",
                           "name":"serving-cert"
                        },
                        {  
                           "mountPath":"/var/webconsole-config",
                           "name":"webconsole-config"
                        }
                     ]
                  }
               ],
               "dnsPolicy":"ClusterFirst",
               "nodeSelector":{  
                  "node-role.kubernetes.io/master":"true"
               },
               "restartPolicy":"Always",
               "schedulerName":"default-scheduler",
               "securityContext":{  

               },
               "serviceAccount":"webconsole",
               "serviceAccountName":"webconsole",
               "terminationGracePeriodSeconds":30,
               "volumes":[  
                  {  
                     "name":"serving-cert",
                     "secret":{  
                        "def 
aultMode":288,
                        "secretName":"webconsole-serving-cert"
                     }
                  },
                  {  
                     "configMap":{  
                        "defaultMode":288,
                        "name":"webconsole-config"
                     },
                     "name":"webconsole-config"
                  }
               ]
            }
         }
      },
      "status":{  
         "conditions 
":[  
            {  
               "lastTransitionTime":"2018-08-17T09:31:34Z",
               "lastUpdateTime":"2018-08-17T09:31:34Z",
               "message":"Deployment has minimum availability.",
               "reason":"MinimumReplicasAvailable",
               "status":"True",
               "type":"Available"
            },
            {  
               "lastTransitionTime":"2018-08-17T09:41:35Z",
               "lastUpdateTime":"2018-08-17T09:41:35Z",
               "message":"ReplicaSet \"webconsole-57d88df7d9\" 
has timed out progressing.",
               "reason":"ProgressDeadlineExceeded",
               "status":"False",
               "type":"Progressing"
            }
         ],
         "observedGeneration":1,
         "replicas":1,
         "unavailableReplicas":1,
         "up 
datedReplicas":1
      }
   }
],
"returncode":0
},
"state":"list"
}
...ignoring



md5-144ae1a0cc5c3b01acda4dd1831a23e7



TASK [openshift_web_console : debug] ********************************************************************************************************************************
ok: [10.0.2.15] => {
    "msg": [
        "LAST SEEN   FIRST SEEN   COUNT     NAME                                           KIND         SUBOBJECT   TYPE      REASON              SOURCE                  MESSAGE",

        "17s         10m          37        webconsole-57d88df7d9-rwmv2.154ba211bdb121b5   Pod                      Warning   FailedScheduling    default-scheduler       0/1 nodes
are available: 1 node(s) didn't match node selector.",
        "10m         10m          1         webconsole-57d88df7d9.154ba2117fd5caba         ReplicaSet               Warning   FailedCreate        replicaset-controller   Error crea
ting: pods \"webconsole-57d88df7d9-\" is forbidden: error looking up service account openshift-web-console/webconsole: serviceaccount \"webconsole\" not found",
        "10m         10m          1         webconsole-57d88df7d9.154ba211bdb9791e         ReplicaSet               Normal    SuccessfulCreate    replicaset-controller   Created po
d: webconsole-57d88df7d9-rwmv2",
        "10m         10m          1         webconsole.154ba2117e3ac9fc                    Deployment               Normal    ScalingReplicaSet   deployment-controller   Scaled up
replica set webconsole-57d88df7d9 to 1"
    ]
}



md5-dcc63a77669f4c4faad6773400d9a703



NAME                            READY     STATUS    RESTARTS   AGE
pod/docker-registry-1-deploy    0/1       Pending   0          15m
pod/registry-console-1-deploy   0/1       Pending   0          15m
pod/router-1-deploy             0/1       Pending   0          16m

NAME                                       DESIRED   CURRENT   READY     AGE
replicationcontroller/docker-registry-1    0         0         0         15m
replicationcontroller/registry-console-1   0         0         0         15m
replicationcontroller/router-1             0         0         0         16m

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
service/docker-registry    ClusterIP   172.30.136.135   <none>        5000/TCP                  16m
service/kubernetes         ClusterIP   172.30.0.1       <none>        443/TCP,53/UDP,53/TCP     19m
service/registry-console   ClusterIP   172.30.207.77    <none>        9000/TCP                  15m
service/router             ClusterIP   172.30.150.187   <none>        80/TCP,443/TCP,1936/TCP   16m

NAME                                                  REVISION   DESIRED   CURRENT   TRIGGERED BY
deploymentconfig.apps.openshift.io/docker-registry    1          1         0         config
deploymentconfig.apps.openshift.io/registry-console   1          1         0         config
deploymentconfig.apps.openshift.io/router             1          1         0         config

NAME                                        HOST/PORT                                        PATH      SERVICES           PORT      TERMINATION   WILDCARD
route.route.openshift.io/docker-registry    docker-registry-default.apps.10.0.2.15.nip.io              docker-registry    <all>     passthrough   None
route.route.openshift.io/registry-console   registry-console-default.apps.10.0.2.15.nip.io             registry-console   <all>     passthrough   None

imranrazakhan on 17 Aug 2018

@isaldarriaga if you don't mind, any chance you can try a fresh deployment using the testing repo while leave the atomic rpm in and let me know if you happy or not ?

reason for that is because i'd like then to push the change to -release and try to close this issue which is getting closer to 100 comments.

I hope we can put to rest the rpms issue and then focus on the other issue you just mentioned.

update

also can you please copy your inventory file @isaldarriaga to http://pastebin.centos.org/ or any other alternative?

DanyC97 on 17 Aug 2018

I don't understand why installing any RPM packages on the masters/nodes when containerized=true

Containerized install has been deprecated in 3.10. It won't do anything now.

Initial reporter has openshift_image_tag=v3.9.0 which would certainly break many things

This issue is becoming a collection of 'I also have problem with repo' but it doesn't mean its all caused by the same problem in the code, so I'm going to close it

Please open new issue for particular problem - and don't forget to attach the inventory, exact branch and the output of playbook run with '-vvv'.

vrutkovs on 17 Aug 2018

@DanyC97 prerequisites were not running in my environment due an error in one of my scripts. deploy_cluster.yaml was being running instead (yesterday). i fixed that.

i ran fresh again

$ git describe
openshift-ansible-3.10.29-1-22-gdec10a6d6

i didn't remove atomic from default_r_openshift_node_image_prep_packages this time, nor patched roles/openshift_repos/tasks/centos_repos.yml

during prerequisites execution:

$ ls -al /etc/yum.repos.d/
total 48
drwxr-xr-x.  2 root root  223 Aug 17 04:23 .
drwxr-xr-x. 78 root root 8192 Aug 17 04:21 ..
-rw-r--r--.  1 root root 1664 Aug  2 03:56 CentOS-Base.repo
-rw-r--r--.  1 root root 1309 Aug  2 03:56 CentOS-CR.repo
-rw-r--r--.  1 root root  649 Aug  2 03:56 CentOS-Debuginfo.repo
-rw-r--r--.  1 root root  314 Aug  2 03:56 CentOS-fasttrack.repo
-rw-r--r--.  1 root root  630 Aug  2 03:56 CentOS-Media.repo
-rw-r--r--.  1 root root  887 Aug 17 04:23 CentOS-OpenShift-Origin.repo
-rw-r--r--.  1 root root 1331 Aug  2 03:56 CentOS-Sources.repo
-rw-r--r--.  1 root root 4768 Aug  2 03:56 CentOS-Vault.repo

$ cat /etc/yum.repos.d/CentOS-OpenShift-Origin.repo # master-01

output

prerequisites ran to completion without errors with this repo file. So the PR in mention was actually ok!

running deploy_cluster.yaml now.

will re-open if another rpm issue is found.

thanks!

isaldarriaga on 17 Aug 2018

🎉1

Openshift-ansible: Failed to install Openshift 3.10 because of missing packages

Description

Version

Steps To Reproduce

Expected Results

Observed Results

Additional Information

Most helpful comment

All 90 comments

Related issues