Origin: Centos PaaS RPM packages for openshift origin v3.9.0

[jperville]

Would it be possible to build RPM packages for the newly released Openshift Origin v3.9.0 and release them on the Centos PaaS repository at http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin39/ ?

Thank you in advance @tdawson

[tdawson]

Already working on it, but was expecting an issue for this.
Found a bug in out automation build scripts, same one that made a mess of 3.7.1. Getting it fixed properly this time instead of the workaround. If it takes too long to fix, we'll just do the work around again.

[tsikorski]

Given that there are no v3.9 Centos RPM's is there any other method of upgrading a v3.7 ansible built cluster to v3.9? Thanks.

[tdawson]

We plan on building and releasing 3.8.0 packages along with 3.9.0. This will allow people to do the upgrade from 3.7 to 3.9.
Remember 3.8.0 isn't supposed to be installed, it is just a transition to 3.9.

[leoluk]

3.9 openshift-ansible will also require this Ansible upgrade to reach stable, since the current 2.4 release is too old: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-7ef392255b

[phemmer]

So I see a RC build showed up yesterday (https://cbs.centos.org/koji/buildinfo?buildID=22539), but this build has the same problem the 3.7.1-1 release did, in that the binaries claim to be version 3.10.0

[tsikorski]

I took a look at the spec file for origin-3.9.0-1.el7.git.0.b1402ef.src.rpm and it has mixed versions specified:

%global os_git_vars OS_GIT_MINOR=10+ OS_GIT_MAJOR=3 OS_GIT_VERSION=v3.10.0-alpha.0+b1402ef-553 OS_GIT_TREE_STATE=clean KUBE_GIT_VERSION=v1.9.1+a0ce1bc657 OS_GIT_PATCH=0 KUBE_GIT_COMMIT=a0ce1bc OS_GIT_COMMIT=b1402ef ETCD_GIT_VERSION=v3.2.16 ETCD_GIT_COMMIT=121edf0

Also is not the v3.9 branch at git release 191fece?

[marziman]

@tdawson any updates on this?
Would be cool to use 3.9 on centos and update from 3.7..1 to 3.9.

Thanks !

[varesa]

3.9 openshift-ansible will also require this Ansible upgrade to reach stable, since the current 2.4 release is too old: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-7ef392255b

ansible-2.5.0-2.el7 has been pushed to stable

[korry8911]

When will Openshift Origin 3.9 be availble to install on Centos 7?

[arnikasky]

Hello guys! Wait for your updates!

[tdawson]

We have successful builds that passed our initial tests (correct code base and build tags). They are currently in testing. If we hear back that people have been able to use them successfully we will release them into the released CentOS repos on the mirrors.

origin 3.9 testing
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin39/
origin-3.9.0-1.el7.git.0.ba7faec
openshift-ansible-3.9.0-0.53.0.git.1.af49d87.el7

DO NOT USE ORIGIN 3.8.0, IT IS FOR UPGRADE PURPOSES ONLY
origin 3.8 testing
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin38/
origin-3.8.0-1.el7.git.0.dd1558c
openshift-ansible-3.8.37-1.git.1.151d57f.el7

[tdawson]

hint hint
If you are able to use these successfully. Please let us know.

[dmetzler]

Hello @tdawson , just ran an upgrade from 3.7, it seems to work:
I just encountered the TSB check problem referenced here: https://github.com/openshift/openshift-ansible/pull/7159 and had to force my etcd image to registry.fedoraproject.org/f27/etcd:latest as it seems the version upgrade algorithm is not the same when using RHEL registry...

Here is what I have as a result: https://www.dropbox.com/s/8tcyhfoq0ma1zdg/Screenshot%202018-04-12%2017.08.47.png?dl=0

Thx

[pgeorgiev333]

Probably a stupid question but how can i test this? I'm using the ansible playbooks. When i specify "openshift_repos_enable_testing=true" the packages are taken from http://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin/

If i manually edit my /etc/yum.repos.d/CentOS-OpenShift-Origin.repo it gets overwritten by the playbooks......

UPDATE:

It seems that i had to add the repo manually inside the playbook folder (roles/openshift_repos/templates). I created a new file called CentOS-OpenShift-Origin39.repo.j2 ans specified the test repo url above. So far it seems to work. I'l start fresh install to check if it is ok.

UPDATE 2:

The fresh install was ok up to the Prometheus playbook which failed because the metrics were not available. Apparently the cassandra pod keeps restarting...

Then i noticed that my logging deployment was not ok as the "logging-es-data-master" deployment was searching for an image that does not exit and i had to replace it with this one "openshift/oauth-proxy:v1.1.0". Possible workaround here: https://github.com/openshift/openshift-ansible/issues/6781

SO it looks ok.

[mshutt]

@dmetzler when we did 3.7 -> 3.7.2 I tripped over the same issue. I set openshift_etcd_upgrade to false and then re-ran byo/config.yml after we finished the upgrade. We're using a containerized haproxy for the master-api as well and the upgrades didn't run through the openshift_loadbalancer role as well as some of the other things related to openshift-metrics, openshift-infra, etc... So re-running byo/config.yml to bring everything up to the truth resolved this for us. I've not looked at the 3.7 -> (3.8) -> 3.9 upgrade plays yet, but we'll be trying then early next week hopefully and it's probably going to have a similar workflow?

[phemmer]

If you are able to use these successfully. Please let us know.

I've built a few clusters now with the new RPMs and they seem to work just fine.

[mlbiam]

After adding the repo, i had to install pyopenssl

[mlbiam]

Got past the pyopenssl issue, now getting:

TASK [etcd : Generate etcd backup] **********************************************************************************************************************************************************
fatal: [os.zzzzz.lan]: FAILED! => {"changed": false, "cmd": "runc exec etcd etcdctl backup --data-dir=/var/lib/etcd/ --backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180416143112", "msg": "[Errno 2] No such file or directory", "rc": 2}
    to retry, use: --limit @/root/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade.retry

UPDATE:
had to hardcode mail.yaml to think etcd was NOT running as a container. once i did that backups started working fine

[Scisci75]

Hi, I've tryed without change repository and the error was :

TASK [openshift_version : fail] *******************************************************************
fatal: [master-0.xxxx]: FAILED! => {"changed": false, "msg": "Package 'origin-3.8*' not found"}
to retry, use: --limit @/opt/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade.retry

[DanyC97]

@Scisci75 can you please open a new issue and provide full details including the inventory file etc ?

@jperville since the rpms are now available, can we close this issue ?

[Scisci75]

Hi, sorry but I think that the error was because in repository http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin/ there aren't the packages for version 3.8 that are needed to upgrade from version 3.7 to 3.9.

[tdawson]

This has been fixed (There are now 3.8.0 origin packages in that repo).
Since 3.9.0 has been released, I am closing this ticket.
Hopefully we have the update issue with 3.8.0 resolved, but if you do have other updating issues, please open a different issue.