Openshift-ansible: Version lock relevant packages to prevent issues when updating system

[elyscape]

Generally, updates to OpenShift require some commands to be run and are best performed with openshift-ansible. However, at least with OpenShift Origin, running yum update or yum upgrade will include any updated OpenShift packages, potentially causing issues.

To avoid the risk of a system update breaking an OpenShift deployment, it would be a good idea to install yum-plugin-versionlock and adding any OpenShift-related packages (i.e. origin* and tuned-profiles-origin-node) to the version lock list at the end of an openshift-ansible run. The packages would then be removed at the beginning of the run, if present.

It might also be a good idea to lock Docker packages as well.

[detiber]

@sdodson @abutcher @dgoodwin we've discussed this in the past and I've always been opposed to using versionlock, excludes, etc for managing these packages through our tooling for various reasons, including that it would only solve a subset of the cases that we have or introduce maintenance costs that could very well outweigh the benefits we'd gain from adding the additional complexity.

If we can gain sufficient support/buy in/desire for going down the path of managing version locking, exclude lists or other tooling to accomplish the same then I'm willing to re-evaluate it.

@elyscape just for some background, in the past we've taken paths towards trying to manage yum plugins and those attempts have generally ended with those efforts taking a ton of time and effort to manage, troubleshoot and generally being more costly to maintain and support than the original issues they were meant to solve.

There are also 2 use cases where this approach does not solve the underlying issue or hinders normal operation of the environments.

The first is for containerized installs (especially on an atomic host) where versionlock would not prevent changes (and we cannot control the docker version outside of recommending a user not update their system past a certain revision).

The second is for minor upgrades that do not require policy changes and can be done simply by updating the RPMs and restarting services (or rebooting the host) without the need for running the upgrade tooling/playbook.

[elyscape]

In that case, on a CentOS host, how do you recommend running system updates while preventing non-minor Origin upgrades from being applied? I understand that OSE has different channels for different major versions, but that isn't the case with OpenShift Origin, and yum upgrade can hose a cluster by upgrading from, say, 1.2.0 to 1.3.0, even if 1.2.1 is available and would be preferable.

[detiber]

@tdawson what are your thoughts here? I don't think we want to go down the path of maintaining multiple repos for the CentOS PaaS SIG.

[tdawson]

I feel the CentOS version of OpenShift Origin should be treated in the same way that the non-CentOS OpenShift Origin is treated. Which brings the next question. How is the non-CentOS OpenShift Origin treated.

This is a good question, and if we want real discussion, it should go to the CentOS mailling list and the PaaS SIG meeting.

My personal opinion. I am against setting version-lock's as the default in any package. It's a case of been-there done-that, got bitten really bad several times. Whether you do it or not, you need good documentation.

My personal opinion solution. Create a way for people to easily version-lock a version of openshift origin. An rpm would be able to do it. Something like origin-version-lock-1.2 and so forth.

[detiber]

I feel the CentOS version of OpenShift Origin should be treated in the same way that the non-CentOS OpenShift Origin is treated. Which brings the next question. How is the non-CentOS OpenShift Origin treated.

Agreed, unfortunately I don't think Fedora is quite as formalized for this currently, hopefully @maxamillion can chime in.

This is a good question, and if we want real discussion, it should go to the CentOS mailling list and the PaaS SIG meeting.

Agreed, as well as the Fedora equivalent.

My personal opinion. I am against setting version-lock's as the default in any package. It's a case of been-there done-that, got bitten really bad several times. Whether you do it or not, you need good documentation.

Indeed, documentation is going to be the biggest factor here, I think I would prefer at least starting with documenting the problem and recommending ways users can avoid the inadvertent upgrades by implementing version locking or excludes on their own. Of course I think whatever path we do take should be chosen by the community as a whole.

My personal opinion solution. Create a way for people to easily version-lock a version of openshift origin. An rpm would be able to do it. Something like origin-version-lock-1.2 and so forth.

Interesting, I never actually thought of that approach. I was thinking of just managing the version locking through Ansible. I think managing it through an RPM might just remove _some_ of the maintenance challenges that I was worried about.

[maxamillion]

I'm against the use of yum-plugin-versionlock because that plugin hasn't been ported to dnf yet (and might never be) so it's not something we can currently use in Fedora land and would likely just mean that this problem would need to be fixed again in the future for downstream distros.

I'd be happy to have a cross discussion with the CentOS PaaS SIG and the Fedora Cloud Working Group to try and come to a consensus across communities about what the most desirable approach would be. I'm open to discussing via irc meeting, mailing list, or otherwise... whatever works best. :)

[tdawson]

We had a discussion on this in the PaaS SIG.

We will create and rpm, that uses excludes instead of versionlock, to keep openshift origin from being updated. (and openshift enterprise as well, if this works) We will also document how to keep your openshift from being updated, as well as how to do the update if you have the rpm installed.

We determined this was the least invasive way of solving this problem.

[maxamillion]

Sounds like a reasonable plan, we'll need to find a good way to inform users of this so they don't think they are getting updates when they are actually not.

[sdodson]

Sounds like a reasonable plan, we'll need to find a good way to inform users of this so they don't think they are getting updates when they are actually not.

That's always been my biggest concern with these proposals. If we can assume that people will use the installer then the installer could handle it, perhaps even provide an equivalent to yum update-check ? Still it's a tool that's outside the normal admin toolchain.

[detiber]

Considering the easiness of accidentally updating across "major" versions for CentOS/Fedora, I think this is warranted in this case.

The tooling should definitely take this into account, and we most definitely need to call it out on the docs to make sure that it isn't a surprise for users.

[elyscape]

So the package would be something like origin-release-1.2 which would exclude versions not in the range 1.3.0 > package >=1.2.0, and then openshift-ansible would replace it with origin-release-1.3 during an upgrade?

[tdawson]

openshift-excluder has been created and documented
https://wiki.centos.org/SpecialInterestGroup/PaaS/OpenShift-Origin-Control-Updates

I believe this will fix the problem for CentOS users. Fedora users would need to rebuild the rpm on Fedora, so that it uses dnf.

[detiber]

@maxamillion ^ also, we are looking at adding this more formally into OCP, which will come with some openshift-ansible changes to manage the excluder as part of the upgrade process.

[elyscape]

Just for my understanding, why was yum exclusion chosen as the way forward rather than using a metapackage with versioned dependencies?

[sdodson]

Just for my understanding, why was yum exclusion chosen as the way forward rather than using a metapackage with versioned dependencies?

Versioned dependencies will yield yum dependency solving errors whenever you try to do something that would break those versioned dependencies and that generates a lot of support questions. Also the version of yum available in RHEL doesn't support ranges ie: docker => 1.0 and docker < 2.0

[maxamillion]

@tdawson @detiber I'll work on that for Fedora space, thanks!