Openshift-ansible: Introduce concept of pre-flight sanity playbook

[jeremyeder]

The use-case is laid out in the trello card [1], but I'll copy it here:
The openstack performance and scale team has developed a set of
pre-flight sanity checks for RHEL OSP.
We could do the same for openshift (does something already exist?)

https://github.com/openstack/browbeat/tree/master/ansible/check

A pre-flight playbook could do things like: - verify nodes are reachable in the first place (sometimes in the real
world, not all nodes are equal. timeouts for ansible may have to be
increased).

• verify subscriptions or access to repos
• verify supported versions of dependent packages
• check against known bugs
• check against known configuration issues (oo-diagnostics ?)
• verify your planned inventory file is sane
• support an install "dry-run"?

[1] https://trello.com/c/3Zb2d9zl/98-3-cfme-integration-prerequisites-playbook

[timothysc]

verify your planned inventory file is sane

• check hosts are up and running
• walk-through or hand-lint inventory file settings (ip-addresses, collisions, etc.)
  ...

[detiber]

We definitely want to do some sanity checking for conflicting variable settings to avoid discovering those down the line.

Also I like the idea of doing some testing around detected host IP addresses and the service/pod networks to make sure that there isn't a conflict there. Obviously we can't get all use cases, but if we can catch the obvious ones it would be a huge win.

[jeremyeder]

And lint ansible.cfg against known limitations such as # of forks.

[detiber]

@jeremyeder validating ansible.cfg may be a difficult task. Some of the values are bubbled up in a way that they can be tested, but others like the value of forks, do not get used outside of the ansible-playbook command itself. Challenge will be to see if there is a way to get the value without diving too deep into internal-use only code that might be fragile with different ansible verisons going forward.

[jeremyeder]

There really should be a way for playbooks to introspect the ansible environment they're working in. Surprised that there isn't one :-( ... hopefully we can do something short term and perhaps file something with Ansible to provide that guaranteed interface?

[timothysc]

Gather state and dump before upgrade.

[tpoitras]

Hey, so I have a few more days left as a writer on the OpenShift documentation team before I leave the company, but this issue echoes an idea I've been chewing on. Initially I was thinking about a script that would be run pre-install, but this pre-flight sanity playbook is a similar idea. I'd hope something could be scripted for the quick install method as well.

There are a lotttt of things listed in the installation prerequisites in the OpenShift documentation. Too many, really. It would be great if all of those things could be incorporated into a pre-installation check, where the tasks are either automated or the user is prompted to enter Y/N or values to make it easy.

I reckon it'd require a great effort, but in my eyes, it would be ideal to include all of these:

• Check system requirements
• Lead user through DNS configuration
• Walk through network access setup and verifies port availability
• Prompts/checks/verifies git access
• Check SELinux is enabled on all servers
• Verify cloud provider config (if required)
• Verify the operating system on hosts
• Register hosts
• Manage packages
• Install docker and modify /etc/sysconfig/docker
• Configure Docker storage
• Ensure host access.

I know this issue is aimed at the Ansible playbook method, but would it be possible to include this kind of pre-install check for the quick installer as well?

Thank you to @adellape for pointing me to this GitHub issue when I mentioned the concept of a pre-install checker.

[detiber]

@tpoitras thanks for the feedback, it definitely helps!

[sosiouxme]

With https://github.com/openshift/openshift-ansible/pull/4079 we introduced playbooks/byo/openshift-checks/pre-install.yml which is basically the beginnings of these preflight checks. We've also added some of the same checks into the BYO install path (not sure what that implies for the quick installer). These abilities will be available in openshift-ansible RPMs and images once we ship 3.6.

Anybody who wants to add more checks is welcome to do so (see the existing checks). The items mentioned here should probably be ported and groomed on the Customer Success Trello board - but shall we consider this issue completed?

[jeremyeder]

How do we get the remaining items ported/groomed?