Robottelo: [Discussion] Virt-who UI config test and approach to automate

[sghai]

All,

I would like to start a discussion on automating virt-who-config UI tests. Earlier we were thinking to automate as part of virt-who repo but then thought if we should automate as part of robottelo. I see value having the automation as part of robottelo but then I wanted to understand if there are concerns keeping then in robottelo. Few considerations, in case we are going to keep tests in robottelo:

a) We will run tests with virt-who jenkins as all the hypervisors are part of China lab
b) Plan to define a config file only for virt-who hypevisor configuration.

Now question is how we should read the virt-who config file ? Do we have some common read functions in robottelo ?

Please expect some questions here are there and lets be open to come up with conclusion.

[sghai]

@san7ket @jyejare @ntkathole: ^^

[ntkathole]

a) We will run tests with virt-who jenkins as all the hypervisors are part of China lab

This would be good. They just need to setup jenkins job that will run tests from robottelo in their Jenkins.

b) Plan to define a config file only for virt-who hypevisor configuration.

The main change, we will require to have same base config file as robottelo https://github.com/SatelliteQE/robottelo/blob/master/robottelo/config/base.py#L25 .
We can create new directory at https://github.com/SatelliteQE/robottelo/tree/master/robottelo with name virtwho_config(similar to config directory used by robottelo) and add functions to read properties file.

For virt_who tests, we can have separate directory same as we have for upgrades under robottelo/tree/master/tests/foreman/

This way we can keep tests as well as properties file of virtwho team separated. It won't affect our automation and they can consume robottelo helpers easily.

Adding just high level idea, need to lookup in depth for overall plan. Let's see what others think.

[jyejare]

I see only following values keeping these tests in Robottelo is:

1. If we are planning to run in upstream(if they are not running already)
2. If we want all the tests in a single place regardless of component. (Just like we moved all upgrade scenario tests in robottelo)

And,

a) We will run tests with virt-who jenkins as all the hypervisors are part of China lab

Yes, but if we really want to run those tests on our environement(may be in future), we may need PRs in robottelo-ci wrt virt-who.

b) Plan to define a config file only for virt-who hypevisor configuration.

I think keeping two config files isnt good for any test-framework So either we can add one-more config section for virt-who. And which I believe would be a challenge as Virt-who might have long list of properties.

Now question is how we should read the virt-who config file ? Do we have some common read functions in robottelo ?

We have INIReader class that can be usefull here but We have to define classes for each Virt-Who section/setting to read/validate/__init__. Its not a big thing but we may need to put efforts depends upoun the number of sections required for virt-who tests.

So overall, It looks like robottelo will be used just as placeholder for virt-who tests and I am ok if thats the intent. Otherwise, I will vote for keeping tests in virt-who repo only as the plot for running those tests would have already been set.

May be understanding the advantages more would help me to change my opinion.

[JacobCallahan]

I don't think we need to get this in-depth around this feature.
We essentially just need to make sure that the UI/CLI/(API?) components work and that the config that is generated correctly.

Once that has been verified, the actual virt-who part isn't within our scope. This is similar to content-type tests. We can verify that Satellite installed Firefox on a host, but we don't need to verify that Firefox can open up a website.

Essentially, I think we should just test the interface, then use the CLI/API to execute the command to put the config on the Satellite. Once that's done, then we'll need some way of verifying that the config is formatted correctly and contains the data we're expecting.

[eko999]

Hi all, I'm Eko,
I'd like to give more introduction about virt-who plugin testing, up to now, there are 6 hypervisor types supported by the plugin ['esx', 'xen', 'hyperv', 'rhevm', 'libvirt', 'kubevirt'],
we have to test all the hypervisors because different hypervisor has different configuration, take vcenter as an example, the UI test steps should be:

1. login satellite server
2. navigate Menu "Infrastructure" and click "Virt-who configurations"
3. click "Create Config" button
4. select "Vcenter" Type, fill out the form and then submit
5. copy the script to virt-who host
6. register the guest to satellite
7. run the script
8. check the hypervisior/guest mapping info in rhsm.log
9. check the guest is associated to hypervisor in satellite webUI
10. attach a physical SKU to the hypervisor, and then check the virtual SKU is generated or not for guest by webUI

After above steps, we can give a test result: the configuration of vcenter is passed to test. Of course, this is just a simple demo, we still need to re-edit the configuration to test different options' value, such as debug enable/disable, different interval time ......

[eko999]

The test environment for virt-who plugin should be as below, virt-who CI system can provision all of them, if we add the test cases to robottelo, we have to set and define these environment in robotello:

[virtwho]
host_ip=
host_user=
host_passwd=

[hypervisor]
type=
server=
server_username=
server_password=
server_ssh_user=
server_ssh_passwd=
server_config=
guest_ip=
guest_name=
guest_user=
guest_passwd=

[register]
type=satellite66-dogfood-rhel7
server=
owner=
env=
admin_user=
admin_passwd=
ssh_user=
ssh_passwd=

[manifest]
vdc=RH00001
vdc_bonus=RH00049
instance=RH00003
limit=RH00204
unlimit=RH00060

[sghai]

I don't think we need to get this in-depth around this feature.
We essentially just need to make sure that the UI/CLI/(API?) components work and that the config that is generated correctly.

Once that has been verified, the actual virt-who part isn't within our scope. This is similar to content-type tests. We can verify that Satellite installed Firefox on a host, but we don't need to verify that Firefox can open up a website.

Essentially, I think we should just test the interface, then use the CLI/API to execute the command to put the config on the Satellite. Once that's done, then we'll need some way of verifying that the config is formatted correctly and contains the data we're expecting.

Jake, virt-who QE team is now part of our group, although I understand that we should just focus on virt-who configuration but having one or two end_2_end tests won't hurt here given that same team is taking care of "virt-who" and "virt-who-config". For end to end test, we can target one hypervisor.

[JacobCallahan]

@sghai ahh I keep forgetting!
With that in mind, by all means, I'd love to have quite a lot more testing around virt-who in Satellite, as that has historically been a customer pain point.

[ntkathole]

Thank you all. Virt-who tests are part of robottelo!!