Robottelo: Config validation functions not taking .yaml files into account

[lhellebr]

We have robottelo.properties and conf/*.yaml files to provide configuration, with yaml files having precedence. However, when using functions intended to validate presence of config, such as setting_is_set(), only robottelo.properties is taken into account.
This means that when the config is only present in robottelo.properties, the function passes. When the config is present in both robottelo.properties and a yaml file, the function passes and yaml file content is used. When the config is only present in a yaml file, the function fails. Example:

$ grep '\[vmware\]' robottelo.properties # vmware section is commented out
#[vmware]
$ cat conf/vmware.yaml # vmware is configured correctly in yaml
VMWARE:
  # Vmware to be added as a compute resource
  # vcenter: vmware vcenter URL
  VCENTER: [...]
$ grep -A1 setting_is_set tests/foreman/ui/test_computeresource_vmware.py # tests should be skipped if config not present
from robottelo.decorators import setting_is_set

if not setting_is_set('vmware'):
    pytest.skip('skipping tests due to missing vmware settings', allow_module_level=True)
$ pytest tests/foreman/ui/test_computeresource_vmware.py # the tests are skipped
====================================================== test session starts ======================================================
platform linux -- Python 3.8.5, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /home/lhellebr/git/robottelo/venv/bin/python
cachedir: .pytest_cache
shared_function enabled - OFF - scope:  - storage: file
rootdir: /home/lhellebr/git/robottelo, configfile: pytest.ini
plugins: xdist-2.2.1, ibutsu-1.14.1, services-2.2.1, mock-3.5.1, forked-1.3.0, cov-2.11.1, reportportal-5.0.8
collected 0 items / 1 skipped                                                                                                   

====================================================== 1 skipped in 0.04s =======================================================
2021-05-06 14:50:40 - robottelo - DEBUG - Selected 0 and deselected 0 tests based on new infra markers.

[lhellebr]

CC @jyejare @rmynar

[JacobCallahan]

@mshriver is this solved by #8572?

[mshriver]

No, as the core functionality of setting_is_set is retained.

Thoughts on separating this from #8572 to isolate removing the unittest.skip behavior from fixing this? I've got an idea for a pretty straightforward approach to bridge the gap to removing SettingsFacade/LegacySettings and having more in depth on-demand validation.

[jyejare]

@JacobCallahan @lhellebr Agree with Mike !

Currently the .validate() method for settings is validating settings in Legacy robottelo.properties and hence the tests are being skipped with setting_is_set if those settings are not initiated in robottelo.properties

Also, this has to be done before we remove LegacySettings / Facade else the number of tests would get skip ! We are success in 6.9.2 coz we still have robottelo.properties with all the details in gitlab and we are still using it in new pipeline.

Thanks @lhellebr for finding out this issue !

[mshriver]

@lhellebr I tested the commit in 8584 against a test with a few settings fields to skip on, with those settings in yaml, in .properties, missing in either/both, and with missing fields in .properties (making validation fail).

The skip behaved as expected in all cases - if it was set in yaml then validation against what is in .properties doesn't matter, and if its set in .properties and not in .yaml, then validation must pass.

Note the TODO about using dynaconf selective validation here, which will be far more effective, and can be lazily evaluated.

I'd appreciate your review of the test skipping behavior with the configuration in your local env for your component. Check out my branch for #8584 and you'll be reviewing #8584 and #8572 at the same time!

https://github.com/mshriver/robottelo/commit/20c42d509245ad354d288dc94949917edbfceae6