Robottelo: Define cross-browser support approach

[oshtaier]

As our entire framework and test coverage are pretty large, it is necessary to describe principals that should be followed in order having proper procedure for supporting different browsers for our testing:

• Do not jump between versions of one browser in unplanned way to not impact release process.
  For example, next scenario should be taken into account:
  1) We are working with Firefox xx for our test runs (xx version is locked and static)
  2) Next day we are going to receive Firefox xx+1 version
  3) Check that we have time to trigger automation run before next compose comes
  4) Run test automation jobs (only UI tests should be executed) to get failures that are specific to the browser upgrade procedure, not the product new version regressions
  5) Check how many failures we get and whether it is possible to fix our code before next compose. If not - run our scripts for next build for xx version. If yes - make sure that necessary PRs are submitted before running jobs.
• Decide what browser should be treated as a main one. Full test coverage should be executed for it. Presumable, it will be Firefox, but, of course, that point is discussable. All other browsers support should be started with smoke test suite and then, bunch by bunch, more tests should be added.
• It is expected that different browsers can have different representation of the same pages or elements, so it is a chance that we will need to have different locator repositories for different browsers
• More browsers - more coverage. More coverage - more efforts to maintain the code, analyze the results, test and re-test different test cases. It is necessary to have proper planning to mitigate the risks and avoid surprises, and be ready that more people and resources will be needed
• Cross-browser support is a very nice feature, but product should always have higher priority, so it is necessary to work to achieve green build first (failure rate is not higher then expected threshold), and only then work on more coverage for other browsers.
• It is possible that we will need to re-configure our CI jobs in a way that we have some kind of description whether that part of tierxx UI tests uses one type of a browser or another

[oshtaier]

Let's discuss in the thread and modify first post with final summary

[elyezer]

We need to have in mind that not all browsers and versions are supported. Check [1] for the official browser support matrix.

The link is for Satellite 6.1 but it will at least give us a better idea to what start with.

[1] https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/6.1/html/Installation_Guide/sect-Red_Hat_Satellite-Installation_Guide-Prerequisites.html#sect-Red_Hat_Satellite-Installation_Guide-Prerequisites-Supported_Browsers

[elyezer]

I would like to also state that we may find differences on how selenium is handled from one browser to the other. That means that we are going to need support that on our UI framework.

[elyezer]

The list below is the current test methods that are being tracked to get performance data, what about we starting with that set of tests in order to start testing against the browsers x versions matrix?

• tests/foreman/ui/test_login.py - test_positive_login
• tests/foreman/ui/test_organization.py:test_positive_create_with_name
• tests/foreman/ui/test_user.py:test_positive_create_with_username
• tests/foreman/ui/test_lifecycleenvironment.py:test_positive_create
• tests/foreman/ui/test_product.py:test_positive_create_with_name
• tests/foreman/ui/test_repository.py:test_positive_create_with_name
• tests/foreman/ui/test_contentview.py:test_positive_create_wth_name
• tests/foreman/ui/test_activationkey.py:test_positive_create_with_name

We can also add other test methods in order to cover more the product, but those test methods are the core for the other features.

Thanks @sthirugn who created the list for the Performance Report, I hope you don't mind me "stealing" it for the browser testing :smile:

[ntkathole]

Please feel free to re-open, if this still needs fixing.