Robottelo: BZ logic should follow the clones to fetch the version flags.
BZ logic should also take a look at the flags on clones to get versions affected by that BZ.
rochacbruno
All 17 comments
Something like this?
- check version of product being tested: 6.1, 6.2, 6.3, 6.n, nightly
- check Bugzilla issue STATUS
- is test affected by bugzilla?
- YES: is bugzilla ready for QE?
- YES: is product version (i.e.
6.2) found in bugzilla flags(i.e.sat-6.2.z+)?
- YES: does the flag contain a
+?
- YES: execute the test
- NO: do not execute the test
- YES: does the flag contain a
- NO: check bugzilla for clones and go back to the
is bugzilla ready for QEstep
- YES: is product version (i.e.
- NO: do not execute the test
- YES: is bugzilla ready for QE?
omaciel
on 27 Jan 2017
We decided to clone BZs for other product versions.
When we will be checking cloned BZs statuses then the automation will start checking bugs automatically once they are switched to ONQA by devs.
We don't expect that devs would also go to the original bug and set there flags appropriately from ? to + for a specific version. If we were checking only flags then the automation would be skipping the bugs forever. Unless that extra (and manual) step would be done.
To minimize commits we will decorate all branches with the original BZ # and the automation will crawl to the relevant clone having '+' for specific version only. (the same as the automation is running for)
lpramuk
on 27 Jan 2017
Clone is fixed and original is still not?
Then we still need to get to clones, regardless of original bug status.
lpramuk
on 27 Jan 2017
@renzon take a look at https://github.com/ldjebran/robozilla we are already using it for wontfixes, and it takes clones and dupes.
rochacbruno
on 3 Mar 2017
Just fixing the version on @omaciel's logic:
- check version of product being tested: 6.1, 6.2, 6.3, 6.n, nightly
- check Bugzilla issue STATUS
- is test affected by bugzilla?
- YES: is bugzilla ready for QE?
- YES: is product version (i.e.
6.2) greater or equal to bugzilla flags(i.e.sat-6.2.z+)?
- YES: does the flag contain a
+?
- YES: execute the test
- NO: do not execute the test
- YES: does the flag contain a
- NO: check bugzilla for clones and go back to the
is bugzilla ready for QEstep
- YES: is product version (i.e.
- NO: do not execute the test
- YES: is bugzilla ready for QE?
renzon
on 7 Mar 2017
Just fixing the version on @omaciel's logic:
- check version of product being tested: 6.1, 6.2, 6.3, 6.n, nightly
- check Bugzilla issue STATUS
- is test affected by bugzilla?
- YES: is bugzilla ready for QE?
- YES: is product version (i.e.
6.2) greater or equal to bugzilla flags(i.e.sat-6.2.z+)?
- YES: does the flag contain a
+?
- YES: execute the test
- NO: do not execute the test
- YES: does the flag contain a
- NO: check bugzilla for clones and go back to the
is bugzilla ready for QEstep
- YES: is product version (i.e.
- NO: do not execute the test
- YES: is bugzilla ready for QE?
renzon
on 8 Mar 2017
This one is waiting to be unblocked.
@renzon we need to meet with @omaciel @JacobCallahan @rplevka @lpramuk again once we have final details of the workflow and which approach to implement.
rochacbruno
on 16 Mar 2017
@rochacbruno @JacobCallahan @rplevka @lpramuk: I spoke to @omaciel and process as follows:
- BZ entered on system, id 20 and state NEW with flags sat-* ?
- QE test decorated with BZ 20
- Fix will land on 6.3, so flag sat-6.3 is set to +
- Fix will be cherry picked to 6.2. BZ 20 is cloned (new id 30) and status is NEW, flag sat-6.2.z set to +
Right now we have 2 BZs which have their satellite target version with respective state. Based on that date our decorator logic is:
- Fetch BZ with id 20 and all its clones (doesn't matter if it is "cloned of" or "cloned from")
- Check Sat version
- Is there some bz marked with Sat version (SAT) o flag as "+" ?
- Yes. Check its state and decide run or skip
- No. Check all BZs where a test should run based on its status. Find the one with min version defined on flag sat-FLAG. Is SAT < FLAG?
- Yes. Skip because Satellite under test is old and the fix is not available for it
- No. Run because Satellite under test is newer than all versions where fix landed, so the fix should be available on it as well, otherwise it is a regression
My intention is working on existing test to add this logic considering clones. This description will be my guide on implementation and I am going to add this on the bz complete workflow on:
What you guys think?
renzon
on 23 Mar 2017
@renzon try to extract the logic out of decorator in helper functions so we can reuse the same workflow, for example there is #4262 which will use the same process.
There is a file for such helpers https://github.com/SatelliteQE/robottelo/blob/master/robottelo/bz_helpers.py
And maybe we can evaluate the approach of loading BZ data once and including on Pytest namespace as we are already doing the same for WONTFIX https://github.com/SatelliteQE/robottelo/blob/master/tests/foreman/conftest.py#L46
(But AFAIK fetching clones consumes more time so maybe better to keep it fetching on demand)
rochacbruno
on 23 Mar 2017
Fix will be cherry picked to 6.2. BZ 20 is cloned (new id 30) and status is NEW, flag sat-6.2.z set to +
when exactly will the clonining happen? as if they'll keep the BZ#20 with 6.2.3+ for a month, we will have a failed 6.2.z tests for a month...if we do the permanent workaround as it looks like it's never going to be backported back to 6.2.z, we will be fine...until they decide after 30 days, that they actually want to backport it :) in such case we will have to undo the workaround a decorate the test again, etc.
Timing is the key
rplevka
on 23 Mar 2017
@rplevka my bad, what I meant:
Fix will be cherry picked to 6.2. BZ 20 is cloned (new id 30). This new cloned BZ has status NEW and flag sat-6.2.z to +. So the flag mark is done exactly the same time we clone, e.g. the moment we decide to cherry pick the fiz. Maybe it's better to discuss on tomorrow's mtg ;)
renzon
on 23 Mar 2017
@renzon is the script going to fetch the clones of BZ#20 in case the flag does not match our version?
so if i put bz_bug_is_open(20) on sat6.2.10, and this BZ has 6.3.z+; it will evaluate as True becasue its clone - bz#30 has 6.2.z+ and is open?
rplevka
on 24 Mar 2017
@rplevka exactly
renzon
on 24 Mar 2017
@renzon
In a conversation last week with @JacobCallahan we decided that:
If BUG has duplicated version flags
['sat-6.3.z+', 'sat-6.2.z+']then we look totarget_milestoneif exists it should be considered as version, ifunspecifiedthe decorator shouldskipthe test.
Then we will setup a new process to send weekly email if that kind of duplication is found so someone can fix it by setting target_milestone and/or cloning the BZ.
rochacbruno
on 25 Apr 2017
- check version of product being tested: 6.1, 6.2, 6.3, 6.n, nightly
- check Bugzilla issue STATUS
- is test affected by bugzilla?
- YES: is bugzilla ready for QE?
- YES: is product version (i.e.
6.2) greater or equal to bugzilla flags(i.e.sat-6.2.z+)?
- YES: There is only one
sat-version flag?
- YES: does the flag contain a
+?
- YES: execute the test
- YES: does the flag contain a
- NO: There are duplicated version flags ['sat-6.3.z+', 'sat-6.2.z+']
- YES: is there
target_milestonespecified?
- YES: is
target_milestonematching the product version?
- YES: execute the test
- YES: is
- NO: is
target_milestoneunspecified or not matched
- NO: Skip the test
- YES: is there
- NO: do not execute the test
- YES: There is only one
- NO: check bugzilla for clones and go back to the
is bugzilla ready for QEstep
- YES: is product version (i.e.
- NO: do not execute the test
- YES: is bugzilla ready for QE?
rochacbruno
on 25 Apr 2017
we should draw this
rochacbruno
on 25 Apr 2017
In fact I have draw the simple version of it, related to clone. You forgot the case where we advance on version, so think sat 6.4. In your case, I would skip it when having dup on flags when it should not. I put thia again for discussion on weekly mtg.
renzon
on 25 Apr 2017
Related issues
sghai
路
9Comments
renzon
路
4Comments
rplevka
路
4Comments
renzon
路
5Comments
renzon
路
4Comments