Robotframework: New `SKIP` status for tests (discussion)

Created on 5 Aug 2015  ·  132Comments  ·  Source: robotframework/robotframework

There have been several requests to be able to mark tests skipped: #812, #837, #964, #1732, #2064. Some of them use different term than skip (e.g. warn or blacklist) but the basic idea is similar.

So far all such requests have been rejected because Robot Framework already has a concept of non-critical tests and non-critical failures with a clear error message can be used to emulate skipping. I'm not inherently against skipping, though, and see that explicit SKIP status would often be more clear than non-critical FAIL. Many other test tool also has such status and being compatible with them would be nice.

I'm not planning to implement this functionality in the foreseeable future, but I wanted to open an issue about it to have a place where to discuss it. The main problem with implementation is that it would be quite a big task. The first step would be designing how the functionality actually should work. At least these things should be taken into account:

  1. How to mark a test skipped. Possibilities include:

    • Based on tags from the command line.

    • Using a BuiltIn keyowrd.

    • Using a custom exception in a test library.

  2. How skipped tests affect statistics?
  3. How skipped tests affect overall execution result? What should the report background be if there are passed and skipped tests in a suite? What if there are only skipped?

I encourage people interested about the skip status to comment here or possibly start a discussion on the robotframework-users mailing list or on the new Slack community. If we can agree with the overall design, someone can then start actually implementing the functionality.

duplicate enhancement high
Source
picture pekkaklarck
👍7

Most helpful comment

The long awaited skip status is coming in RF 4.0 that's planned for December! I submitted #3622 with more concrete design and close this one as its duplicate.

As you'll see from #3622, criticality will go when skipping comes. I know its a controversial decision but I've been thinking this for a long time and believe that's the best way forward. For more reasoning see the separate issue #3624.

picture pekkaklarck on 3 Aug 2020
❤2 👍2 🎉1

All 132 comments

This is my idea:
- SKIP status is very useful when test environment is complex and not stable(even though stable environment test is very recommended, but sometimes it needed in development period). After check test environment and test case could SKIP dynamically.
1. Tags with SKIP and Builtin keyword(maybe useful in setup keyword) would be nice.
( Why don't you allow all of them? )
2. Sum of test cases does not contain SKIP status, but it's count should be showed with that.
3. All of result ( include SKIP ) should be reported, but it must be overtly divided because that's why support SKIP status. I think how to divide that is not important but to take previous pattern, new page(skip) would be nice like report, log page.
If all test are skipped in a suite, that could be SKIP but in the other cases, SKIP don't have to affect overall execution result.

This is my first idea so I think this is not perfect. Please add a comment. Anyway I'm glad to you open this issue, because robot framework have a strict rule I thought. But robot could be bigger framework by effort like this. Thank you.

synsun picture synsun on 10 Aug 2015

I have a need for this 'SKIPPED' status. This happen when robot is hooked with test case management system (TCMS) when executing a test run and some cases have already 'PASSED'; therefore, no need to run that particular cases anymore; thus, robot should mark as 'SKIPPED'

In my case BuiltIn keyword is needed due to its flexibility. If it was a custom exception, I wonder if external libraries can 'raise' that.

jedsadas picture jedsadas on 20 Aug 2015

I think the simplest solution is a built-in keyword (eg: Skip test), which works via a custom exception that can also be used by libraries.

Then, I would add another column in the statistics so we now have Total (including skipped), Pass, Fail, and Skipped. Beyond that, skipped tests shouldn't affect anything else -- they don't change the color, don't change the overall status, etc.

boakley picture boakley on 23 Sep 2015
👍2

I completely agree with boakley above in the way I would like this to work.

There are 2 things to address in the solution when I compile in my mind what
the user community is asking. Namely first skipping a testcase, second reporting.

Skipping the testcase is the real important part, that’s the functionality users want.
:It would be nice if the status of the testcases were reported as SKIP, but that is not
100% necessary. What I mean is, at the very minimum, implement the ability to skip
a testcase via being able to call a keyword from * Settings ** within an *init.py,
a testcase.txt, a resource.txt file or a libray.py file. The point being the decision to skip or
not is made and taken before the actual testcase starts. In sudo code this would look like this:

*** Settings ***
Documentation
...    Objective: TAG Test 1
...

Library          BuiltIn
Library          SSHLibrary
Library          Collections
Default Tags     smoke_test
Suite Setup      Example

*** Keywords ***
Example
    # Check if Testcase is Relevant (should be skipped)
    # -------------------------------------------------
    ${result}=    Skip If Testcase Not Applicable
    Run Keyword If    '${result}' == 'skip'    Skip Test

*** Test Cases ***
TAG_Test_1
    Log  ===========> *** Test TAG_1 Running *** <===========
    etc.

Hence if the testcase can be prevented from actually starting (hence totally skipped)
then result reporting does not need to change because the test should not even by
executed/run (hence behave as if its Tags did not match the selection etc.). This would
then allow users to dynamically skip tests and this part would not be that difficult to
anyone who is familiar with the robot source.

This to me would be perfectly acceptable as the PASS/FAIL counts would be consistent
and so a Development & Regression team could make software release decisions to
ship once a percentage ‘%’ rate is achieved. Managers (Development or Regression teams)
are not interested in how many tests were skipped, only the PASS/FAIL rate. If an Engineer
wants to know if a testcase was skipped then they could check the report, log etc.

If you are going to support reporting skipped testcases, then again as boakley stated above, e.g.

“add another column in the statistics so we now have Total (including skipped), Pass, Fail, and Skipped. Beyond that, skipped tests shouldn't affect anything else -- they don't change the colour, don't change the overall status, etc.”

If you would lie any clarity or for me to expand on this I will do all I can to help.
I would love this functionality ASAP.

I’m willing to be an Alpha/Beta tester for this.

Regards,
Doug

Doug8080 picture Doug8080 on 6 Oct 2015
👍2

With reference the pull request: PR #2179

The changes I would like to proposed in this pull request are to add support for skipping testcases
via the use of a new keyword ‘Skip Execution’. Additionally support for a command line option
--skip has been added. The --skip option works like the --exclude option except that tests are shown
in the statistics. Finally support has been added for a special attribute ROBOT_SKIP_ON_FAILURE
on exceptions to mark tests skipped. This is consistent with continuable and fatal failures
(ROBOT_CONTINUE_ON_FAILURE and ROBOT_EXIT_ON_FAILURE) and simplifies logic at the test case
level.

I have fixed up all the unit tests (under the utest directory), they all pass successfully, e.g.

debian@debian:/usr/local/lib/python2.7/dist-packages/robotframework-GITHUB/utest$ sudo ./run.py 
...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 1475 tests in 5.708s

OK
debian@debian:/usr/local/lib/python2.7/dist-packages/robotframework-GITHUB/utest$

I have also update the source code documentation for the --skip option (under the doc directory).

I have created 5 example/demo testcase files to demonstrate the use of the ‘Skip Execution’ keyword and --skip option.

Example/demo Testcase Files:
TEST_SKIP.txt

*** Settings ***
Documentation
...    Objective:  Test Skipped Testcase Reporting
...

Library          BuiltIn
Default Tags     smoke_test

*** Keywords ***
Skip Test
    Set Test Variable    ${release}      R1
    Run Keyword If      '${release}' == 'R1'
    ...    Skip Execution   Skipping Testcase (it is not relevant to this image)

*** Test Cases ***
Test_Skip
    # Check if Testcase is Relevant (should be skipped)
    # -------------------------------------------------
    Skip Test

    Log  ===========> *** RUNNING TEST_SKIP *** <===========

TEST_SKIP_STATIC_TAG.txt

*** Settings ***
Documentation
...    Objective:  Test Skipped Static Tag Testcase Reporting
...

Library          BuiltIn
Default Tags     smoke_test  my_skip

*** Test Cases ***
Test_Skip_Static_Tag
    # Run the testcase
    # ----------------

    Log  ===========> *** RUNNING TEST_SKIP_STATIC_TAG *** <===========

TEST_SKIP_DYNAMIC_TAG.txt

*** Settings ***
Documentation
...    Objective:  Test Skipped Testcase Reporting
...

Library          BuiltIn
Default Tags     smoke_test

*** Test Cases ***
Test_Skip_Dynamic_Tag
    [Tags]    my_skip
    # Run the testcase
    # ----------------

    Log  ===========> *** RUNNING TEST_SKIP_DYNAMIC_TAG *** <===========

TEST_PASS.txt

*** Settings ***
Documentation
...    Objective:  Test Passed Testcase Reporting
...

Library          BuiltIn
Default Tags     smoke_test

*** Keywords ***
Skip Test
    Set Test Variable    ${release}      R2
    Run Keyword If      '${release}' == 'R1'
    ...    Skip Execution   Skipping Testcase (it is not relevant to this image)

*** Test Cases ***
Test_Pass
    # Check if Testcase is Relevant (should be skipped)
    # -------------------------------------------------
    Skip Test

    Log  ===========> *** RUNNING TEST_PASS *** <===========

TEST_FAIL.txt

*** Settings ***
Documentation
...    Objective: Test Failed Testcase Reporting
...

Library          BuiltIn
Default Tags     smoke_test

*** Test Cases ***
Test_Fail
    Log  ===========> *** RUNNING TEST_FAIL *** <===========

    # Force a testcase failure
    Set Test Variable     ${result}    FORCE_FAILURE
    Should Be Equal       ${result}    Passed

Issues
I have this working 99.9%. There are two small issues remaining that I have not been able to address.

Issue 1:
How to fix up the file below for adding ROBOT_SKIP_ON_FAILURE support.

atest/robot/output/html_output_stats.py

Issue 2:
The --skip option works exactly like the --exclude option at present. I do not quite understand how
to show the skipped tests in the statistics when the --skip option is used. Skipped tests via the keyword
‘Skip Execution’ are sown in the statistics, but I do not know how to show the tests skipped via the
--skip option in the statistics.

Issue 3:
The final issue remaining that I have not been able to address is the case when a critical testcase is
skipped the console output is correct, however the HTML report is classifying a SKIP result as a critical failure (i.e. in the HTML report it states for example 'Status: 1 critical test failed'). I have
figured out (I think) that this needs to be fixed in the file in src/robot/htmldata/rebot/testdata.js
in the code block below. However I cannot figure out what exactly is in ‘stats’. My Jython is not good
– for now. This code block needs to not consider a SKIP result as a failure. Fix this and it will fix the
HTML reporting for critical testcases with SKIP results.

    function suiteStats(stats) {
        return {
            total: stats[0],
            totalPassed: stats[1],
            totalFailed: stats[0] - stats[1],
            critical: stats[2],
            criticalPassed: stats[3],
            criticalFailed: stats[2] - stats[3]
        };
    }

If you or anyone on the robot developers team can help me with this then I’ll do all I can to
incorporate this in this pull request. Alternatively if you would like to simply take my changes and address this issue in the team then I am more than happy with this. My interest is in getting this implemented and released to help my company and the wider user community here.

Regards,
Doug

Testing Results
Below are test runs using the above example/demo files. I’ve included console outputs and screen
shots of the HTML reports generated for the runs.

TEST: Skip Test Non Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP --include smoke_test -n smoke_test tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                                  
====================================================================================================================================================================================
Test_Skip                                                                                                                                                                   | SKIP |
Skipping Testcase (it is not relevant to this image)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                          | SKIP |
1 critical test, 0 passed, 0 failed, 1 skipped
1 test total, 0 passed, 0 failed, 1 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | SKIP |
1 critical test, 0 passed, 0 failed, 1 skipped
1 test total, 0 passed, 0 failed, 1 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

skip_test_non_critical

TEST: Skip & Pass Tests Non Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP -s TEST_PASS --include smoke_test -n smoke_test tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Pass                                                                                                                                                                   | PASS |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                           | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                                  
====================================================================================================================================================================================
Test_Skip                                                                                                                                                                   | SKIP |
Skipping Testcase (it is not relevant to this image)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                          | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 0 passed, 0 failed, 1 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
2 tests total, 1 passed, 0 failed, 1 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

skip_test_pass_test_non_critial

TEST: Skip Static Tag Test Non Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP_STATIC_TAG --include smoke_test -n smoke_test --skip my_skip tests/skip_tests/
[ ERROR ] Suite 'Skip Tests' contains no tests with tag 'smoke test' and without tag 'my skip' in suite 'TEST_SKIP_STATIC_TAG'.

Try --help for usage information.
debian@debian:~/work/auto$

TEST: Skip Static Tag & Pass Test Non Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP_STATIC_TAG -s TEST_PASS --include smoke_test -n smoke_test --skip my_skip tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Pass                                                                                                                                                                   | PASS |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                           | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

skip_test_static_tag_pass_test_non_critial

TEST: Skip Dynamic Tag Test Non Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP_DYNAMIC_TAG --include smoke_test -n smoke_test --skip my_skip tests/skip_tests/
[ ERROR ] Suite 'Skip Tests' contains no tests with tag 'smoke test' and without tag 'my skip' in suite 'TEST_SKIP_DYNAMIC_TAG'.

Try --help for usage information.
debian@debian:~/work/auto$

TEST: Skip Dynamic Tag & Pass Test Non Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP_DYNAMIC_TAG -s TEST_PASS --include smoke_test -n smoke_test --skip my_skip tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Pass                                                                                                                                                                   | PASS |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                           | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

skip_test_dynamic_tag_pass_test_non_critial

TEST: Pass Test Non Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_PASS --include smoke_test -n smoke_test --skip my_skip tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Pass                                                                                                                                                                   | PASS |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                           | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

pass_test_non_critical

TEST: Fail Test Non Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_FAIL --include smoke_test -n smoke_test --skip my_skip tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST FAIL :: Objective: Test Failed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Fail                                                                                                                                                                   | FAIL |
FORCE_FAILURE != Passed
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST FAIL :: Objective: Test Failed Testcase Reporting                                                                                                           | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 0 passed, 1 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 0 passed, 1 failed, 0 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

fail_test_non_critical

TEST: All Tests Non Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP -s TEST_SKIP_STATIC_TAG -s TEST_SKIP_DYNAMIC_TAG -s TEST_PASS -s TEST_FAIL --include smoke_test -n smoke_test --skip my_skip tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST FAIL :: Objective: Test Failed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Fail                                                                                                                                                                   | FAIL |
FORCE_FAILURE != Passed
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST FAIL :: Objective: Test Failed Testcase Reporting                                                                                                           | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 0 passed, 1 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Pass                                                                                                                                                                   | PASS |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                           | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                                  
====================================================================================================================================================================================
Test_Skip                                                                                                                                                                   | SKIP |
Skipping Testcase (it is not relevant to this image)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                          | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
1 test total, 0 passed, 0 failed, 1 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped
3 tests total, 1 passed, 1 failed, 1 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

all_test_non_critical

TEST: Skip Test Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP --include smoke_test tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                                  
====================================================================================================================================================================================
Test_Skip                                                                                                                                                                   | SKIP |
Skipping Testcase (it is not relevant to this image)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                          | SKIP |
1 critical test, 0 passed, 0 failed, 1 skipped
1 test total, 0 passed, 0 failed, 1 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | SKIP |
1 critical test, 0 passed, 0 failed, 1 skipped
1 test total, 0 passed, 0 failed, 1 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

skip_test_critical

TEST: Skip & Pass Tests Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP -s TEST_PASS --include smoke_test tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Pass                                                                                                                                                                   | PASS |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                           | PASS |
1 critical test, 1 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                                  
====================================================================================================================================================================================
Test_Skip                                                                                                                                                                   | SKIP |
Skipping Testcase (it is not relevant to this image)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                          | SKIP |
1 critical test, 0 passed, 0 failed, 1 skipped
1 test total, 0 passed, 0 failed, 1 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | PASS |
2 critical tests, 1 passed, 0 failed, 1 skipped
2 tests total, 1 passed, 0 failed, 1 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

skip_test_pass_test_critial

TEST: Skip Static Tag Test Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP_STATIC_TAG --include smoke_test --skip my_skip tests/skip_tests/
[ ERROR ] Suite 'Skip Tests' contains no tests with tag 'smoke test' and without tag 'my skip' in suite 'TEST_SKIP_STATIC_TAG'.

Try --help for usage information.
debian@debian:~/work/auto$

TEST: Skip Static Tag & Pass Test Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP_STATIC_TAG -s TEST_PASS --include smoke_test --skip my_skip tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Pass                                                                                                                                                                   | PASS |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                           | PASS |
1 critical test, 1 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | PASS |
1 critical test, 1 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

skip_test_static_tag_pass_test_critial

TEST: Skip Dynamic Tag Test Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP_DYNAMIC_TAG --include smoke_test --skip my_skip tests/skip_tests/
[ ERROR ] Suite 'Skip Tests' contains no tests with tag 'smoke test' and without tag 'my skip' in suite 'TEST_SKIP_DYNAMIC_TAG'.

Try --help for usage information.
debian@debian:~/work/auto$

TEST: Skip Dynamic Tag & Pass Test Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP_DYNAMIC_TAG -s TEST_PASS --include smoke_test --skip my_skip tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Pass                                                                                                                                                                   | PASS |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                           | PASS |
1 critical test, 1 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | PASS |
1 critical test, 1 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

skip_test_dynamic_tag_pass_test_critial

TEST: Pass Test Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_PASS --include smoke_test --skip my_skip tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Pass                                                                                                                                                                   | PASS |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                           | PASS |
1 critical test, 1 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | PASS |
1 critical test, 1 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

pass_test_critical

TEST: Fail Test Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_FAIL --include smoke_test --skip my_skip tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST FAIL :: Objective: Test Failed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Fail                                                                                                                                                                   | FAIL |
FORCE_FAILURE != Passed
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST FAIL :: Objective: Test Failed Testcase Reporting                                                                                                           | FAIL |
1 critical test, 0 passed, 1 failed, 0 skipped
1 test total, 0 passed, 1 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | FAIL |
1 critical test, 0 passed, 1 failed, 0 skipped
1 test total, 0 passed, 1 failed, 0 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

fail_test_critical

TEST: All Tests Critical

debian@debian:~/work/auto$ pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP -s TEST_SKIP_STATIC_TAG -s TEST_SKIP_DYNAMIC_TAG -s TEST_PASS -s TEST_FAIL --include smoke_test --skip my_skip tests/skip_tests/
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                            
====================================================================================================================================================================================
Skip Tests.TEST FAIL :: Objective: Test Failed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Fail                                                                                                                                                                   | FAIL |
FORCE_FAILURE != Passed
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST FAIL :: Objective: Test Failed Testcase Reporting                                                                                                           | FAIL |
1 critical test, 0 passed, 1 failed, 0 skipped
1 test total, 0 passed, 1 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                                   
====================================================================================================================================================================================
Test_Pass                                                                                                                                                                   | PASS |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST PASS :: Objective: Test Passed Testcase Reporting                                                                                                           | PASS |
1 critical test, 1 passed, 0 failed, 0 skipped
1 test total, 1 passed, 0 failed, 0 skipped
====================================================================================================================================================================================
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                                  
====================================================================================================================================================================================
Test_Skip                                                                                                                                                                   | SKIP |
Skipping Testcase (it is not relevant to this image)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Skip Tests.TEST SKIP :: Objective: Test Skipped Testcase Reporting                                                                                                          | SKIP |
1 critical test, 0 passed, 0 failed, 1 skipped
1 test total, 0 passed, 0 failed, 1 skipped
====================================================================================================================================================================================
Skip Tests :: Test Suite                                                                                                                                                    | FAIL |
3 critical tests, 1 passed, 1 failed, 1 skipped
3 tests total, 1 passed, 1 failed, 1 skipped
====================================================================================================================================================================================
Debug:   /home/debian/work/auto/debug.txt
Output:  /home/debian/work/auto/output.xml
Log:     /home/debian/work/auto/log.html
Report:  /home/debian/work/auto/report.html
debian@debian:~/work/auto$

all_test_critical

Doug8080 picture Doug8080 on 16 Oct 2015

I would suggest changing it to just "skip execution". If you need to do it
conditionally you can combine it with "run keyword if" "run keyword if".
By only supporting "skip execution if", there will be times when a test may
do "skip execution if | True", which is somewhat cumbersome.

If you feel strongly about the need for "skip execution if", then I humbly
suggest you provide two keywords - "skip execution" and "skip execution
if".

Otherwise, this is very encouraging!

--bryan

On Thu, Oct 15, 2015 at 7:23 PM, Doug Gordon [email protected]
wrote:

Hi, I have proposed a source code enhancement to the robot framework via:

GitHub robot framework pull request #2179
https://github.com/robotframework/robotframework/pull/2179

This pull request adds support for skipping testcases via the use of a new
keyword
‘Skip Execution If’. I would very much welcome your comments and feedback
on the
information below where I have tried to explain to you how this would work.

I have created 3 example/demo testcase files to demonstrate the use of the
‘Skip Execution If’
keyword.

_Example/demo Testcase Files:_

_TEST_SKIP.txt_

* Settings *
Documentation
... Objective: Test Skipped Testcase Reporting
...

Library BuiltIn
Default Tags smoke_test

* Keywords *
Skip Test
Set Test Variable ${release} R1
Skip Execution If '${release}' == 'R1' Skipping Testcase (it is not relevant to this image)

* Test Cases *
Test_Skip
# Check if Testcase is Relevant (should be skipped)
# -------------------------------------------------
Skip Test

Log  ===========> *** RUNNING TEST_SKIP *** <===========

_TEST_PASS.txt_

* Settings *
Documentation
... Objective: Test Passed Testcase Reporting
...

Library BuiltIn
Default Tags smoke_test

* Keywords *
Skip Test
Set Test Variable ${release} R2
Skip Execution If '${release}' == 'R1' Skipping Testcase (it is not relevant to this image)

* Test Cases *
Test_Pass
# Check if Testcase is Relevant (should be skipped)
# -------------------------------------------------
Skip Test

Log  ===========> *** RUNNING TEST_PASS *** <===========

_TEST_FAIL.txt_

* Settings *
Documentation
... Objective: Test Failed Testcase Reporting
...

Library BuiltIn
Default Tags smoke_test

* Test Cases
Test_Fail
Log ===========> ** RUNNING TEST_FAIL *** <===========

# Force a testcase failure
Set Test Variable     ${result}    FORCE_FAILURE
Should Be Equal       ${result}    Passed

Below are test runs using the above example/demo files. I’ve included
console outputs and screen
shots of the HTML reports generated for the runs.

I have this working 99.9%. The one issue remaining that I have not been
able to address is the case
when a critical testcase is skipped the console output is correct, however
the HTML report is
classifying a SKIP result as a failure. I have figured out that this needs
to be fixed in the file in
src/robot/htmldata/rebot/testdata.js in the code block below. However I
cannot figure out what exactly
is in ‘stats’. My Jython is not good – for now. This code block needs to
not consider a SKIP result as
a failure. Fix this and it will fix the HTML reporting for critical
testcases with SKIP results.

function suiteStats(stats) {
    return {
        total: stats[0],
        totalPassed: stats[1],
        totalFailed: stats[0] - stats[1],
        critical: stats[2],
        criticalPassed: stats[3],
        criticalFailed: stats[2] - stats[3]
    };
}

If you or anyone on the robot developers team can help me with this then
I’ll do all I can to
incorporate this in this pull request. Alternatively if you would like to
take my changes and address
this issue in the team then I am more than happy with this. My interest is
in getting this implemented
and released to help my company and the user community here.

Regards,
Doug

_TEST: Skip Test Non Critical_

debian@debian:~/work/robot pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP --include smoke_test -n smoke_test tests/skip_tests

Flow Monitoring :: Test Suite

Flow Monitoring.TEST SKIP :: Objective: Test Skipped Testcase Reporting

Test_Skip | SKIP |

Skipping Testcase (it is not relevant to this image)

Flow Monitoring.TEST SKIP :: Objective: Test Skipped Testcase Reporting | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped

1 test total, 0 passed, 0 failed, 1 skipped

Flow Monitoring :: Test Suite | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped

1 test total, 0 passed, 0 failed, 1 skipped

Debug: /home/debian/work/robot/debug.txt
Output: /home/debian/work/robot/output.xml
Log: /home/debian/work/robot/log.html
Report: /home/debian/work/robot/report.html
debian@debian:~/work/robot

HTML Report
[image: skip_test_non_critical]
https://cloud.githubusercontent.com/assets/12747505/10530661/cd965f4a-73a1-11e5-8b07-577826eb0e9a.jpg

_TEST: Pass Test Non Critical_

debian@debian:~/work/robot pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_PASS --include smoke_test -n smoke_test tests/skip_tests

Flow Monitoring :: Test Suite

Flow Monitoring.TEST PASS :: Objective: Test Passed Testcase Reporting

Test_Pass | PASS |

Flow Monitoring.TEST PASS :: Objective: Test Passed Testcase Reporting | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped

1 test total, 1 passed, 0 failed, 0 skipped

Flow Monitoring :: Test Suite | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped

1 test total, 1 passed, 0 failed, 0 skipped

Debug: /home/debian/work/robot/debug.txt
Output: /home/debian/work/robot/output.xml
Log: /home/debian/work/robot/log.html
Report: /home/debian/work/robot/report.html
debian@debian:~/work/robot

HTML Report
[image: pass_test_non_critical]
https://cloud.githubusercontent.com/assets/12747505/10530689/05f79b9c-73a2-11e5-80ba-bffaf4afab18.jpg

_TEST: Fail Test Non Critical_

debian@debian:~/work/robot pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_FAIL --include smoke_test -n smoke_test tests/skip_tests

Flow Monitoring :: Test Suite

Flow Monitoring.TEST FAIL :: Objective: Test Failed Testcase Reporting

Test_Fail | FAIL |

FORCE_FAILURE != Passed

Flow Monitoring.TEST FAIL :: Objective: Test Failed Testcase Reporting | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped

1 test total, 0 passed, 1 failed, 0 skipped

Flow Monitoring :: Test Suite | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped

1 test total, 0 passed, 1 failed, 0 skipped

Debug: /home/debian/work/robot/debug.txt
Output: /home/debian/work/robot/output.xml
Log: /home/debian/work/robot/log.html
Report: /home/debian/work/robot/report.html
debian@debian:~/work/robot

HTML Report
[image: fail_test_non_critical]
https://cloud.githubusercontent.com/assets/12747505/10530695/25af130c-73a2-11e5-8508-a244fece0a33.jpg

_TEST: Skip Pass Fail Test Non Critical_

debian@debian:~/work/robot pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP -s TEST_PASS -s TEST_FAIL --include smoke_test -n smoke_test tests/skip_tests

Flow Monitoring :: Test Suite

Flow Monitoring.TEST FAIL :: Objective: Test Failed Testcase Reporting

Test_Fail | FAIL |

FORCE_FAILURE != Passed

Flow Monitoring.TEST FAIL :: Objective: Test Failed Testcase Reporting | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped

1 test total, 0 passed, 1 failed, 0 skipped

Flow Monitoring.TEST PASS :: Objective: Test Passed Testcase Reporting

Test_Pass | PASS |

Flow Monitoring.TEST PASS :: Objective: Test Passed Testcase Reporting | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped

1 test total, 1 passed, 0 failed, 0 skipped

Flow Monitoring.TEST SKIP :: Objective: Test Skipped Testcase Reporting

Test_Skip | SKIP |

Skipping Testcase (it is not relevant to this image)

Flow Monitoring.TEST SKIP :: Objective: Test Skipped Testcase Reporting | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped

1 test total, 0 passed, 0 failed, 1 skipped

Flow Monitoring :: Test Suite | PASS |
0 critical tests, 0 passed, 0 failed, 0 skipped

3 tests total, 1 passed, 1 failed, 1 skipped

Debug: /home/debian/work/robot/debug.txt
Output: /home/debian/work/robot/output.xml
Log: /home/debian/work/robot/log.html
Report: /home/debian/work/robot/report.html
debian@debian:~/work/robot

HTML Report
[image: skip_pass_fail_test_non_critical]
https://cloud.githubusercontent.com/assets/12747505/10530709/4e7532a8-73a2-11e5-9cd2-e51258a2743c.jpg

_TEST: Skip Test Critical_

debian@debian:~/work/robot pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP --include smoke_test tests/skip_tests

Flow Monitoring :: Test Suite

Flow Monitoring.TEST SKIP :: Objective: Test Skipped Testcase Reporting

Test_Skip | SKIP |

Skipping Testcase (it is not relevant to this image)

Flow Monitoring.TEST SKIP :: Objective: Test Skipped Testcase Reporting | SKIP |
1 critical test, 0 passed, 0 failed, 1 skipped

1 test total, 0 passed, 0 failed, 1 skipped

Flow Monitoring :: Test Suite | SKIP |
1 critical test, 0 passed, 0 failed, 1 skipped

1 test total, 0 passed, 0 failed, 1 skipped

Debug: /home/debian/work/robot/debug.txt
Output: /home/debian/work/robot/output.xml
Log: /home/debian/work/robot/log.html
Report: /home/debian/work/robot/report.html
debian@debian:~/work/robot

HTML Report
[image: skip_test_critical]
https://cloud.githubusercontent.com/assets/12747505/10530741/709f83c4-73a2-11e5-82a2-9fa3649938f2.jpg

_*TEST: Pass Test Critical_

debian@debian:~/work/robot pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_PASS --include smoke_test tests/skip_tests

Flow Monitoring :: Test Suite

Flow Monitoring.TEST PASS :: Objective: Test Passed Testcase Reporting

Test_Pass | PASS |

Flow Monitoring.TEST PASS :: Objective: Test Passed Testcase Reporting | PASS |
1 critical test, 1 passed, 0 failed, 0 skipped

1 test total, 1 passed, 0 failed, 0 skipped

Flow Monitoring :: Test Suite | PASS |
1 critical test, 1 passed, 0 failed, 0 skipped

1 test total, 1 passed, 0 failed, 0 skipped

Debug: /home/debian/work/robot/debug.txt
Output: /home/debian/work/robot/output.xml
Log: /home/debian/work/robot/log.html
Report: /home/debian/work/robot/report.html
debian@debian:~/work/robot

HTML Report
[image: pass_test_critical]
https://cloud.githubusercontent.com/assets/12747505/10530788/d7d91c30-73a2-11e5-96e4-e59740de7321.jpg

_TEST: Fail Test Critical_

debian@debian:~/work/robot pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_FAIL --include smoke_test tests/skip_tests

Flow Monitoring :: Test Suite

Flow Monitoring.TEST FAIL :: Objective: Test Failed Testcase Reporting

Test_Fail | FAIL |

FORCE_FAILURE != Passed

Flow Monitoring.TEST FAIL :: Objective: Test Failed Testcase Reporting | FAIL |
1 critical test, 0 passed, 1 failed, 0 skipped

1 test total, 0 passed, 1 failed, 0 skipped

Flow Monitoring :: Test Suite | FAIL |
1 critical test, 0 passed, 1 failed, 0 skipped

1 test total, 0 passed, 1 failed, 0 skipped

Debug: /home/debian/work/robot/debug.txt
Output: /home/debian/work/robot/output.xml
Log: /home/debian/work/robot/log.html
Report: /home/debian/work/robot/report.html
debian@debian:~/work/robot

HTML Report
[image: fail_test_critical]
https://cloud.githubusercontent.com/assets/12747505/10530818/13c975a0-73a3-11e5-9db9-514f5fff2bcf.jpg

_TEST: Skip Pass Fail Test Critical_

debian@debian:~/work/robot pybot -W 180 -b debug -v testbed_name:uks3sb-1 -v fail_recover:False -s TEST_SKIP -s TEST_PASS -s TEST_FAIL --include smoke_test tests/skip_tests

Flow Monitoring :: Test Suite

Flow Monitoring.TEST FAIL :: Objective: Test Failed Testcase Reporting

Test_Fail | FAIL |

FORCE_FAILURE != Passed

Flow Monitoring.TEST FAIL :: Objective: Test Failed Testcase Reporting | FAIL |
1 critical test, 0 passed, 1 failed, 0 skipped

1 test total, 0 passed, 1 failed, 0 skipped

Flow Monitoring.TEST PASS :: Objective: Test Passed Testcase Reporting

Test_Pass | PASS |

Flow Monitoring.TEST PASS :: Objective: Test Passed Testcase Reporting | PASS |
1 critical test, 1 passed, 0 failed, 0 skipped

1 test total, 1 passed, 0 failed, 0 skipped

Flow Monitoring.TEST SKIP :: Objective: Test Skipped Testcase Reporting

Test_Skip | SKIP |

Skipping Testcase (it is not relevant to this image)

Flow Monitoring.TEST SKIP :: Objective: Test Skipped Testcase Reporting | SKIP |
1 critical test, 0 passed, 0 failed, 1 skipped

1 test total, 0 passed, 0 failed, 1 skipped

Flow Monitoring :: Test Suite | PASS |
3 critical tests, 1 passed, 1 failed, 1 skipped

3 tests total, 1 passed, 1 failed, 1 skipped

Debug: /home/debian/work/robot/debug.txt
Output: /home/debian/work/robot/output.xml
Log: /home/debian/work/robot/log.html
Report: /home/debian/work/robot/report.html
debian@debian:~/work/robot

HTML Report
[image: skip_pass_fail_test_critical]
https://cloud.githubusercontent.com/assets/12747505/10530842/2f6130d2-73a3-11e5-9622-9a5803e6cf6c.jpg

—
Reply to this email directly or view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-148559931
.

boakley picture boakley on 16 Oct 2015

Don't have much time for this today or during the weekend but some quick comments:

  1. On a quick look the PR #2179 looks pretty good. Nice to see implementing skipping like this isn't too complicated. More importantly, it's great that there is a concrete version that can be tested.
  2. If skipping is supported, I think it should also be supported based on tags e.g. like --skip mytag. Tests having skipped tags shouldn't be executed at all, and a dynamically added skip tag should mark test skipped afterwards. In practice --skip would work like --exclude but tests would be shown in statistics.
  3. To make it easier for libraries to mark tests skipped, we could consider supporting special attribute like ROBOT_SKIP_ON_FAILURE on exceptions to mark tests skipped. This would be consistent with continuable and fatal failures (ROBOT_CONTINUE_ON_FAILURE and ROBOT_EXIT_ON_FAILURE) and would simplify logic on test case level.
  4. I'm not entirely sure is it a good idea to add skip columns to statistics if no tests are skipped. If 2. and 3. above are implemented, skipping gets more usages and showing it in tables would be OK.
  5. Tests are needed. Possibly broken existing tests need to be fixed.
  6. Skipping concept needs to be explained in the User Guide.
pekkaklarck picture pekkaklarck on 16 Oct 2015

This is Nissar from Brocade QA team.
It's good to have the SKIP testcases count added to the report.
From the execution perspective, it gives clear report which shows the total passed, failed and skipped testcases.
Even in the case when none of the testcases are skipped, it indicates that all the testcases execution is covered, which would be useful for tracking the testcases execution coverage.

nshariff picture nshariff on 21 Oct 2015

Hi pekkaklarck, interested to know your thoughts & views on the comment provided by boakley above.
Personally i feel "skip execution if | True" will do all that one needs. One is going to have to do
something to determine dynamically if the current testcase should be skipped and so testing the
condition via the 'If' is all that is needed.

Doug8080 picture Doug8080 on 21 Oct 2015

On Wed, Oct 21, 2015 at 8:35 AM, Doug Gordon [email protected]
wrote:

... One is going to have to do
something to determine dynamically if the current testcase should be
skipped and so testing the
condition via the 'If' is all that is needed.

While it's true that some conditional may have to be done, it may not be
done in the same place you are calling skip. For example, I may do a
conditional check in a test case, and call a special keyword if I'm
supposed to skip the test case. This keyword may do several things: it may
close database connections, write to or close a file, log a message, etc.
AND it will do the skip. There's no need for that keyword to do some sort
of test (eg: "Skip if ..."), because the only way that keyword will be
called is if it has already been determined that the test should be
skipped. A second conditional is redundant.

Here is a somewhat contrived example:

*** Keywords ***
| Skip tests for unsupported feature
| | [Arguments] | ${feature}
| | log to console | Feature ${feature} is not supported; all tests for that feature will be skipped.
| | append to list | ${unsupported features} | ${feature}
| | remove values from list | ${supported features} | ${feature}
| | skip

| Check for supported feature
| | [Arguments] | ${feature}
| | Run keyword if | ${feature} not in ${supported features}
| | ... | Skip tests for unsupported feature | ${feature}

*** Test cases ***
| Example
| | [Setup] check for supported feature | multi-touch
| | <test body>
boakley picture boakley on 21 Oct 2015

Ah ok, thank you boakley I understand better now. Ok let me take this up and make the changes.

Doug8080 picture Doug8080 on 21 Oct 2015

+1
Would also be nice to be able to use tags to auto-skip testcases that cannot succeed. I currently use "pybot --exclude=manual" but would prefer "pybot --skip=manual" to be able to include them in the report without failing them.
That might deserve a separate ticket though

erikalfthan picture erikalfthan on 22 Oct 2015
👍1

Hi I'm dealing with the sudden death of my Father in-law at this time. I will try to address these points
ASAP, but must give my family my attention at this time.

regards,
Doug

Doug8080 picture Doug8080 on 26 Oct 2015

Hi boakley, committed changes to removed keyword "Skip Execution If' and added keyword 'Skip Execution'. See second commit.

pekkaklarck working on adding the --skip tag support now.

Regards,
Doug

Doug8080 picture Doug8080 on 5 Nov 2015

Hi, pekkaklarck I have pushed new changes to:

PR #2179

I have tried to address all your points and have gotten as far as I can given my understanding of
the source and the architecture of Robot Framework. I've updated my original comment on this
thread to reflect the functionality as is now (look for my comment under Added support for new keyword 'Skip Execution If' #2179)

Remaining Issues
I have this working 99.9%. There are two small issues remaining that I have not been able to address.

Issue 1:
How to fix up the file below for adding ROBOT_SKIP_ON_FAILURE support.

atest/robot/output/html_output_stats.py

Issue 2:
The --skip option works exactly like the --exclude option at present. I do not quite
understand how to show the skipped tests in the statistics when the --skip option is used. Skipped tests via the keyword Skip Execution are sown in the statistics, but I do not know how to show the tests skipped via the --skip option in the statistics.

Issue 3:
The final issue remaining that I have not been able to address is the case when a critical testcase is
skipped the console output is correct, however the HTML report is classifying a SKIP result as a critical failure (i.e. in the HTML report it states for example 'Status: 1 critical test failed'). I have
figured out (I think) that this needs to be fixed in the file in src/robot/htmldata/rebot/testdata.js
in the code block below. However I cannot figure out what exactly is in ‘stats’. My Jython is not good
– for now. This code block needs to not consider a SKIP result as a failure. Fix this and it will fix the
HTML reporting for critical testcases with SKIP results.

function suiteStats(stats) {
    return {
        total: stats[0],
        totalPassed: stats[1],
        totalFailed: stats[0] - stats[1],
        critical: stats[2],
        criticalPassed: stats[3],
        criticalFailed: stats[2] - stats[3]
    };
}

If you or anyone on the robot developers team can help me with this then I’ll do all I can to
incorporate this in this pull request. Alternatively if you would like to simply take my changes and address this issue in the team then I am more than happy with this. My interest is in getting this implemented and released to help my company and the wider user community here.

Regards,
Doug

Doug8080 picture Doug8080 on 6 Nov 2015

Any test suite should include a smoke test; By definition: IF smoke test Fails, there is no point to continue the test suite. In this case we may use the skip.

danieljtag picture danieljtag on 23 Nov 2015

Sorry for not paying too much attention for this issue lately. Been busy with RF 3.0 that I have promised for my client to be ready by the end of this year.

Big thanks for @Doug8080 for creating an actual pull request. Have others already tested it? Any comments? Unfortunately the big changes required by this feature and the aforementioned deadline make it impossible for me to even consider including it in RF 3.0.

If the general consensus is that this would be a good addition, we will have more time before RF 3.1. If you want to get more feedback, you may even want to create a separate release with this functionality and ask the wider community to test it. I have absolutely nothing against such forks, as long as they are not advertised as being official Robot Framework versions.

pekkaklarck picture pekkaklarck on 25 Nov 2015

I very quickly looked at PR #2179 and tested it on my machine. There were two major issues I wanted to point out and hear what others think about them:

  • Skip column is added to statistics even if no tests were skipped. I don't think that is a good idea unless skipping becomes a very widely used functionality.
  • Skipped tests themselves are totally removed from log and report. I think they should be visible so that it would be possible to see why they were skipped, especially if they are skipped dynamically.
pekkaklarck picture pekkaklarck on 25 Nov 2015

The skip functionality is not working fine. If we execute 2 testcases and skip 1st testcase then the logs for 2nd testcase is not getting displayed in html file. Please advise ASAP on same.

Execute a script as given below and check log.html, you can also see the issue
* Settings *
Documentation
... Objective: Test Skipped Testcase Reporting
...

Library BuiltIn

Default Tags smoke_test

* Keywords *
Skip Test
Set Test Variable ${release} R1
Run Keyword If '${release}' == 'R1'
... Pass Execution Skipping Testcase (it is not relevant to this image)

* Test Cases *
Test_Skip
# Check if Testcase is Relevant (should be skipped)
# -------------------------------------------------
Skip Test

Log  ===========> *** RUNNING TEST_SKIP *** <===========

Test_DSkip
# Check if Testcase is Relevant (should be skipped)
# -------------------------------------------------
#Skip Test

Log  ===========> *** RUNNING TEST_DSKIP *** <===========
jbkrishna picture jbkrishna on 9 Dec 2015

I find that the skip feature can be very useful in some situations. I my case, I would like to do a check in a data structure to see if my test can run in a specified testbed (with physical equipment), and if it is not suitable, just skip the suite. Today our approach is to tag every test suite with the name of the testbeds the test can run against.

It is not clear to me from the PR if that use case is covered. Can @pekkaklarck or @Doug8080 clarify if it will be possible to skip an entire test suite by a check in the suite setup keyword?

vkruoso picture vkruoso on 28 Dec 2015

The main reason we want a Skip feature is because our software greatly integrates with other third party environments, which can be flaky at times.

So, let's say every night I want to run a suite of tests (--includes=CriticalANDFeature). Some of the Critical tests rely on the fact that in our virtualized infrastructure, certain VMs are both online and are highly responsive. What I want to do is (during Suite Setup) attempt to connect to the virtualized infrastructure. If this connection attempt fails, then I know in advance that all of my tests for that suite will fail. I don't want to have tests marked as Failed because of infrastructure issues. A failed test should be a failure in the application under test.

In this case, a Skip indicates an environmental issue that blocked the test from being able to be properly executed at runtime. It means the Dev/Ops and SysAdmin teams will have a ticket created the next day, rather than Product Developers getting a bug assigned to them.

bdfariello picture bdfariello on 5 Jan 2016

Hi vkruoso, it would be possible to skip entire test suites with this functionality.

Doug8080 picture Doug8080 on 7 Jan 2016

I do have the opposite need: skipping a specific keyword easily. We do have a keyword that's used in multiple test cases, but that needs, sometimes, to be disabled, because the functionality is not available at the time. Being able to just skip that, by doing a --skip tag on the command line would be awesome!

Xaelias picture Xaelias on 19 Jan 2016

In this case, can't your keyword simply check for a condition and then do
nothing if that condition is true?

On Tue, Jan 19, 2016 at 12:44 PM, Xaelias [email protected] wrote:

I do have the opposite need: skipping a specific keyword easily. We do
have a keyword that's used in multiple test cases, but that needs,
sometimes, to be disabled, because the functionality is not available at
the time. Being able to just skip that, by doing a --skip tag on the
command line would be awesome!

—
Reply to this email directly or view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-172946837
.

boakley picture boakley on 19 Jan 2016

I can, and that's what I'm currently doing, but it implies creating a new variable, that I need to pass, and check on. Plus, the visual on this is far less explicit than a skip step would be.
And it also makes more sense on the command line IMO.

Just saying, if that happened, I would be happy ^^
Anyway, thanks for implementing this, I'm sure we will need it at some point too!

Xaelias picture Xaelias on 19 Jan 2016

In PR #2179 @snyderr showed interest to work with this issue. We agree to discuss about problems in PR #2179 and this whole issue in general on Slack tomorrow (Wed 20 January) at 8pm UTC. Not sure yet will we chat on the #general channel or create a new channel for this purpose, but I definitely want the discussion be public. Join us if you are interested!

pekkaklarck picture pekkaklarck on 19 Jan 2016

Hello everyone. Since we last talked I have been working on the old pull request (https://github.com/robotframework/robotframework/pull/2179) and have moved the code for skip execution into RFW 3.0.1. while also removing the --skip feature for now. If anyone has time to look over my changes I would appreciate any input that you may have. https://github.com/snyderr/robotframework. Feel free to post any comments/concerns in the Robotframework Slack in the #devel channel that Pekka created for this topic.

snyderr picture snyderr on 2 Feb 2016

Great to see this moving forward. Thank you snyderr for taking this on. I got as far as I could take it.
Very much appreciate your time and effort on this.

Regards,
Doug

Doug8080 picture Doug8080 on 3 Feb 2016

Here is the updated pull request.

snyderr picture snyderr on 18 Feb 2016

Hello.

I would like to add my comment regarding skip execution keyword. We are doing E2E TA for very complex telecommunication OSS system at Nokia, that interacts with many so called network elements. So in general our system under tests has functionalities that are implemented and all are testable, but it is interacting with network elements that can or cannot support certain features.

We would like to have general test suites with all possible test cases included. And it would be great to just skip execution of those test cases that are not relevant for specific network element.
I hope that I was able to give you general overview of our RF use case.

In general we want to skip those test cases that will be not relevant because of some external conditions in our system under test.

What are the current plans for inclusion pull request from snyderr in RF?

BR, Krzysztof

ksokolowski picture ksokolowski on 4 Apr 2016
👍1

Hi @ksokolowski,

Thank you for your input on use cases for this keyword. If you have any time to take a look at my branch I would appreciate any feedback that you may have. Feel free to contact me on the Robot Framework slack with any questions you may have.

Ryan

snyderr picture snyderr on 5 Apr 2016

Is there an ETA when this features will be available?

bkhouri picture bkhouri on 16 May 2016
👍1

What is pending here? I'm waiting for this feature for very long time, it will be grate if we merge this asap.

mkumatag picture mkumatag on 6 Jun 2016
👍1

yes, that feature is also something we would like to have.

zhehaoshi picture zhehaoshi on 7 Jun 2016

@pekkaklarck / @snyderr : Any news / what is the status about this feature I also find interesting ?
Thx for feedback.

jcdevil picture jcdevil on 18 Aug 2016

@pekkaklarck Looking forward to this feature. Any progress?

jontaydev picture jontaydev on 21 Sep 2016

I'd love to have this - being able to skip tests based on github issue status would be superb.
I guess this could be done currently if every single test had a tag and a github query was run prior to launch to generate an exclude list.

hickeng picture hickeng on 7 Oct 2016

Sorry for not keeping you updated with the status of this issue. I still think this would be a nice feature, but due to the following reasons it's unlikely to be implemented in the near future:

  1. The main problem is that I'm still not sure how skipping should work with related functionalities to mark tests non-critical tests and to exclude them from execution altogether. We've been discussing about this earlier, but I'm not entirely happy about any of the proposed approaches. Adding a new status along with the current binary PASS and FAIL also requires thinking. Most importantly, I want features to work together consistently and don't want add unnecessary complexity.
  2. Once we know how this functionality should work, implementing it requires so big changes that they are only possible in a major release. I'm currently with RF 3.0.1 but even that's pretty slow because of my other work. With my full calendar I cannot realistically think about RF 3.1 this year or even early 2017.
  3. Related to the above, nobody pays me to work with this feature. Robot Framework Foundation sponsors the development, but they don't have that much money and that money is targeted for bug fixes and reviewing pull requests. There have been PRs implementing this functionality, but I'm not sure is their overall design good enough (see the point 1. above).

I hope that the Foundation would be able to collect a bit more money for 2017 (being able accept members outside Finland would help but that would require lot of paper work first) and could start spending it for implementing bigger enhancements like this. In that case I could concentrate on RF development full time and development would be more active.

It would obviously be possible for someone to sponsor development of this particular feature. If you are interested, please contact me privately. Help developing this is highly appreciated too, but I'm afraid my time is required at least with the overall design and probably also with some of the internal changes.

pekkaklarck picture pekkaklarck on 7 Oct 2016

Just as a reminder, @snyderr has a pull request for a solution in #2317 it's just awaiting review and merge. The solutions to the major problems discussed here were decided on in a Slack meeting with @pekkaklarck about 8 months ago.

fkberthold picture fkberthold on 28 Oct 2016

Also would like to see Skip added. The existing PR seems good enough to me (but I'm no expert)

garyo picture garyo on 27 Jan 2017

Please can we see this issue moving forward. The request for the SKIP support is over 2 years old now. Users would really like (and want) the SKIP feature. It would make life in our Regression shop much much easier. The consensus of all the commentators is that the PR is good to go.

Doug8080 picture Doug8080 on 2 Mar 2017

Unfortunately there is still no consensus on how this should be implemented. The main open question is what to do with the current criticality concept if we introduce skipping. My strong opinion is that having non-critical tests and skipped tests doesn't make sense.

pekkaklarck picture pekkaklarck on 2 Mar 2017

Hello Pekka.

I don't see any contradiction when skipping critical tests. If test is not
applicable, because of some prerequisite according to you we should change
the tag of test to NON-CRITICAL and FAIL it. But why we should we have FAIL
status in the log if we haven't really tested anything and just aborted
test execution?
So from my practical experience skipping tests makes sense.

BR, Krzysztof

2017-03-02 11:04 GMT+01:00 Pekka Klärck notifications@github.com:

Unfortunately there is still no consensus on how this should be
implemented. The main open question is what to do with the current
criticality concept if we introduce skipping. My strong opinion is that
having non-critical tests and skipped tests doesn't make sense.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-283610475,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJPiO5YyK3LDsmqFlSIT53_M_MK3TWpRks5rhpQhgaJpZM4FmCsr
.

ksokolowski picture ksokolowski on 2 Mar 2017
👍1

NON-CRITIAL and CRITICAL tests classify testcases according to how important they are. SKIP means that for the given image under test at the time (according to the test) the test is not applicable to run/execute on the current image.

For example lets say you have a testcase that tests feature X, if the given built image does not include feature X (feature X in not in the branch) the the testcase should SKIP because its not a failure that the current image (branch) does not support feature X.

Have SKIP support means we do not have to constantly update Tags and manage them per release etc. It
also makes the job of trasing reports and logs much easier for Regression shops because its clear that the testcases are SKIP not FAIL.

Doug8080 picture Doug8080 on 2 Mar 2017
👍1

I think skip is more important than critical, and probably should have been implemented first. In my scenario, a test is only critical to a feature if the feature exists in the first place. But if the feature does not exist for whatever reason, it by definition cannot be critical. On the other hand, non-critical tests don't make sense for my scenario either. If a feature exists, then it should always work. I don't see a problem skipping either a critical or non-critical test if the test logic is explicitly designed the way.

The bigger problem is that now, we are in a situation where developers don't take the report seriously. Red no longer means "Attention! Something is wrong!" Now it means, "eh, it's probably a skipped test. I won't even bother checking through thousands of tests to find out."

IMHO, a low-priority feature (from my perspective) should not be holding back a high priority feature. Skip > critical.

jontaydev picture jontaydev on 2 Mar 2017
👍1

I like how @Doug8080 put it. Critical defines the test priority, while Skip defines the test applicability. So it follows that priority is a subset of applicability (priority is ignored if not applicable).

@pekkaklarck, based on this thread alone, there is overwhelming support for skip > critical. Is this being disputed elsewhere?

jontaydev picture jontaydev on 2 Mar 2017

I agree with @Doug8080 , SKIP is just another status, equivalent to PASS and FAIL. It should not impact the test's "severity" (ie: critical or non-critical). So if a test is marked as SKIP, the respective test's severity/critical-ity should not be impacted.

I would really like to see this as well.

bkhouri picture bkhouri on 2 Mar 2017

First of all, I would also like to see the skip status introduces. My problem with adding it if we also have critical and non-critical tests is that in effect we would have these six different test outcomes:

  • Critical PASS
  • Non-critical PASS
  • Critical FAIL
  • Non-critical FAIL
  • Critical SKIP
  • Non-critical SKIP

Many of these are overlapping (e.g. non-critical fail, any skip) and it would be hard to explain the difference between them. I value simplicity and would rather just have:

  • PASS
  • FAIL
  • SKIP

Having just these three test outcomes would also simplify logs and reports, frameworks internal implementation. We must, however, think about backwards compatibility, and cannot just blindly remove the whole criticality functionality. The main use case for criticality is being able to run tests that aren't ready yet (either tests are incomplete or the functionality is not ready) without them failing the whole test run. This is a critical feature when using the ATDD/BDD approach.

To summarize:

  • I'd like the skip status to be added
  • I want criticality to go if it is added
  • Possibility to include not-ready tests in test runs must be preserved
pekkaklarck picture pekkaklarck on 3 Mar 2017

My proposal for implementing the skip status, taking the requirements in my previous comment into account, is as follows:

  1. Add the new SKIP status to tests in addition to the existing PASS and FAIL.

  2. Don't separate tests to critical and non-critical tests anymore. Remove information related to that from logs and reports.

  3. Add new command line option, e.g. --SkipOnFailure, that can be used to tell that tests with certain tags should be automatically considered skipped if they fail. This would allow including tests that are not ready yet (or functionality is not ready yet) to be included in tests runs. As I commented earlier, this is a critical functionality when using the ATDD/BDD approach.

  4. Change the current --critical and --noncritical so that they provide the same functionality as the --SkipOnFailure. This makes the removal of the criticality functionality mostly backwards compatible. These options can then later be deprecated and removed.

I've proposed this on Slack earlier and was surprised that there were mainly negative responses. It is interesting to see what comments we get there.

pekkaklarck picture pekkaklarck on 3 Mar 2017

Hi pekkaklarck I appreciate the balance you have to play. I am happy with what you are proposing as the way forward. In the worst case (for Robot implementation) you could offer a --noskipping command-line option that is present would revert back to current Critical & NonCritical functionality with no SKIP support (just how it is today). If --noskipping is not present on the command-line you proceed as you have stated above.

Personally I could not offer the option (avoiding the obvious maintenance issues) and just proceed as you have suggested.

Doug8080 picture Doug8080 on 3 Mar 2017

The problem with making this functionality configurable is that we needed to keep code for both feature sets, needed to document both, etc. In practice that would just increase complexity and be a maintenance nightmare.

Instead I've been thinking about implementing what I proposed above in a branch to see how it works in practice. Could also create a preview release based on the branch for others to easily test it. Based on the feedback we could then decide should the branch be merged or deleted.

pekkaklarck picture pekkaklarck on 3 Mar 2017

I would be more than happy to have my teams test a branch with the SKIP implemented as you have proposed and provide you with feedback.

Doug8080 picture Doug8080 on 3 Mar 2017

Pekka, I think a lot of the push-back that this suggestion received on
Slack was because there is a fundamental difference between a test that is
not run at all, i.e. one that is Skipped, and one that runs but Fails due
to a product issue.

Now, if the --SkipOnFailure became something along the lines of
--SkipOnSetupFailure I think THAT would be an attractive alternative. This
would allow tests to be designed where all prerequisites are checked during
Suite/Test Setup, and so any failure during test runtime would be a
legitimate failure. It would be even better if it handled failures during
both Setup and Teardown steps, though calling it
--SkipOnSetupOrTeardownFailure is getting a bit wordy.

On Fri, Mar 3, 2017 at 9:03 AM, Pekka Klärck notifications@github.com
wrote:

The problem with making this functionality configurable is that we needed
to keep code for both feature sets, needed to document both, etc. In
practice that would just increase complexity and be a maintenance nightmare.

Instead I've been thinking about implementing what I proposed above in a
branch to see how it works in practice. Could also create a preview release
based on the branch for others to easily test it. Based on the feedback we
could then decide should the branch be merged or deleted.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-283960631,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIaBxwX63KupClYMaSnG8I9xAX90WbLVks5riB2egaJpZM4FmCsr
.

bdfariello picture bdfariello on 3 Mar 2017

Yes, most of the negative feedback was about "automatically" turning a failing test to skipped test. I wrote "automatically" in quotes on purpose as it isn't really automatic, it would occur _only_ if you use a certain command line option and add a matching tag. Teams considering that a bad idea could simply decide not using this functionality. Also, this isn't much different to the current functionality to turn a critical test into a non-critical test.

I appreciate your --SkipOnSetupFailure idea, but unfortunately it would not be backwards compatible. Requiring teams that use ATDD/BDD to add setups to their tests in order to migrate to the latest RF versions wouldn't be nice, and in general the need to use setups just for this purpose sounds a little odd.

pekkaklarck picture pekkaklarck on 3 Mar 2017

Personally, I've seen a use case were they use critical and non-critical test in a way to control "expected" results. For example, if a system expected 10 entries, but there are 8 due to a product issue, a test would be created with "Expected 10 Entries" (which would be marked as non-critical, and a "Actual is 8" test would be marked as critical. This way, the results shows a PASS even though there are known issues.

As a result, there may be some value in keeping the 'critical' and 'non-critical' even with the SKIP feature. But go with the majority.

bkhouri picture bkhouri on 3 Mar 2017

I'll chime in one last time, to add to the chorus of those asking for SKIP
in addition to PASS/FAIL. Lacking the ability to skip a test based on
runtime conditions is one of robot's biggest weaknesses, in my opinion.

I also think it's a bad idea to get rid of criticality. This would be a
potentially major backward incompatibility, causing grief without really
providing any benefit (that I can see). Maybe I'm wrong, and maybe nobody
uses that feature. Personally I like the criticality feature, it makes it
real simple to keep an eye on troublesome tests without failing the build.
Every team I've worked on that used robot has taken advantage of that
feature. To me, that's a selling point of robot rather than a defect that
needs to be removed. I don't use it often, but when I do use it I am
thankful it is there.

That being said, if I had to choose between SKIP and non-critical, I would
pick SKIP. I just don't understand why there has to be a choice. I am
unable to see the problem with mixing non-critical and SKIP. I've been
trying hard to understand it, but I just don't. I'm still hoping to be
convinced. It's an edge case that will likely never happen in the wild.
When it does it will probably prove to be useful for that particular
situation. To me, SKIP and criticality are completely orthogonal, and each
provides real value.

On a more pragmatic side, simply introducing SKIP as an additional
disposition seems much easier than doing the major surgery of removing the
old feature and adding the new, along with all the documentation required
for the new --skiponfailure or --skiponsetupfailure or whatever else is
proposed. I could be wrong about that, because I'm not overly familiar with
robot internals. I think Pekka once said he would like to simply treat a
non-critical failure as a SKIP, and tweak the report to display them that
way. If so, maybe that's the quickest path, but that comes at the expense
of backward compatibility. I think if it were implemented in such a way
that I could use one feature or the other but not both, that would be a
major fail.

It seems to me that, given the size of the core development team, a simpler
and backwards-compatibile solution is better than a more complex
backwards-incompatible solution. If it proves to be the case that having
both non-critical and SKIP is confusing, non-critical support could be
removed in a future release after a suitable waiting period where it is
documented as deprecated.

Given that there is already a pull request that supports both, I think it
would be better to roll it in than throw it away.

On Fri, Mar 3, 2017 at 8:06 AM, bdfariello notifications@github.com wrote:

Pekka, I think a lot of the push-back that this suggestion received on
Slack was because there is a fundamental difference between a test that is
not run at all, i.e. one that is Skipped, and one that runs but Fails due
to a product issue.

Now, if the --SkipOnFailure became something along the lines of
--SkipOnSetupFailure I think THAT would be an attractive alternative. This
would allow tests to be designed where all prerequisites are checked during
Suite/Test Setup, and so any failure during test runtime would be a
legitimate failure. It would be even better if it handled failures during
both Setup and Teardown steps, though calling it
--SkipOnSetupOrTeardownFailure is getting a bit wordy.

On Fri, Mar 3, 2017 at 9:03 AM, Pekka Klärck notifications@github.com
wrote:

The problem with making this functionality configurable is that we needed
to keep code for both feature sets, needed to document both, etc. In
practice that would just increase complexity and be a maintenance
nightmare.

Instead I've been thinking about implementing what I proposed above in a
branch to see how it works in practice. Could also create a preview
release
based on the branch for others to easily test it. Based on the feedback
we
could then decide should the branch be merged or deleted.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
issues/2087#issuecomment-283960631>,
or mute the thread
AIaBxwX63KupClYMaSnG8I9xAX90WbLVks5riB2egaJpZM4FmCsr>

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-283961461,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABEmYlK1GvFj5FuzuA58usmosfEp-EGqks5riB54gaJpZM4FmCsr
.

boakley picture boakley on 3 Mar 2017

We are eagerly waiting for this feature to roll in. I see the discussion started in 2015. Most of the framework support quick methods to SKIP unwanted tests/suites on conditions defined by testers environments.
Also, If I haven't missed the crux, I don't know how skip tags would be helpful if we already have excludes.

For me; currently passing the execution which I want to SKIP due to unwanted conditions are making product developer happy thinking that we have huge coverage and if intend to fail the same it crops a huge burden on test engineers to regroup test in suite which we run at end with low-priority status.

I also support the statistic/consensus of SKIP test to be displayed in reports because it give a fair idea on the test quality of the release.

I really hope I see this feature in 303 or a fork.

mungayree picture mungayree on 4 Mar 2017

>

I see the discussion started in 2015.

More like 2011. See
https://github.com/robotframework/robotframework/issues/812

Also, If I haven't missed the crux, I don't know how skip tags would be

helpful if we already have excludes.

For some people it will be completely useless. For others, it may be
incredibly valuable. If excluding tests by tag from the command line works
for you, there's not a lot of new functionality to be gained by using the
SKIP feature.

For large suites, there can be times when you don't know until the test is
running whether or not to skip a block of tests. For me, if I determine at
runtime that a set of tests are going to fail then I would rather simply
not run them. For example, imagine that you have tests that rely on a
geolocation server for some subset of tests, and you detect that the
geolocation server is down, or maybe it goes down while the test is
running. If you run the tests it might add many minutes to the overall test
execution as each test or keyword waits for the service to time out. With
the SKIP feature, we could immediately skip the test and not have it show
up in the report as a PASS or FAIL, since neither one tells the actual
truth.

On Fri, Mar 3, 2017 at 6:18 PM, mungayreelal notifications@github.com
wrote:

We are eagerly waiting for this feature to roll in. I see the discussion
started in 2015. Most of the framework support quick methods to SKIP
unwanted tests/suites on conditions defined by testers environments.
Also, If I haven't missed the crux, I don't know how skip tags would be
helpful if we already have excludes.

For me; currently passing the execution which I want to SKIP due to
unwanted conditions are making product developer happy thinking that we
have huge coverage and if intend to fail the same it crops a huge burden on
test engineers to regroup test in suite with run at end with low-priority
status.

I also support the statistic/consensus of SKIP test to be displayed in
reports because it give a fair idea on the test quality of the release.

I really hope I see this feature in 303 or a fork.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-284106511,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABEmYk6yXKz7ZNh6Bvw_fhHDmXJi-w4pks5riK3FgaJpZM4FmCsr
.

boakley picture boakley on 4 Mar 2017

Four uses for this have been discussed for SKIP:

  1. Test can't run due to a runtime environment issue (often detected during suite setup and applied to all tests in the suite).
  2. Test is inappropriate for this branch (e.g., tests a feature not available on this branch).
  3. Test is inappropriate based on runtime settings (e.g., a variable that determines run enviornment).
  4. Test is in development, so failure of test should not fail the test run.

RF presently has no way to address item 1; so that is the key reason I support adding SKIP.

Item 2 can often be simulated with tags and --exclude, but that's inferior to a SKIP status because dynamic detection of feature availability doesn't work, and because the fact the test was skipped may be important for cross-version statistics comparison (--exclude behaves as if the test wasn't present at all whereas SKIP contributes to metrics). So SKIP improves this use case.

Item 3 can be worked around by using tags and --exclude and documenting in your test procedure that if you run with "-v environment:foo" that you also have to add "--exclude skip_if_environment_foo". However, that makes use of RF clunkier so adding SKIP would improve usability.

Item 4 is interesting. I'd argue the correct way to implement item 4 varies based on team dynamics. Some teams would find critical/non-critical a better semantic fit because these are failures rather than skips and they are part of the failure backlog to fix. My team's QA group had bad management for a while that felt that it was ok to set a percentage pass rate that allowed the product to ship. Due to that history, I oppose any practice that equates "fail" with "ok to ship" for my team and have thus chosen not to use the critical/non-critical feature for that reason. I worked around the lack of SKIP status for this by creating a "Needs Work" keyword that does an ERROR level log and calls Pass Execution. That messes up our "pass" statistics a bit but for my team I think it's a better semantic than non-critical failure. SKIP would fit my goals for this use case, although I recognize that is not a universal opinion.

My conclusions:

  • I strongly support the addition of SKIP
  • the key requirement for SKIP is the ability to dynamically skip a test or suite of tests based on code in suite/test setup that detects an environmental problem. An exception that triggers skip status would suffice, but a keyword to skip all remaining tests in the suite would also be convenient.
  • I am fine with removal of critical/non-critical, but I am also fine with keeping critical/non-critical and having SKIP > criticality (criticality is ignored if test is skipped). The former may mean less code in RF core, while the later may better accommodate human team preference variations. This is a tradeoff with no clear right answer so we won't reach consensus. The only wrong decision, IMHO, is to delay the SKIP feature while waiting for a decision that's not amenable to consensus.
cjnewman picture cjnewman on 4 Mar 2017

Some notes about this topic (mostly answering to this comment from Pekka)

  • having a way to use RED colour (FAILED status) only for real failure would be a great improvement. With the “non critical” concept, we can handle known issues, but they are too much mixed with the unexpected failures. So having a SKIP status would be nice. It should have a new colour in the report (blue?).

  • we use a lot the concept of "critical / non critical” though the wording is not great because we don’t use it to state the importance (priority) of the tests, but that’s the way we manage our known issues (if it is set as non critical, we expect it to fail). So replacing this concept by a clearer one would be good for us ("known issues are skipped when they fail")

  • when we have non critical tests (mostly known issues) we keep an eye of them so we want them to be executed, so that if the test is passing, it can mean a bug was fixed and we did not notice yet. If our understanding is correct, then we could use the —SkipOnFailure argument for that. Test would be run and “skipped” (as a status) only when failed.

laurentbristiel picture laurentbristiel on 5 Apr 2017

+1 - is there any progress on this issue / what's the current status?

We would also love this feature as explained multiple times.

edvinkuric picture edvinkuric on 27 Aug 2017

My plan is to prototype the approach I proposed earlier when we can start RF 3.1 development. The problem is that I only have limited time for Robot Framework related development and I want SeleniumLibrary with Python 3 support out first, need to look at SSHLibrary's Python 3 support too, and it seems RF 3.0.3 is also needed before we can start with 3.1. I'd be disappointed if I couldn't at least start this year, but I don't want to make any concrete promises about it. If this is urgent, you can consider sponsoring the development to allow me to fully concentrate on coding.

pekkaklarck picture pekkaklarck on 28 Aug 2017
👍2

I would like to be able to indicate a skipped test inside the test reports with a different color (result status), so let me share some thoughts on why this would be beneficial.

What many people don't realize about the reports and how they are used, is that they are often viewed by non technical people (i.e. management).
From my experience, non technical people view the two colors (for "passed" and "failed") in the following manner:
Green - Everything's a-ok, we are going according to the plan, relax.
Red - Something needs to be done - let's ask testers what's going on.

Explaining why something failed is often the most time consuming part of presenting the results of your work. There are often times where the test automation team hits a wall because of a bug in the software, for which there is a workaround - we have to move forward with our testing, and for that we need to apply the workaround on the bug.

For sake of the example, let's say there is a feature A, and in order to test it a code of the feature B needs to be executed, but that code has a bug X for which we would need to apply a workaround.
Either we make the test of feature A fail, which leads to all sorts of situations where we waste our time explaining what went wrong, and that the feature under test actually works, or we make the test pass - which makes the visibility of bug X in the usecase of A null. Testing the feature B is out of the scope of our work.

A proper tests architecture should alleviate the need for this - yes. But the sad reality is, that it isn't always possible to have it when you are testing hardware components.

A third color would indicate the following:
The testcase code is working properly.
The feature it is concerned about works properly.
The prerequisites for the feature aren't fulfilled, the investigation is ongoing, a workaround has been applied.

MrMino picture MrMino on 24 Oct 2017
👍2

Hi MrMino, I fully agree. If you refer at my comment on "Doug8080 referenced this issue on Oct 16, 2015" referring to my original work on this, you will see that a new color for SKIP was exactly what I had envisioned.

Doug8080 picture Doug8080 on 24 Oct 2017
👍2

We had a need for this feature to easily skip tests during development.
Instead of raising a failure whenever we wanted to skip a test, I implemented a short workaround so the tests just 'pass', here's the code for anyone with similar requirements:

from robot.api import logger
from robot.errors import PassExecution

class CustomLibrary():
    def _is_true(self, condition):
        if isinstance(condition, str):
            condition = self.evaluate(condition, modules='os,sys')
        return bool(condition)
    def skip_execution_if(self, condition):
        if self._is_true(condition):
            logger.info('Execution skipped')
            raise PassExecution('Execution skipped')

can now be used like this:

*** Variables ***
${start_at_step}            701

*** Test Cases ***
700 Example Test Case
    skip execution if           ${${start_at_step} > 700}
    some keywords
mhofer117 picture mhofer117 on 13 Dec 2017
👍2

I've just been looking for functionality like this. My reason for wanting to skip tests is one mentioned by @cjnewman, I have an unstable test environment and failure of one test means others will inevitably fail since the functionality they test is inacessible.

The main thing I'm looking for is a way to see what the actual problem was; if the main page shows a big red "SYSTEM DOWN NOTHING WORKS OMG" I want that test to be the one that fails and the thirty tests for functionality that's inaccessible to have another status.

I'm probably going to add a Fail with some informative message to each test (or suite) setup for now. A Pass would probably be closer to doing what I want but that just feels wrong :)

lennartq picture lennartq on 15 Feb 2018
👍1

Hi, need this feature to skip some test in data driven test suite based on a global variable.
Example:
need to test registration with TV and WEBSITE.
Some test are for TV and other for WEBSITE, and I used the same data driven robot suite.

I have to skip some test because TV have different key layout and cannot write some non unicode characters, but I want to maintein only one data driven file for both system.

Bye!

acaovilla picture acaovilla on 30 Oct 2018

Any progress on that? This feature is common in frameworks I used to work so far (JUnit, TestNG, even for end-to-end tests) and I am really missing it in Robot.

dzieciou picture dzieciou on 22 Nov 2018
👍2

Maybe a way to get this through is implementing this function in small steps. I opened a PR which only implements highlighting on the console:

https://github.com/robotframework/robotframework/pull/3018

windelbouwman picture windelbouwman on 1 Dec 2018

Other small steps I could foresee:

  • Add skip status as an outcome to tests (next to pass and fail). Leave criticality in place. Add skip column in the report.
  • Add skip keyword and exceptions to at runtime be able to skip tests.
  • Add skip by tag from command line. This really is an add on, and maybe duplicate with --exclude?
windelbouwman picture windelbouwman on 1 Dec 2018

Is there a timeline for this feature? We are evaluating frameworks for test automation on my team and a SKIP status for tests is a major want for us.

selenareneephillips picture selenareneephillips on 7 Dec 2018

Good question! I think the skip feature lacks a proper design, that is why it is taking this long. My desired approach would be tiny steps, just add skip status and see how it works out. So no doing no design, but more trial and error approach. I have the feeling that trying to make a proper design will take too much time.

windelbouwman picture windelbouwman on 7 Dec 2018

@windelbouwman is right, the first thing to do is agreeing on design. Our current criticality functionality is similar to skipping, and I don't personally like the idea we'd add skipping in addition to it. Instead I'd like to replace criticality altogether with skipping, but there have been pretty vocal opponents. I understand the criticism, but still think that's the best idea. This is obviously something we can discuss more here or in our Slack. If we can actually agree on the design, this would be a great feature for RF 3.2 that's initially targeted for H1/2019.

pekkaklarck picture pekkaklarck on 7 Dec 2018

So, for me a tipical scenario is: I have a bunch of user for Facebook test.
Test case: do sign in using Facebook method, and erase user with API. The
user remain in the system with status 'pending' for 1 day. At the setup, I
check if I have a free user to register it. And if I have no user? Need to
skip the test case for no user avaible :). Another example: sometimes test
condition depend from external services, at the setup I call the service to
verify that is up and running.. and if it is down ? Need to skip that test
case, for no precondition :). Ps, sorry for my Tarzan-like English.

Il giorno ven 7 dic 2018, 23:28 Pekka Klärck notifications@github.com ha
scritto:

@windelbouwman https://github.com/windelbouwman is right, the first
thing to do is agreeing on design. Our current criticality functionality is
similar to skipping, and I don't personally like the idea we'd add skipping
in addition to it. Instead I'd like to replace criticality altogether with
skipping, but there have been pretty vocal opponents. I understand the
criticism, but still think that's the best idea. This is obviously
something we can discuss more here or in our Slack. If we can actually
agree on the design, this would be a great feature for RF 3.2 that's
initially targeted for H1/2019.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-445385215,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aphlx6PWD1rVxJWquORFB5kjGgEoWSb1ks5u2utwgaJpZM4FmCsr
.

acaovilla picture acaovilla on 7 Dec 2018

So -- I have a question about the critical / non-critical terminology. Why can't you have both? This really isn't a 'status'. In my history as a QA engineer, there are typically three terms associated with a test:

Priority: Is this test run for every code check-in? Is this test case run daily? Only at the end of the release? Only when specific things are changed such as when dependencies like third-party libraries or databases are upgraded. It is not related to the outcome of the test execution.

Severity: This _is_ related to test execution status and it is usually specifically related to FAIL/SKIP execution status in my experience. It is a measure of how bad it is for a test to fail or be skipped. The test in question might be a low priority test case because it needs to be run infrequently such as when dependency upgrades happen. A failure or skip is high-severity because the failure for the test to execute with PASS status means one of two things:

  • If the test failed, the application unusable for the customer or perhaps a failure of the test case is related to important security precautions that must be verified before release.
  • If the test was skipped, you cannot release because you don't have the prerequisite confidence in application functionality.

Status: This is the outcome of the test. Typically we have: PASS / FAIL / SKIP. A test that was skipped was not run because preconditions were not met. A test case that passed did execute because all preconditions (if any) were met and all of the test's assertions met expected checks. A test case that failed executed because all preconditions for execution (if any) were met, but one or more assertions in that test.

So, it doesn't seem to me that criticality is incompatible with 'SKIP'. They can compliment one another.

selenareneephillips picture selenareneephillips on 8 Dec 2018

Where is the design located / created? Is this in a wiki page? In my opinion, a wiki-like place would be best to capture all use cases and the design. This issue and all its comments contains a lot of ideas, but it is not that simple to read all comments and get the consensus.

windelbouwman picture windelbouwman on 8 Dec 2018

So far design discussion has happened here in the issue tracker or in Slack (and earlier IRC). In this particular case that may not be adequate, and a wiki or similar might work better. Does someone have experience from such design forums from other projects? Anyone interested to set up wiki or something else? This GitHub project has a wiki so it would be easy to try it out.

pekkaklarck picture pekkaklarck on 8 Dec 2018

I agree with @selenareneephillips that critical marked tests are not the same as skipped tests. Skip is a test status, like pass and fail. Critical is more a test category / severity.

This is my attempt to summarize the above discussion:

  • Consensus: it would be good to add skip status
  • No consensus: skipping is the same as failing a non-critical test

Is it an idea to first add skip status as an outcome, and leave the criticality feature in place? We can then later deprecate the criticality. I am aware that this might be tricky, since there is no such thing as a temporary solution, but it might be a pragmatic way to go forward.

windelbouwman picture windelbouwman on 8 Dec 2018

Setting up a wiki page for this in this github repository might be a good idea.

windelbouwman picture windelbouwman on 8 Dec 2018

It's true that criticality isn't really a status, but on the other hand it affects the outcome of test execution. Currently when a test is run, the result can be one of these:

  1. PASS, Critical
  2. FAIL, Critical
  3. PASS, Non-critical
  4. FAIL, Non-critical

Understanding what 1 and 2 mean is easy and 4 is effectively the same as SKIP, but 3 is pretty useless state.

If we add SKIP status to the mix, a test can have any of these results:

  1. PASS, Critical
  2. FAIL, Critical
  3. SKIP, Critical
  4. PASS, Non-critical
  5. FAIL, Non-critical
  6. SKIP, Non-critical

Now the problem is that 3, 5 and 6 are effectively same. They all mean that test was picked for execution, but either not run at all or execution was terminated but we don't need to care. It would also be hard to explain what's the difference between critical and non-critical SKIP (3 and 6) and we'd still have useless non-critical PASS (4). Trying to document how all this works and explaining when to use different features would be complicated.

If the implementation is hard to explain, it's a bad idea. (Zen of Python)

pekkaklarck picture pekkaklarck on 8 Dec 2018

In my opinion critical-skip is not the same as non-critical-fail.

Critical-skip would mean, that during runtime the test was not applicable for some reasons determined half way the tests. This could for example be, reading a parameter from the system under test, and detecting that part of the system was disabled. The test could have been de-selected by a tagging system, but this is not always possible beforehand.

Non-critical-fail would mean that the test actually failed, but the test is not super reliable or a work in progress. Also, this cannot be used as a skip/not applicable status since people (managers / non technical people) will see red colors in the report, and this will set of alarms and subsequently require explanation.

windelbouwman picture windelbouwman on 8 Dec 2018

My proposal is that we should add the SKIP status, but at the same time remove criticality as an outcome. After that a test result is simply one of these:

  1. PASS
  2. FAIL
  3. SKIP

We should, however, support the current --critical/--noncritical command line options for backwards compatibility reasons and also because the functionality that they provide is actually pretty useful when doing ATDD/BDD. My proposal is that we should change the behavior so that if a test is marked non-critical using the aforementioned options, it would get SKIP status instead of FAIL. For me SKIP and non-critical FAIL are effectively same and this change would not be dramatic, but others have been against automatically skipping tests like this.

pekkaklarck picture pekkaklarck on 8 Dec 2018

@windelbouwman wrote:

In my opinion critical-skip is not the same as non-critical-fail.

There are semantic differences, but I don't think they are important enough to warrant having altogether six possible test results. Having just PASS, FAIL, and SKIP and using tags for more detailed classification is a lot simpler and also more flexible.

pekkaklarck picture pekkaklarck on 8 Dec 2018

I did not use yet the critical/non-critical feature, but to me it appears that critical tests determine the overall pass/fail status. If any critical test failed, overall result is failed. Otherwise, overall result is passed. Non-critical tests are simply ignored in the overall result.

windelbouwman picture windelbouwman on 8 Dec 2018

I all for only 3 statuses, and not 6, but on the other hand, that would require removal of the criticality code. I think it is easier to add skip and leave critical in place. I understood that people use the critical feature as well. I can understand why they do, since it allows marking failed tests as a don't care. This is different than skip due to a test being non-applicable.

windelbouwman picture windelbouwman on 8 Dec 2018

I think critical-not critical it's a total different concept that
pass-skip-fail...

Il giorno sab 8 dic 2018, 11:41 Pekka Klärck notifications@github.com ha
scritto:

Yes, non-critical failures don't affect the overall verdict of the
execution. Similarly skipped tests won't affect it. These features are so
similar that I don't think it makes sense to have both. One way to think
about this is that we'd re-brand our criticality functionality as skipping.
That's a more commonly used term and works better in our context as well.

Implementation is also a lot easier if we just re-brand. It would also
help visualizing results in report and log. Understanding

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-445449625,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aphlx7sCmrM2LWi4kSw9aNjItUatT3MPks5u25dUgaJpZM4FmCsr
.

acaovilla picture acaovilla on 8 Dec 2018

Yes, non-critical failures don't affect the overall verdict of the execution. Similarly skipped tests won't affect it. These features are so similar that I don't think it makes sense to have both. One way to think about this is that we'd re-brand our criticality functionality as skipping. That's a more commonly used term and works better in our context as well.

Implementation is also a lot easier if we just re-brand. It would also help reporting results in reports, log, and elsewhere. Currently we write results like

5 critical tests, 4 passed, 1 failed
7 tests total, 4 passed, 3 failed

but it could be simplified to

7 tests executed, 4 passed, 2 skipped, 1 failed

or something similar.

In logs are reports visualizing results would get easier as well. Currently the graphs in statistics only show pass/fail ratios, but you cannot see how many of these failures are non-critical. We prototyped having two graphs, one for critical tests and the other for all tests, but that looked horrible and didn't provide any benefits when all tests were critical. Showing pass/skip/fail ratios would solve this problem as well.

pekkaklarck picture pekkaklarck on 8 Dec 2018

We cannot do this re-brand operation.

The problem with this is the following. Say I want to skip a test during runtime. The way I would do this, is by marking it non-critical, and fail the test if it is not applicable. But I also want to indicate failure when the test fails. For this, the test needs to be critical. But having the test critical will not allow me to skip the test.

windelbouwman picture windelbouwman on 8 Dec 2018

I don't understand your concerns @windelbouwman. Could you clarify?

Could you @acaovilla also explain why you think skip and non-critical fail are totally different? I know they aren't exactly the same, but in my opinion the difference is semantic.

pekkaklarck picture pekkaklarck on 8 Dec 2018

I have a bunch of critical test case, but sometimes can't prepare
precondition.. and the setup isn't a part of the functionality. But it
isn't a critical fail, and I can't change tags every time :p

Il giorno sab 8 dic 2018, 11:47 Pekka Klärck notifications@github.com ha
scritto:

Yes, non-critical failures don't affect the overall verdict of the
execution. Similarly skipped tests won't affect it. These features are so
similar that I don't think it makes sense to have both. One way to think
about this is that we'd re-brand our criticality functionality as skipping.
That's a more commonly used term and works better in our context as well.

Implementation is also a lot easier if we just re-brand. It would also
help reporting results in reports, log, and elsewhere. Currently we write
results like

5 critical tests, 4 passed, 1 failed

7 tests total, 4 passed, 3 failed

but it could be simplified to

7 tests executed, 4 passed, 2 skipped, 1 failed

or something similar.

In logs are reports visualizing results would get easier as well.
Currently the graphs in statistics only show pass/fail ratios, but you
cannot see how many of these failures are non-critical. We prototyped
having two graphs, one for critical tests and the other for all tests, but
that looked horrible and didn't provide any benefits when all tests were
critical. Showing pass/skip/fail ratios would solve this problem as well.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-445449919,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aphlx0Tg08Namf7ophKjRVz1oeH1pnN3ks5u25ingaJpZM4FmCsr
.

acaovilla picture acaovilla on 8 Dec 2018

Critical - non critical is about the importance of the functionality of the
test case. And is tag-level design (run, after I can check result for
critical or not). Passed skip fail is a arter-run status that means
execfail, execpass, not exec. But if setup fail, doesn't mean the test case
fail, and I can have a false critical fail that simply don't run the setup

Il giorno sab 8 dic 2018, 11:52 Pekka Klärck notifications@github.com ha
scritto:

I don't understand your concerns @windelbouwman
https://github.com/windelbouwman. Could you clarify?

Could you @acaovilla https://github.com/acaovilla also explain why you
think skip and non-critical fail are totally different? I know they aren't
exactly the same, but in my opinion the difference is semantic.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-445450223,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aphlx6qMYhCKX7lbyWChNgff7_-B-DkFks5u25n2gaJpZM4FmCsr
.

acaovilla picture acaovilla on 8 Dec 2018

To clarify:

In the current robotframework version, I cannot pass/skip/fail a critical test. The way I skip tests now, is by using pass execution half way the testcase. But this is wrong, since the test should be skipped (it did not pass). Failing the test is also no option, since this fails the whole test suite, which I do not want. Another option would be to mark the test non-critical and fail the test when I want to skip it, but this is also no option, since I want the overall result to be failed when the test fails.

Ergo, changing non-critical-failed into blue color and naming it skipped, does not solve this issue, right? I still would have to mark my test as non-critical and fail the test when I want to skip it, which prevents it from failing in a critical manner.

Hopes this clarifies :smile:

windelbouwman picture windelbouwman on 8 Dec 2018

Btw, I think I am unable to make a proper design before hand. I propose to add skip status, introducing the 6 different statuses, leave criticality in place and see how it goes from there.

windelbouwman picture windelbouwman on 8 Dec 2018

why six status?

simply mantain critical-not critical at tags level and introduce skip in
result status.
tags are for classify test case, select test case for a run, or analize
result tag-based.

skip is a final-run status.

what about this idea?

Il giorno sab 8 dic 2018 alle ore 12:08 Windel Bouwman <
[email protected]> ha scritto:

Btw, I think I am unable to make a proper design before hand. I propose to
add skip status, introducing the 6 different statuses, leave criticality in
place and see how it goes from there.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-445451115,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aphlx3kpd-WNgtUF8cbXCoY1mq4R7g5Jks5u252ogaJpZM4FmCsr
.

--

Angelo Andrea Gabriele Caovilla

Angelo.[email protected] angelo.caovilla@chili.com

CHILI | Via Ambrogio Figino 16
https://maps.google.com/?q=Via+Ambrogio+Figino+16+20156+Milano&entry=gmail&source=g

20156 Milano
https://maps.google.com/?q=Via+Ambrogio+Figino+16+20156+Milano&entry=gmail&source=g

www.chili.com http://www.chili.com/

[image:
https://drive.google.com/a/chili.com/uc?id=1bFwOvsXotAuO35MIZf10RBPHZsu6IK69&export=download]

acaovilla picture acaovilla on 8 Dec 2018

@acaovilla Sorry, I meant 3 statuses (pass/skip/fail) and two categories (non-critical/critical). Resulting in 6 statistics, not statuses.

windelbouwman picture windelbouwman on 8 Dec 2018

approved :p

Il giorno sab 8 dic 2018 alle ore 12:20 Windel Bouwman <
[email protected]> ha scritto:

@acaovilla https://github.com/acaovilla Sorry, I meant 3 statuses
(pass/skip/fail) and two categories (non-critical/critical). Resulting in 6
statistics, not statuses.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-445451849,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aphlx0Y5UkPIx_SXwq9nnmzXCHoDVHrIks5u26CJgaJpZM4FmCsr
.

--

Angelo Andrea Gabriele Caovilla

Angelo.[email protected] angelo.caovilla@chili.com

CHILI | Via Ambrogio Figino 16
https://maps.google.com/?q=Via+Ambrogio+Figino+16+20156+Milano&entry=gmail&source=g

20156 Milano
https://maps.google.com/?q=Via+Ambrogio+Figino+16+20156+Milano&entry=gmail&source=g

www.chili.com http://www.chili.com/

[image:
https://drive.google.com/a/chili.com/uc?id=1bFwOvsXotAuO35MIZf10RBPHZsu6IK69&export=download]

acaovilla picture acaovilla on 8 Dec 2018

I still don't understand your problem @windelbouwman.

At the moment if you want to skip a test, you can emulate it by dynamically making it non-critical (add or remove a tag, depending on configuration) and then failing it. Test fails but it doesn't affect the overall verdict of the test suite. Alternatively you can use Pass Execution keyword to make the test pass, but I find this worse.

My idea is that in the future you'd either use a keyword like Skip Test in test data or raise a special exception like SkipTest in a library to skip a test. The old way to make a test non-critical and fail it would work too, good for backwards compatibility, but it would be a lot more cumbersome.

pekkaklarck picture pekkaklarck on 8 Dec 2018

@pekkaklarck I was not aware that a test could be made non-critical dynamically. Then there is no problem, and you can fail the test and "skip" (non-critically fail) the test as well dynamically.

Still my preference would be to keep the topics critical/non-critical and pass/skip/fail strictly seperated. They really should be different things, and not be mixed. By mixing the topics, things become too complex in my opinion.

My suggestion would be to keep the critical/non-critical topic out of this skip status discussion and remove it in a seperate task/branch.

windelbouwman picture windelbouwman on 8 Dec 2018

I totally agree that mixing topics adds complexity. That's exactly the reason I want to remove the current criticality concept if we add skipping.

pekkaklarck picture pekkaklarck on 8 Dec 2018

But why? 2 thing can pacifically convive. Remove critical it's like buy a
car, add GPS but remove stereo.

Il giorno sab 8 dic 2018, 15:21 Pekka Klärck notifications@github.com ha
scritto:

I totally agree that mixing topics adds complexity. That's exactly the
reason I want to remove the current criticality concept if we add skipping.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-445462771,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aphlx1_d1CkOrcRh_cb9jwO8qkfZSEjqks5u28rxgaJpZM4FmCsr
.

acaovilla picture acaovilla on 8 Dec 2018

I totally agree that mixing topics adds complexity. That's exactly the reason I want to remove the current criticality concept if we add skipping.

This I do not understand. I am not very attached to critical/non-critical, but I still think the function has a use. I see its use case unrelated to skipping tests. I understand that the report might become more complex, but this is fine with me.

I would like to keep both features. That being said, I would rather have skipping than criticality if there must be a choice.

windelbouwman picture windelbouwman on 8 Dec 2018

The current criticality feature allows running tests so that possible failures don't affect the overall test outcome. As I've written above, that is useful functionality in the ATDD/BDD context and I don't want it to be removed. That's why my proposal is to change the behavior so that if a test is marked non-critical with --critical/--noncritical and it fails, the test status is SKIP, not FAIL. The end result would be the same as today.

Does someone have other use cases for the current criticality feature? Something that could be accomplished with skipping.

pekkaklarck picture pekkaklarck on 8 Dec 2018

But why failure status in test case that don't really fail but simply can't
be rubbed for no precondition? It is a very bad information.

Il giorno sab 8 dic 2018, 15:46 Pekka Klärck notifications@github.com ha
scritto:

The current criticality feature allows running tests so that possible
failures don't affect the overall test outcome. As I've written above, that
is useful functionality in the ATDD/BDD context and I don't want it to be
removed. That's why my proposal is to change the behavior so that if a test
is marked non-critical with --critical/--noncritical and it fails, the
test status is SKIP, not FAIL. The end result would be the same as today.

Does someone have other use cases for the current criticality feature?
Something that could be accomplished with skipping.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-445464472,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AphlxyFtR4-MYEAhQ1J-ojefkP1USZdnks5u29DWgaJpZM4FmCsr
.

acaovilla picture acaovilla on 8 Dec 2018

Simply, someone need skip in critical e not critical.. a critical not
executed isn't failure. And a not critical not executed isn't failure..

Il giorno sab 8 dic 2018, 15:46 Pekka Klärck notifications@github.com ha
scritto:

The current criticality feature allows running tests so that possible
failures don't affect the overall test outcome. As I've written above, that
is useful functionality in the ATDD/BDD context and I don't want it to be
removed. That's why my proposal is to change the behavior so that if a test
is marked non-critical with --critical/--noncritical and it fails, the
test status is SKIP, not FAIL. The end result would be the same as today.

Does someone have other use cases for the current criticality feature?
Something that could be accomplished with skipping.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-445464472,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AphlxyFtR4-MYEAhQ1J-ojefkP1USZdnks5u29DWgaJpZM4FmCsr
.

acaovilla picture acaovilla on 8 Dec 2018

>

This I do not understand. I am not very attached to critical/non-critical,
but I still think the function has a use. I see its use case unrelated to
skipping tests. I understand that the report might become more complex, but
this is fine with me.

I would like to keep both features. That being said, I would rather have

skipping than criticality if there must be a choice.

This has always been my position as well.

Pekka made a good comment earlier, that if a feature is hard to describe or
document then it's probably poorly designed. However, I absolutely think
that keeping critical/non-critical and adding SKIP is trivially easy to
document:

  • Tests can have one of three statuses: PASS, FAIL, or SKIP
  • If any critical tests fail, the test run as a whole is considered to
    fail. Otherwise, the test run is considered to have passed.

That's it. That's all it takes to describe it. There aren't six statuses
(PASS/critical, PASS/non-critical, SKIP/critical, ...). There are just
three: PASS, FAIL, and SKIP. The fact that you can have a SKIP/non-critical
and a SKIP/critical is just not interesting to me at all: either you run
the test or you skip the test. If you choose to mark a test as critical and
then skip it, you should be allowed to do that.

This to me seems like the logical course of action. No backwards
compatibility issues, and no code has to be removed. Someone a long time
ago already did most (or all?) of the work so it wouldn't take much effort
to roll it into the next big release.

boakley picture boakley on 8 Dec 2018
👍1

>

That's why my proposal is to change the behavior so that if a test is
marked non-critical with --critical/--noncritical and it fails, the test
status is SKIP, not FAIL.

Of all of the various ways SKIP has been proposed, I think that is the
absolute worst possible solution. If a non-critical test fails, it still
runs so you can't simply say it was skipped. You get right back to where we
are today where the report lies about what actually happened. The report
needs to be accurate, and should never say a test was skipped if it
actually ran.

boakley picture boakley on 8 Dec 2018

>

Then there is no problem, and you can fail the test and "skip"
(non-critically fail) the test as well dynamically.

I disagree about there being no problem. The problem is that the report
will not be representing the truth. The fact that a test actually ran
should not be reported as SKIP, since it literally was not skipped.

SKIP should not just mean "not included in the critical results", it should
mean exactly what it says: that a test was skipped and did not run.
Anything else would be very confusing.

boakley picture boakley on 8 Dec 2018

Do you Bryan mean it shouldn't be possible to skip tests dynamically after they've started? I thought that's the primary feature here. If you know beforehand test cannot be run, you can --exclude it already now.

If you only consider skip problem with --(non)critical? If yes, do you actually use that feature?

pekkaklarck picture pekkaklarck on 8 Dec 2018

>

Do you Bryan mean it shouldn't be possible to skip tests dynamically after
they've started? I

No. The whole point of SKIP is for a test setup to be able to determine
that running the test is pointless and thus ask robot to skip it.

If you only consider skip problem with --(non)critical? If yes, do you

actually use that feature?

I'm not sure I understand the first part of that question. Yes, every team
where I've used robot, we've used the --noncritical feature for designating
tests that can fail without failing the build. We don't use it often, but I
think it's a valuable feature that I would really hate to lose. I like
being able to designate some tests as non-critical while still allowing
them to run. For example, we would use a non-critical tag for experimental
features or maybe for some temporary tests we've thrown in for debugging
purposes.

boakley picture boakley on 8 Dec 2018

Will this feature be usable to skip tests based on conditions that are not known until runtime? For my team, the ability to skip tests if upstream tests fail is the desired functionality.

selenareneephillips picture selenareneephillips on 8 Dec 2018

Given boakly's concern about correctness, what about having a test or suite that fails non-critically during Setup be marked as Skipped? That will allow both for skipping tests at run time, while not marking tests that have been run as though they haven't.

fkberthold picture fkberthold on 8 Dec 2018

Will this feature be usable to skip tests based on conditions that are not known until runtime? For my team, the ability to skip tests if upstream tests fail is the desired functionality.

@selenareneephillips Yes, I believe so.

windelbouwman picture windelbouwman on 8 Dec 2018

I have the feeling that there is no consensus about the skip feature. How to proceed now? I think most arguments where on the table at some point. Given this issue is some 3 years old, could we ever expect this feature, or are we in a sort of deadlock?

I am willing to put some time into this, I looked at the 2 previous attempts and they look promising, but it is too bad that this work was neglected. I think it should be possible to have some first version ready within 16 hours.

For me, not having skip status is a major downside of robotframework. The rest I really like a lot. A fork is not my favorite solution.

@pekkaklarck any suggestions on how to proceed? I think it might be best if you implement this feature yourself?

windelbouwman picture windelbouwman on 8 Dec 2018

>

Given boakly's concern about correctness, what about having a test or
suite that fails non-critically during Setup be marked as Skipped?

In my opinion, if a test fails, it should be marked as failed. Saying it
was skipped is a lie since it wasn't skipped -- some condition that was
expected to be true wasn't. The report should report what actually
happened, and not mark failing tests as skipped or skipped tests as pass or
fail.

boakley picture boakley on 8 Dec 2018

Precondition is part of test case, but not a part of test.. es: test login.
Before login, call registration API. Registration services temporary down.
Test named "login" fail.. but.. not really, because login run already.
Next, another test case "registration" fail. But if login and registration
report say 2 fail, it's simply a "lie"

Il giorno sab 8 dic 2018, 23:52 Bryan Oakley notifications@github.com ha
scritto:

>

Given boakly's concern about correctness, what about having a test or
suite that fails non-critically during Setup be marked as Skipped?

In my opinion, if a test fails, it should be marked as failed. Saying it
was skipped is a lie since it wasn't skipped -- some condition that was
expected to be true wasn't. The report should report what actually
happened, and not mark failing tests as skipped or skipped tests as pass or
fail.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-445495936,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aphlx1qRBTRAcL0SEM8qvbrG0_dgyjSBks5u3EK1gaJpZM4FmCsr
.

acaovilla picture acaovilla on 8 Dec 2018

16 hours? That seems awesome. I would really really like to see this functionality. It is the biggest roadblock to my team's adoption of Robot as the framework of choice for test automation.

There also does not appear to be any other really robust test automation tool for Python that has mature feature support for dependencies, skipping and test ordering.

I am willing to put some time into this, I looked at the 2 previous attempts and they look promising, but it is too bad that this work was neglected. I think it should be possible to have some first version ready within 16 hours.

selenareneephillips picture selenareneephillips on 10 Dec 2018

Probably I am a little optimistic, but using the previous attempts should really help. The problem however is that there is no clear picture on how the function should work. At least, there appears no consensus.. Unfortunately I do not know how to solve this design.

My fork with skipping support is located here: https://github.com/windelbouwman/robotframework it is still work in progress, I need to add the keyword Skip Test and the exception handling still.

windelbouwman picture windelbouwman on 10 Dec 2018

If u want I can try to design it. I work in QA for 6 years in different
situation.

Il giorno lun 10 dic 2018, 19:33 Windel Bouwman notifications@github.com
ha scritto:

Probably I am a little optimistic, but using the previous attempts should
really help. The problem however is that there is no clear picture on how
the function should work. At least, there appears no consensus..
Unfortunately I do not know how to solve this design.

My fork with skipping support is located here:
https://github.com/windelbouwman/robotframework it is still work in
progress, I need to add the keyword Skip Test and the exception handling
still.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087#issuecomment-445922188,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aphlx242rJ_pKRizmVirWlyWo3-kjOsEks5u3qj6gaJpZM4FmCsr
.

acaovilla picture acaovilla on 10 Dec 2018

Design of the result log page:

image

windelbouwman picture windelbouwman on 10 Dec 2018

Hi windelbouwman, on Oct 16, 2015 when I first pushed a set of changes to implement the SKIP functionality, see comment:

https://github.com/robotframework/robotframework/issues/2087#issuecomment-148559931

I had it such that the report would show Skip Test results in CYAN (Fail Test results being RED). if you look at the example reports (generated from my source and tests at that time) you will see that this presents a nice clear set of test results were the user can see instantly what was skipped etc.

It would be good in your changes if you carried this idea forward in the final implementation. I am sure this will be populate within the user community.

Doug8080 picture Doug8080 on 13 Dec 2018
👍1

Needed some time to read all the latest comment :)

I'm totally agree with 3 final statuses and keep the non-critical tag feature!
@pekkaklarck the critical tag feature is very nice and it's one of the different concept that RF has compared to other frameworks so I wouldn't remove it.
Renaming is not a solution otherwise people wouldn't have opened / commented this thread, the fact is that the expected SKIP functionality is completely different from critical and other already explained the idea perfectly.

In my experience I used SKIP to avoid tests to be executed at all depending on the environment.
For instance I've a set of test cases to be executed only for a specific version of the system under test that I might not know before running the test.
Or a test that cannot be executed if a feature is disabled, so the test verifies whether this feature is enabled and if not test is skipped.

Reports should include this information separately.
About the 6 combinations I think that SKIP + critical is a problem that might cause an overall failure since a critical test should not be skipped.
SKIP + non-critical who cares just skipped a non important test..

To have more simple and readable reports maybe critical and skip tests data might not be shown if not used so that people can choose the approach with just one of the two things SKIP / critical, none or both.
Maximum flexibility without adding to much complexity.
It has to be said the SKIP functionality is widely adopted in other frameworks so nothing very difficult to understand, while critical/non-critical tags are self explanatory..

@pekkaklarck we might use the sprint time of the RoboCon to work on this long awaited feature ;)

lucagiove picture lucagiove on 16 Dec 2018
👍4

@windelbouwman I have a copy of your forked repo with Skip changes. Any way i could test the Skip feature?

ram-sid picture ram-sid on 28 Dec 2018

@ram-sid feel free to copy paste the remaining changes required from the previous attempts and create a pull request. Ultimately this feature should go into this robotframework repository, so maybe you could await the sprint at RoboCon as mentioned above?

windelbouwman picture windelbouwman on 29 Dec 2018

@windelbouwman Yes, i'll wait for the sprint to complete. Thank you!

ram-sid picture ram-sid on 30 Dec 2018

Any updates?

sebastianciupinski picture sebastianciupinski on 8 May 2019
👍4

I second the need of the SKIP feature. I would appreciate something easy, e.g. Pytest's user-friendly: @pytest.mark.skip(reason="API is broken now")

R3gi picture R3gi on 19 Aug 2019
👍3

After reading most of the comments I think one problem is that there are many different situations in which there is a need for a skip functionality. My own reason for reading this thread was a need for having preconditions for a tests, and if the preconditions were not met I would not like the test to run and would like to have it market as skipped, instead of failed or passed. But, obviously there are several different situations. So here are some thoughts about this in the form of different scenarios:

1) I know beforehand that there are certain tests I want to run and certain tests that should be skipped. These have tags that I can use. This case is solved by using --include, --exclude as it is today. If there are critical and non-critical tests they are also identified in the same way as today.

2) There are certain resources (or similar) that must be available for the test to run. If not so, there is no need for the test to run so it should be skipped. In this case it is actually an error that the test did not run, but it is not a fault of the test itself. Personally I think the test should be market as skipped, and failing the overall test run if critical.

3) A test should only be run if the tested system is in a certain configuration, otherwise skipped. In this case it is NOT an error that the test is skipped, we just did not know beforehand what the actual configuration was. If the test itself is critical (if it should be run) one could change the tagging to non-critical, but perhaps it would be better to use some other indicator. It would also be possible just not to show this test in logs, but I think many people might want to be able to see why the test was skipped.

So in summary I think something like below is appropriate. Skipping means if skipping the test was OK or not (case 2 or 3 above).

Action Skipping Critical/non-Critical Color (suggestion) Overall test

Test runs and fails Critical Red (as today) Fails
Test runs and fails Non-critical Red (as today) OK
Test skipped Not OK Critical Orange Fails
Test skipped Not OK Non-critical Orange OK
Test skipped OK Critical Yellow OK
Test skipped OK Non-critical Yellow OK

The skipping OK/Not OK could perhaps be set by setting some variable, using some function or similar.

magand01 picture magand01 on 21 Feb 2020

The table in the previous comment did not really come out as expected. New attempt here:

Action | Skipping | Critical/Non-critical | Color (suggestion) | Overall test
------- | ---------- | --------------------- | --------------------- | ------------
Test runs and fails | | Critical | Red (as today) | Fails
Test runs and fails | | Non-critical | Red (as today) | OK
Test skipped | Not OK | Critical | Orange | Fails
Test skipped | Not OK | Non-critical | Orange | OK
Test skipped | OK | Critical | Yellow | OK
Test skipped | OK | Non-critical | Yellow | OK

magand01 picture magand01 on 21 Feb 2020

Hemmm.. if I skip test I think the result Is Just skipped.. not "ok -
skipped" or "not-ok skipped"

Il ven 21 feb 2020, 22:54 magand01 notifications@github.com ha scritto:

The table in the previous comment did not really come out as expected. New
attempt here:
Action Skipping Critical/Non-critical Color (suggestion) Overall test
Test runs and fails Critical Red (as today) Fails
Test runs and fails Non-critical Red (as today) OK
Test skipped Not OK Critical Orange Fails
Test skipped Not OK Non-critical Orange OK
Test skipped OK Critical Yellow OK
Test skipped OK Non-critical Yellow OK

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/robotframework/robotframework/issues/2087?email_source=notifications&email_token=AKMGLR2VGCJGDATE2EMJTNTREBEQZA5CNFSM4BMYFMV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMUHEGY#issuecomment-589853211,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKMGLR3O4IGQ4DK5AZ6ISBLREBEQZANCNFSM4BMYFMVQ
.

acaovilla picture acaovilla on 21 Feb 2020
👍2

Sorry but any actions for skip status. I have been looking for this feature. Thanks.

duychu picture duychu on 28 Apr 2020

@magand01 I'm sorry, but where's the value of failing critical tests on a skip? A skip is a skip. People are looking for an ability to communicate "for one reason on another we won't run this, period".

Assuming any other intentions or a testing philosophy is a mistake and will lead to making this feature unusable for some parts of the community.

MrMino picture MrMino on 13 May 2020
👍1

The long awaited skip status is coming in RF 4.0 that's planned for December! I submitted #3622 with more concrete design and close this one as its duplicate.

As you'll see from #3622, criticality will go when skipping comes. I know its a controversial decision but I've been thinking this for a long time and believe that's the best way forward. For more reasoning see the separate issue #3624.

pekkaklarck picture pekkaklarck on 3 Aug 2020
❤2 👍2 🎉1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

pekkaklarck picture pekkaklarck  ·  3Comments
gkeishin picture gkeishin  ·  5Comments
pekkaklarck picture pekkaklarck  ·  7Comments
gazsiazasz picture gazsiazasz  ·  5Comments
marcinmath picture marcinmath  ·  4Comments