Reportportal: Report Portal to separate between Error & Fail

[roniezr]

Describe the solution you'd like
It seems that pytest separates between Error and Fail.

• Fail is when the tested issue is not working as expected.
• Error is when a failure occurs before and after the tested issue (setup & teardown).

For example:
If my test issue is to check ping between two Clients and the test fail during client initialization
then it is an Error, but if ping is not replying then it is a Fail

The Problem:
Report Portal does not distinguish between the two statuses, both will be signed as Fail.

Notes:

• The word Error is confusing, but that's how Pytest calls it. Think of it as a Fail-At-The-Non-Tested-Issue
• The idea is that I can't say that "ping between two clients" is not working when I have an Error status,
  but with Fail status, I can be sure that it is not working.
• It should also be helpful by tracking the test history
  E.g.: History example for the Ping between Clients test:
  
  
  
  1. With current report portal: Fail > Fail > Fail > Fail > Fail # we may think that Ping was always fail
  
  
  2. With suggested report portal: Error > Error > Error > Error > Fail # we immediately see that this is the
     
     first time we check Ping
  
  

Is your feature request related to a problem? Please describe.
...

[DzmitryHumianiuk]

@roniezr will it work, if you send it like FAIL status, but push it with custom defect type?
for instance, send as Failed, ErrorFail (from system issue group)?

[DzmitryHumianiuk]

@reportportal/python-contributors what do you guys think?
is it the common approach to use ERROR status?

[roniezr]

I'd glad to see new STATUS type called "Error" instead of all "FAILED" as we have today

[trinhpham]

IMO, agree to @roniezr, we should separate these statuses.

• ERROR means something make the test cannot be run correctly => Either test script, configuration or test environment need to be reviewed and fixed. This work should be allocated to Testers/QAs side.
• FAILED means the test ran correctly, and its check is failed => Report to Developer for the issue.

[nvborisenko]

Please don't introduce more test statuses on the server side. Doesn't matter how pytest differentiates failures, I can bring more different statuses from others unit frameworks.

In my opinion test should be good or bad on the server. Issues customization should answer on the question why exactly test was bad. So we should think about how to improve issue categories. Might be tree-view categories, or possibility to assign several categories.

[DzmitryHumianiuk]

Agree with @nvborisenko

Don't forget what you do with tests. You test!
Test can pass or fail.
it is the ultimate goal for test: check condition or show that you failed.
ReportPortal should show you tests statuses first of all. Represent fail roots is secondary.

All the rest are the reasons of fail. This should be defect types.

Resolution: rejected