Azure-devops-docs: Vstest fails build pipeline, when "Flaky tests included in test pass percentage" is unchecked under "Flaky test options"

[luweilevi]

Flaky test failures should not fail my pipeline, when flaky tests are not included in the Test Summary. As described in this document (https://docs.microsoft.com/en-us/azure/devops/pipelines/test/flaky-test-management?view=azure-devops#flaky-test-management-and-reporting).
However, It still marked Vstest task failed and pipeline failed, when the only flaky tests failed.

[buckleyGI]

I can also reproduce this
https://stackoverflow.com/questions/58507830/flaky-tests-are-not-taken-into-account-by-the-publish-test-results-task

[KathrynEE]

@luweilevi, @buckleyGI - Thank you for the feedback. I'm sending this information on to the PM for this feature and will get back to you with their response.

[Additi]

Hi,
This is a known issue and we are working on the fix. I'll post the update once we deploy the changes from our end.
Thanks,
Aditi

[buckleyGI]

Hello @KathrynEE. Please don't close the issue if it is still open.

As of today, the task PublishTestResults@2 still reports failure when there are only flaky tests that fail.

When failTaskOnFailedTests: true the expectation is that the task will report partial success when only flaky tests fail right?

Note that the behavior was changed recently for VSTest@2 where it reports partial success when only flakies fail in combination with publishRunAttachments: true (the default).

The problem atm is that PublishTestResults is not changed to take into account flaky tests.

[buckleyGI]

Note that the original author created the issue for the VSTest@2 task but I was expecting the same change for the PublishTestResults@2 task. I acknowledge that I am slightly off topic but it's basically the same issue.
Can you disclose your plans for the PublishTestResults task? Thank you.

[NickMSW]

Hi, I'm having the same issue, however, I'm using the dotnet test task which automatically publishes the results to the DevOps pipeline. Also, this issue is listed as closed but in reading the comments it appears that it's not. Is there any update on this issue as a whole?

[WilliamAntonRohm]

@Additi -- Aditi, please look into this issue.

[Ansssss]

Still encountering this issue, forces us to re-run builds unnecessarily.

[sphanley]

My organization is seeing the same issue. Is any update available on this? It's really disappointing to spend time and energy making changes to our CI workflow to incorporate a feature that's been in production for at least a year, per this post, only to find out that it doesn't actually work as described.

[amit0912]

Has anyone figured out a workaround for this? Possibly uncheck the Fail if there are test failures in the Publish Test Results task and add another task which evaluates whether there any failures in the Tests Section and then determines pipeline Pass/Fail

[lygasch]

Adding a comment to keep this thread active. Was just about to onboard for this. If the flaky process won't ignore flaky tests or automatically file bugs, and reporting the flaky tests involves looking at individual test run results (doc to reporting does not show how to retrieve a list of flaky bugs) then I don't really see any value. If it will ignore the failures and at least give me an easy daily/weekly view (and maybe I will find this if I keep digging around) so that we can have a process for handling, then it would be very useful.

[KathrynEE]

@Additi - D o you have an update on this issue? If so, please add a comment to this Issue.
@luweilevi, @buckleyGI, @NickMSW, @Ansssss, @amit0912, @lygasch - I'd like to close this issue as it isn't a doc issue. It would be better served for it to be tracked as a bug in our Developer Community page. I'm keeping it open for now, waiting for a response from @Additi.

[sandy-wang]

This bug still exist, it's really annoying that we have to have "failTaskOnFailedTests: false" which means new failures won't fail the build at all. Will create a ticket in Developer Community page.

[sandy-wang]

@amit0912 very interested in knowing how did you manage to get this working? "add another task which evaluates whether there any failures in the Tests Section and then determines pipeline Pass/Fail". I am trying to find if there is a way to get "pass percentage" by rest api.