Azure-pipelines-tasks: Task "Run Functional Tests" get current state of test case in real time

[kardinal3000]

As far I know in MTM we have had opportunity to see results of test case in real time. In my case
I had TFS 2013 Test Agent and I have run remotely my test suite using Lab definition (xaml). After that I open MTM and had opportunity to watch in what state my "test case" in my test suit.

As I can see, right now we lost that!

That small example of my log:

DistributedTests: Test run 'XXXX' is in 'InProgress' state.
DistributedTests: Total Tests : 983, Passed Tests : 0
DistributedTests: Test run 'XXXX' is in 'InProgress' state.
DistributedTests: Total Tests : 983, Passed Tests : 0
DistributedTests: Test run 'XXXX' is in 'InProgress' state.
DistributedTests: Total Tests : 983, Passed Tests : 0
...

My main question is: How can we return this opportunity? or maybe create new solution or start use some custom plugins for that if they exist of course. It's really important to see the result in real time!

Would be great if we will get the next console output.

DistributedTests: Total Tests : 983, Passed Tests :  Test 1 -  Passed
DistributedTests: Total Tests : 982, Passed Tests :  Test 2 -  Field
DistributedTests: Total Tests : 981, Passed Tests :  Test 3 -  Passed    
DistributedTests: Total Tests : 980, Passed Tests :  Test 4 - InProgress 
...

Thanks!

[atinb]

We update after the execution finishes on each machine. If you go to the Test tab and look at the In progress Run, you will see the test results updating as each dll or machine executing the tests finishes execution.

[kardinal3000]

@atinb Yes, you are right! When I have run my "CodedUITests" . During runs I go to the Test tab - Runs tab - then find tests with state "in progress" open and I can see only number of tests

Open Tests results tab I see all my tests with the status "None"

So, now my problem. I have no idea in what state my tests are in, even though I open Test tab. But if I run my CodedUITests using xaml build definition (lab definitions) I can see updates in real time on Tests tab - works fine! That issue confused me why it happened.

1. On test VM I use last version Test agent Microsoft Agents for Visual Studio 2015 with Update 3 - 14.025420.1
2. TFS 2015 with update 3
3. Build Agent 1.95.3

maybe I missed something? Please need your help

[atinb]

Like I said, we update results after the execution finishes on each machine. If you have only 1 machine, then the status of all 8 tests will be updated post execution of all 8. If you have 2, you should see an update after 4 tests...

[kardinal3000]

@atinb Thanks, but just only my curiosity. In future, maybe next TFS update, we will have such opportunity? like to see updates of "Tests" in real time. Or it will be never supported.

Why I so interested in it. We have huge Tests Suites and sometimes it takes more then 15 hours execution. Furthermore situation from real life. I have Test Suit with 1500 regression tests. One test VM. I know, if some tests will fail, it makes no sense to wait for the whole process is done. With new build system we lose such opportunity. I have to wait for the full completion of the process, because I can't see current state of my test.

Thank for advance.

[SashaSidorenko]

Also would like to see where we are on long time regression testing :)

[varnk]

I would like to see the console output update in real-time as well. This is how the Maven test step works. The Run Functional Tests step should do the same.

[mahadevblore]

I would like to have the test cases updated real time. This makes it easy for us to determine whether there was an issue with Test Run well before it gets completed. Since typical GUI Test Runs for a large suite takes 10 -15 hours to execute. It will be really a good option to see the test case execution status in real time.

[rishavsharan]

Hi folks
I am one of the PMs working on Automated Testing and we are keenly looking into the idea of showing the test results in real time. Allowing tests to fail fast and the developers to make decisions based on real time test insights, is a direction we want to go in.

You can keep an eye on the https://docs.microsoft.com/en-us/azure/devops/release-notes/ for our high level roadmap. Once we have closed on the design for this feature, we will be adding it to this list.