Appcenter: Support Specflow 3 in App Center Test

[grzegorz-jarosz]

What App Center service does this affect?
App Center Test Cloud - SpecFlow test results show no step results

Describe the bug
I've upgraded my project to SpecFlow 3 (and NUnit 3). Tests are executed and I can see scenario result in App Center, but results show NO sceario steps results (and screenshots). Only single entry with double dash "--" is shown.
At the top of the logs, a following note about SpecFlow can be seen:
SpecFlow integration aborted: Unable to find 'testTracer' field.

To Reproduce
Steps to reproduce the behavior:

1. Build tests using SpecFlow 3 and NUnit 3
2. Run the tests in App Center
3. Check Scenario results
4. Look at each step result

Expected behavior
Each scenario step and result is shown on a page with attached screenshot.

Screenshots
Okay, I`ll "borrow" the screenshot from similar issue reported for SpecFlow @SpecFlow, as it demonstrates the problem:

Additional context
I've had a conversation using chat feature in App Center. I was informed that issue is tracked/reported internally. When I asked about public URL I could show with my team I was asked to report the bug on GitHub.

[Oddj0b]

Thank you for reporting the bug, we will investigate.

[Oddj0b]

We have triaged the bug and the plan current plan is to fix it later with more bugs we have found in the Test reports.

[arek-avanade]

Any update on this? This issue affects my team as well.

[maku99]

Guys, we are still waiting for your fix. Any news on this ?

[michaelrobayoGasTag]

I am also eagerly waiting for this fix, it was very very good at helping me determine visually where the app crashed and what state the UI was in when it did.

[Oddj0b]

Sorry, we still haven't made any progress.

[King-of-Spades]

I came across this while researching related issues, and it appears that SpecFlow 3 support wasn't ever added to Xamarin.UITest in the first place. The highest version that is present in Xamarin.UITest is SpecFlow 2.

I believe this is also consistent with the findings of the SpecFlow team here:

when using SpecFlow >= 3, code-behing generation of feature files is not supported using SpecFlowSingleFileGenerator.
To make sure this is not the cause of the problem, I'd ask you to follow our migration guideline for SpecFlow 3: https://specflow.org/2019/updating-to-specflow-3/.

• https://github.com/techtalk/SpecFlow/issues/1578#issuecomment-493861022

As such, I'm converting this from a 'bug' to a 'feature request' to add SpecFlow 3 support.
Tracking ID: ADO - 62163

[yma44]

We have upgraded our test to use specflow3 a few month ago, and we haven't been able to run test in app center for the last few months because of this issue. we are very keen to run our test in app center and automation testing is quite crucial. I would appreciate if this gets fixed sooner. Thank you

[grzegorz-jarosz]

It looks like I was so excited to see that AppCenter supports NUnit 3, that SpecFlow 3 (which can only use NUnit 3) was a natural fit to me.

@King-of-Spades Highlighted SpecFlow comment mentions .feature.cs file generation. With version 3.0 they moved from generating .feature.cs when saving file in VS to generating them during the build. (which seems like a nice improvement to me). Could share some insights how this two things are related? For me it looks like moving from SpecFlow 2.4 to 3.0 they changed some reporting stuff as well, although I couldn't find anything interesting in their changelog.

Anyway, We're waiting for this feature then, thank you for your support

[King-of-Spades]

@grzegorz-jarosz Unfortunately I don't think I can answer your questions myself, I pieced together discrepancies and confirmed with our engineering team that only SpecFlow & SpecFlow 2 were enabled (while SF3 was not), but saw that wasn't clearly described on this issue.

@Oddj0b Aare you able to comment further on those observations?

[komireddy407]

Hi @grzegorz-jarosz @Oddj0b @King-of-Spades @yma44 @arek-avanade
I'm also facing same issue and i have upgraded to VS 2019 and still I'm getting the same issue
Can any one please suggest best compatibility for VS 2019 (SpecFlow)?

1. SpecFlow
2. SpecFlow.NUnit
3. Nunit
4. XamarinUITest
   Can not we use SpecFlow.Nuint and Nunit at same, refer below image
   

Thanks in Advance.

[asafetech1]

Hi Ya what is the status of this issue ,I wish to use xamarin + specflow 3

• App center

[maku99]

Guys, any news ?

[YevhenKuraiev]

@grzegorz-jarosz @King-of-Spades Hi guys, any news?

Unable to scan assembly: nunit.engine, Version=3.10.0.0, Culture=neutral, PublicKeyToken=2638cd05610744ebException: System.IO.FileNotFoundException: Could not load file or assembly 'Mono.Cecil, Version=0.9.6.0, Culture=neutral, PublicKeyToken=0738eb9f132ed756' or one of its dependencies.File name: 'Mono.Cecil, Version=0.9.6.0, Culture=neutral, PublicKeyToken=0738eb9f132ed756'
SpecFlow integration aborted: Unable to find 'executionEngine' field

Spec Flow 3.1.89
NUnit 3.12.0

[King-of-Spades]

@Oddj0b would have to comment if there's any new info; I just don't have those details myself, sorry.

[msftbot[bot]]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[grzegorz-jarosz]

Two years have passed, yet the only meaningful update we had is a cryptic ADO ticket number in one of the comments.

Few days ago SpecFlow released version 3.9.8 of their library, AppCenter Test still doesn't support appium tests with different languages than Java and new test tools appear improving and going way beyond the features we had in early Selenium/Appium days.

I think that makes you solution less appealing for users seeking mobile automation and some users will leave your solution and use competitors products.

If you really need to close it, please do so, this will make things clear that we won't get such support anytime soon (or even anytime at all)

[elnate23]

Team,

I am deeply concerned by this issue. Specflow 3 has been going for quite a while and I just purchased a license for App Center because it looked like a great product. This limitation is a great showstopper for the integration of our mobile testing framework. Is there going to be any updates at all?

I would appreciate any response at all on the situation.

Thanks