Specflow: Specrun CLI: Tests are being discovered, but wait long time before starting even with valid license

Created on 22 May 2019 · 17Comments · Source: SpecFlowOSS/SpecFlow

SpecFlow Version:

[x] 3.0
[ ] 2.4
[ ] 2.3
[ ] 2.2
[ ] 2.1
[ ] 2.0
[ ] 1.9

Used Test Runner

[x] SpecFlow+Runner
[ ] MSTest
[ ] NUnit
[ ] Xunit

Version number: 3.0.337

Project Format of the SpecFlow project

[ ] Classic project format using packages.config
[x] Classis project format using <PackageReference> tags
[ ] Sdk-style project format

.feature.cs files are generated using

[x] SpecFlow.Tools.MsBuild.Generation NuGet package
[ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

[ ] VS 2019
[x] VS 2017
[ ] VS 2015

`Enable SpecFlowSingleFileGenerator Custom Tool` option in Visual Studio extension settings

[ ] Enabled
[x] Disabled

Are the latest Visual Studio updates installed?

[x] Yes
[ ] No

.NET Framework:

[x] >= .NET 4.5
[ ] before .NET 4.5
[ ] .NET Core 2.0
[ ] .NET Core 2.1
[ ] .NET Core 2.2
[ ] .NET Core 3.0

Test Execution Method:

[ ] Visual Studio Test Explorer
[ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
[x] Command line – "{path to specrun.exe in bin folder}" run {our profile}.srprofile --baseFolder "{path to project folder}" --outputFolder "{path to a file share}"

Section in app.config or content of specflow.json

{it's empty}

Repro Project

Unfortunately it's internal, so we cannot include it

Issue Description

We just migrated from SpecFlow 2.4 and SpecRun 1.8 to SpecFlow 3+ and SpecRun 3+. Before when we would run our test from the command line, the tests would be discovered and then ran immediately on the desired number of threads. We upgraded the versioning and fixed all of the breaking changes, then tested everything successfully from within Visual Studio 2017 (Enterprise, though I imagine that shouldn't make a difference). Now we are trying to run the tests via the command line (command included above) and for some reason the tests ARE discovered immediately, but won't run for about 30 seconds to a minute. The trace will be output to the console with "(0/{# of tests}) completed" and the lane for each thread displayed, but the first test finally gets cued after about 30-60 seconds.

Attempts at troubleshooting have been to retry running the tests in VS (which was successful) and unregistering then re-registering the license to ensure that it is being respected (successfully executed, but didn't fix our issue). We then ran the command again with Task Manager open to see if something was consuming too many resources to find that the SpecRun.exe process was consuming no resources until that 30-60 seconds had passed.

We have no clue how else to tackle this. Is there anything that we could be failing to set to ensure the same speed provided by VS is found using the CLI?

Steps to Reproduce

Ensure you have your license set up
Build the project
Call the above CLI command with valid parameters
See the command indicate that the tests were discovered and have it start the trace, but with no progress for 30-60 seconds

Bug SpecFlow+ hard medium

Source

btvanhooser

All 17 comments

@btvanhooser There should be a logfile, normally named specrun.log. Could you send this to [email protected]?
Is the baseFolder parameter really empty or did you removed it?

SabotageAndi on 22 May 2019

I get that right now, and no I used angle brackets to detail what was in each double quotes and forgot that it was markup... so it got removed..