Specflow: The code inside AfterTestRun fails to complete before the tests are unloaded

[godrose]

SpecFlow Version:

• [x] 3.0.150-beta

Used Test Runner

• [x] xUnit 2.4.1

Visual Studio Version

• [x] VS 2017 Enterprise

Are the latest Visual Studio updates installed?

• [x] Yes

.NET Framework:

• [x] .NET Core 2.0

Test Execution Method:

• [x] Resharper 2018

<SpecFlow> Section in app.config

{
"bindingCulture": {
"language": "en-us"
},
"language": {
"feature": "en-us"
},

"plugins": [],
"stepAssemblies": [
{
"assembly": "BDD.NetCore.Steps.Adapters"
},
{
"assembly": "BDD.NetCore.Launcher"
}
]
}

Repro Project

https://github.com/godrose/specflow-failing-after-test-run

Issue Description

The [AfterTestRun]-decorated method is not invoked. This does not work in the previous 3.0.0 beta release either.

[SabotageAndi]

@godrose Could you provide us a complete project to reproduce this? Our integration tests are green for AfterTestRun hook.

[godrose]

@SabotageAndi Here you go:
https://github.com/godrose/specflow-failing-after-test-run
Happens for both beta versions on Windows and MacOS
FYI, all other hooks are invoked successfully

[godrose]

@SabotageAndi I might be wrong but it seems like you don't actually test the AfterTestRun hook
https://github.com/techtalk/SpecFlow/blob/master/Tests/TechTalk.SpecFlow.Specs/Features/Hooks/HookSupport.feature

[SamJongenelen]

@SabotageAndi I might be wrong but it seems like you don't actually test the AfterTestRun hook
https://github.com/techtalk/SpecFlow/blob/master/Tests/TechTalk.SpecFlow.Specs/Features/Hooks/HookSupport.feature

I think you're right; since NUnit is being used, the test will never pass if commented out.

[SabotageAndi]

@godrose & @SamJongenelen: It was a long day, so I am not sure if I oversee something.
Why do you think, we don't test the AfterTestRun hook?

Here is the test:

[godrose]

@SabotageAndi @SamJongenelen Have you seen the repo? The issue is reproduced quite well there.

[godrose]

@SabotageAndi Where are these tests located?

[SabotageAndi]

I looked now at the repo and send you a PR https://github.com/godrose/specflow-failing-after-test-run/pull/1 to better see, that the hooks are executed.
Yes, the AfterTestRun hook is executed.

The reason you don't see it, is that xUnit doesn't capture Console.Writeline. So you don't see your Console.WriteLine in the output.
Additionally it looks like if you debug it, the test process is stopped, also if you are on a break point.

So I changed the hook to write into a file. Now you see that the hook is executed.

About the tests:
They are these: https://github.com/techtalk/SpecFlow/blob/master/Tests/TechTalk.SpecFlow.Specs/Features/Hooks/HookSupport.feature
We generate with a generator plugin (https://github.com/techtalk/SpecFlow/tree/master/Tests/TechTalk.SpecFlow.Specs.Generator.SpecFlowPlugin) scenarios for the multiple configurations.
You have to build one time the solution that you see the tests.

[SamJongenelen]

thank you for the reply. I based the assumption on the following line:
https://github.com/techtalk/SpecFlow/blob/master/Tests/TechTalk.SpecFlow.Specs/Features/Hooks/HookSupport.feature#L83

[godrose]

@SabotageAndi Thanks. I'm checking the real project where this issue has happened. Interestingly enough, when I run the tests via dotnet test everything works as expected. The issue only is reproduced when I use Resharper.
I guess the issue can be closed now...

P.S. What I found out after some digging is the following:
If you add some waiting inside the AfterTestRun and then try to call the log line it doesn't work.
Again, I use Resharper and run tests from within VS. Sometimes I can actually see the breakpoint being hit but then the tests run is stopped. It looks like there's some race condition here...

[SabotageAndi]

The AfterTestRun hook is a little bit special, because in the unit test runners there aren't any events that we can use.
So we are using the AssemblyUnload event.

From https://specflow.org/documentation/Hooks/:

Note: As most of the unit test runners do not provide a hook for executing logic once the tests have been executed, the [AfterTestRun] event is triggered by the test assembly unload event. The exact timing and thread of this execution may therefore differ for each test runner.

Perhaps this makes problems for you.

[godrose]

Yes. This is the problem here. This used to work in the latest 2.x version which used netframework, though...
What could be the workaround here?

[godrose]

https://stackoverflow.com/questions/13829737/xunit-net-run-code-once-before-and-after-all-tests

[SabotageAndi]

ok, this would be possible. But it needs a lot of changes.
Currently we can't generate a class that has nothing to do with a feature file.

@gasparnagy Good ideas how we do this?

[nvborisenko]

This issue is reproducible for me too. Target framework is netcore. I use MsTest framework as unit provider. The issue is not reproducible if change target framework to full net. Will come back to you soon with simple example to reproduce it. Please move this issue to ToDo list. Thx.

[nvborisenko]

This is simple example.
Issue1348.zip

Unpack it and just execute tests via dotnet test. Test project is multi-targeted: net471 and netcoreapp2.1. Command executes tests against 2 target frameworks. We have [BeforeTestRun] and [AfterTestRun] hooks which writes some text message to bin\debug\log.txt file.

We can see that [AfterTestRun] hook is not awaited in netcoreapp2.1 framework, and works good in net471 framework. If you remove Thread.Sleep() in hook and execute tests again, we will see correct text message in the file.

I can suppose that actually [AfterTestRun] hook is executed, but not awaited for completion in netcore tests.

[SabotageAndi]

I am not sure if Assembly unloading is already implemented in .NET Core ...
@nvborisenko Did you try it also with .NET Core 2.2?

[nvborisenko]

Even with .Net Core 3.0-preview, doesn't work with vstest. Raised issue to vstest repository.

[godrose]

@SabotageAndi Any update on this issue? Is it planned to be a part of 3.0.0 release?

[SabotageAndi]

No idea tbh. I am the second day into the office after vacation and have now read the VSTest issue.
I have to think how to solve this.

[tsecomandi]

I have the same issue.

Changing the project target framework to net471would make it wait for the completion of the [AfterTestRun] hook?

[nvborisenko]

Just try. What I gathered is nunit gives 30secs for [AfterTestRun] completion (nunit-console.exe supports only full net framework). VsTest gives 100ms for 'netcoreapp' tests. Not sure, need verify.

Please don't forget to report your results.

@SabotageAndi it seems specflow should rely on [TestAssemblyCleanup] attribute, which is specific for every unit test framework. I guess this is not easy fix for specflow :( and I hope specflow team will be able to resolve this issue.

[SabotageAndi]

In <= SpecFlow 2.4 we have a NUnit Addin for it.
See https://github.com/techtalk/SpecFlow/blob/line-v2.4/Installer/NuGetPackages/SpecFlow.NUnit.Runners/SpecFlowNUnitExtension.cs

Not sure if it worked after 1.9.
Now it will not, because it is using ScenarioContext.Current.

One of the problems I don't yet know how to solve, is that the AssemblyCleanup attributes are sometimes only allowed once per assembly.
This is a problem, if a user has already used it.

The other one is, that we can only generate code files for a feature file. We have no option, to add a single code file to the assembly to compile.

Has anyone an idea?

[nvborisenko]

As to me AssemblyCleanup per one assembly is expected.

What about single code file generation: may be it can be resolved by msbuild props and targets. For example if my project references to "SpecFlow.NUnit" package, and I want to build my project, then target from SpecFlow.NUnit package adds file for compilation on "prepare for build" phase. I hvae no big experience here.. just thoughts.

[StevenBlair123]

We have been experiencing this problem. Hoping someone can come up with a solution.

We are using:

xUnit
net core 2.0
SpecFlow.xUnit 3.0.161-beta

AfterTestRun is not being fired.
We tried running with our debugger, and occasionally, the source file containing the AfterTestRun method is shown, but immediately the process stops.

[StuartFergusonVme]

Done a bit more investigation on this issue and have confirmed as the issue suggests that the AfterTestRun method is indeed being fired but the code in this does not complete :(

Added in a foreground thread starting in BeginTestRun in an effort to keep the process from being unloaded but this does not seem to help, in fact it seems to hinder the issue as the AfterTestRun stops being called.

Also added the thread code into AfterTestRun to try and allow the code in there to complete but again the process is getting killed before this can completed.

[ninjaboy]

hi guys, we are using xUnit, .netcore2.2, specflow.xunit 3.0.161-beta as well
Can confirm that AfterTestRun is not running neither by using dotnet test, nor by running the tests in VS Tests runner in debug.
I can also confirm that I am certain that it is not running because there are some processes to be closed in the AfterTestRun which remain hanging.

[StuartFergusonVme]

@ninjaboy I did a decent amount of testing on this the other day and the method is getting fired consistently BUT the time the method gets to complete appears to be very short which is why any cleanup code you are doing in there is not getting done.

I proved this by adding a line of trace logging similar to the following

sw.WriteLine("In After Test Run")
// Do a clean up task
sw.WriteLine("Clean up task 1 done")

I could see the "In After Test Run" line written to my log file consistently. However never got the "Clean up task 1 done" log entry.

Is any one from the SpecFlow team looking at this issue as this is causing many people serious issues as shared clean up code is not getting run under .net core.

[SabotageAndi]

We have it on our backlog, but didn't had yet the time for it. So feel free to send us PRs to fix this.

And I still have no idea how to solve this. see my comment https://github.com/techtalk/SpecFlow/issues/1348#issuecomment-453631649. So if somebody has a brilliant idea, I am open for suggestions.

In the Issue https://github.com/Microsoft/vstest/issues/1873 @nvborisenko opened, the VSTest teams explains the behaviour. They kill it after 100ms.

[godrose]

Hi, everyone.
I managed to find a working solution for xUnit.
Disclaimer: It's neither clean nor generic but it works and can be made clean and generic and integrated into SpecFlow.
These are the steps:

• Update to the 3.0.169-beta version of the packages
• Add the bridge class to hook into real AfterAllTestsHaveBeenRunAndTheAssemblyIsAboutToBeUnloaded event:

    [CollectionDefinition("All")]
    public class AssemblyCleanupBridge : ICollectionFixture<AssemblyCleanupBridge>, IDisposable
    {
        void IDisposable.Dispose()
        {
            ScenarioContext.Current.Get<ILifecycleService>().Teardown();
        }
    }

• Decorate the features that require cleanup with the @xunit:collection(All) attribute
• Sit back and enjoy

[SabotageAndi]

I think, this would be possible to add to the code- generator.

[narcis-pv]

Another temporary workaround. Place the file bellow anywhere in your Acceptane Tests projects.
Does the unloading when there are no more features to run

using System.Collections.Generic;
using Willow.Test.Acceptance;
using Xunit;
using Xunit.Abstractions;

[assembly: CollectionBehavior(DisableTestParallelization = true)]
[assembly: TestCollectionOrderer(
    CollectionOrderer.TypeName,
    CollectionOrderer.AssemblyName)]

namespace Your.Test.Acceptance
{
    using System.Linq;
    using StepDefinitions;
    using TechTalk.SpecFlow;

    [Binding]
    public class CollectionOrderer : ITestCollectionOrderer
    {
        public const string TypeName = "Your.Test.Acceptance.CollectionOrderer";

        public const string AssemblyName = "Your.Test.Acceptance";

        private static int _currentSessionScenarios = 0; 

        public IEnumerable<ITestCollection> OrderTestCollections(
            IEnumerable<ITestCollection> testCollections)
        {
            _currentSessionScenarios = testCollections.Count();

            return testCollections;
        }

        [AfterFeature]
        public static void ScenarioTeardown()
        {
            _currentSessionScenarios--;

            if (_currentSessionScenarios <= 0)
            {
                ScenarioContext.Current.Get<ILifecycleService>().Teardown();
            }
        }
    }
}

[SabotageAndi]

The release of SpecFlow 3 is near. We (TechTalk Team) will not have time to fix this before.

[narcis-pv]

@SabotageAndi Honestly, this seems like a MAJOR issue. Better not release it, then release with this kind of broken behavior.

[SabotageAndi]

It will go to the known issues (FYI: @Stephen-Mc).
We have to release because of the VS 2019 release, which happens on April 2nd. Can't change that.

The problem exists "only" for .NET Core. Full Framework is still working. And if you need more than 100 ms.
A Workaround also exists. Do the cleanup before the TestRun, so that you are sure to have a clean environment.

And for NUnit and MSTest we still have no idea how to solve this. Had anybody a flash of inspiration yet?

But SpecFlow is open source. We are happy to accept PR that fixes this issue.

[nvborisenko]

For NUnit: https://github.com/nunit/docs/wiki/SetUpFixture-Attribute

A SetUpFixture outside of any namespace provides SetUp and TearDown for the entire assembly.

[SabotageAndi]

@nvborisenko Do you know if there can be more than one class without namespace with the attribute?

[nvborisenko]

Yes.
NUnitTestProject3.zip

[nvborisenko]

For MSTest: https://www.meziantou.net/2018/02/12/mstest-v2-test-lifecycle-attributes

UnitTestProject2.zip

And cannot define more that one [AssemblyInitialize] attribute.

Actually cannot understand why Specflow generator requires more than one definition of setup attribute.

[SabotageAndi]

@nvborisenko SpecFlow doesn't require multiple setup attributes.
But there are users out there, that are using the test frameworks attributes already and mix them with hooks.
When we now create a new class with them, two are defined and I have no idea what the test runners are doing.

[nvborisenko]

Now I see. In my opinion SpecFlow should work as documented. Hooks should work by default, because of by default clients use SpecFlow feature files as tests "container". But for whom, who mixes feature files and unit tests "container", SpecFlow should leave a comment how to combine several invocations of setup methods into one if unit framework doesn't support multiple setup methods (MsTest). Rare case... Hence SpecFlow generator should take a control.

When we now create a new class with them, two are defined and I have no idea what the test runners are doing.

It can be easily verified, ask me to help to verify runners behavior. In any case, if setup methods of unit frameworks don't work as documented, it's a bug on runner side.

Adding:
Regarding versions, as for me it doesn't matter which SpecFlow release the fix should be included in, in 3.0 or 3.0.1. But this is high priority issue.

[SabotageAndi]

I thought about how this could be implemented.

For getting the additional file into the project, I would create an additional MSBuild task, that add the file to the Compile- ItemGroup.
If possible, I would write the file to the obj- Folder.

The xUnit Code- Generator (https://github.com/techtalk/SpecFlow/blob/master/TechTalk.SpecFlow.Generator/UnitTestProvider/XUnitTestGeneratorProvider.cs & https://github.com/techtalk/SpecFlow/blob/master/TechTalk.SpecFlow.Generator/UnitTestProvider/XUnit2TestGeneratorProvider.cs) need the additional code for the collection definition.

One thing I don't know yet, is how to get a test runner instance without reflection or ScenarioContext.Current. But I will have a look at it in the next days.

So if someone wants to create a PR for this, you have now a starting point to do it.

[gasparnagy]

@SabotageAndi yes, this is a good idea: see #1458 for an easy way to ensure that the AfterTestRun hooks are called.

[gasparnagy]

@SabotageAndi the code to generate the necessary calls for xunit is still have to be added. Shall we keep this issue open until that has been done?

[SabotageAndi]

The close what not on purpose. Sometimes GitHub automations work to good. ;-)

I would left this issue open as long as it works for the 3 frameworks.

[psboies]

I have the same issue right now, just made a dummy test going through all the hooks and the breakpoint in [AfterTestRun] stops for less than half a second and then the execution stops. I can no longer clean my tables after my tests runs since I passed to dotnet core (I was on dotnet framework before).

I used
vs version: VS 15.9.12
net framework: netcoreapp2.2
specflow version: 3.0.213
specflow.tools.msbuild.generation version: 3.0.213
specflow.xunit: 3.0.213
xunit: 2.4.0

[putnap]

I have the same issue. I start some console programs and RestAPI before all tests run so I don't have to start them manually and then I want to dispose of all the processes and kill the host, but [AfterTestRun] is never called or never completes. The workaround https://github.com/techtalk/SpecFlow/issues/1348#issuecomment-460639875 worked for me, except I removed the bit with concurrent tests.

What is the current status on a fix?

[SabotageAndi]

We are working on this currently, because we need it for a new feature.

@godrose
xUnit Collections will not work, because you can only have one collection per test class. Additionally putting all test classes into one collection disables parallel execution, because tests in a test collection aren't executed in parallel. :-/

We have some ideas how to solve this, but aren't yet sure if they are possible.

[SabotageAndi]

We are trying to use the sample code from https://github.com/xunit/samples.xunit/tree/master/AssemblyFixtureExample
We weren't yet successful with it.

But if it works, @gasparnagy we will break SpecFlow.xUnitAdapter as we are creating our own enhanced xUnit test framework executor.

[gasparnagy]

@SabotageAndi No problem (we can even keep the name), but one thing you should be aware of: the xunit extensions are not supported by all test runners. For example with ReSharper, the users will not be able to run the tests... :(

[SabotageAndi]

the xunit extensions are not supported by all test runners. For example with ReSharper, the users will not be able to run the tests... :(

🤦‍♂️

Do you have any other idea, how to fix it then in xUnit?

[gasparnagy]

@SabotageAndi not at the moment. shall we have a call next week to discuss? send me a mail then

[SabotageAndi]

@david1995 and me fixed this issue for MSTest and NUnit and the changes are in 3.1.2-beta.
We will try to fix it also for xUnit in the next sprint.

@gasparnagy I had an idea on the weekend:

for .NET Core, we use the enhanced xunit test framework executor, because R# is using dotnet test in the background for that.
For .NET Framework, we stay with the current Assembly.Unload event.

[gasparnagy]

@SabotageAndi sounds good

[SabotageAndi]

@gasparnagy I did some tests with R# & VS Test Explorer.
In both of them the custom xUnit test framework executor is working.
So we will use it for .NET Core and .NET Framework.

[gasparnagy]

@SabotageAndi good news. do we provide a backup plan for the .net users in case something is wrong with this approach?

[SabotageAndi]

@gasparnagy No

[gasparnagy]

@SabotageAndi brave :) please let me know if there is a package to try before the release, I can also make an intensive test.

[SabotageAndi]

This issue is fixed with the latest 3.1 beta version.

[nathanchere]

Still happens with Specflow 3.1.2-beta and .Net Core 3.0.100 RC1. AfterTestRun doesn't fire at all.

[SabotageAndi]

@nathanchere It is fixed with 3.1.32-beta. 3.1.2-beta is too old.

[nathanchere]

OK thanks, I'll give it a try

[Meligy]

I think I still see the same issue with 3.1.32-beta (.NET Core 2.1 LTS), BeforeTestRun is not called or waited till it finishes.

Currently the workaround for BeforeTestRun is using BeforeFeature, a static variable to indicate that the method has ran so I don't keep calling it multiple times (it's a relatively expensive Elasticsearch setup), and disabling parallelism in XUnit by adding a C# file containing:

using Xunit;

[assembly: CollectionBehavior(DisableTestParallelization = true)]

[SabotageAndi]

@Meligy please provide a complete example where we can see the behaviour.
Our integration tests (https://github.com/techtalk/SpecFlow/blob/master/Tests/TechTalk.SpecFlow.Specs/Features/Hooks/HookSupport.feature) are green for this.

[godrose]

@SabotageAndi
I can confirm proper functionality at both Mac and Windows.
It just works.

[ramgkn]

Hi Experts - TargetFramework is .Net Core 2.1 with below listed Nugets added the code is not hitting the AfterTestRun binding; can somebody advise which version to use.

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.0.0" />
<PackageReference Include="SpecFlow" Version="3.1.32-beta" />
<PackageReference Include="SpecFlow.NUnit" Version="3.1.32-beta" />
<PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.1.32-beta" />

The agenda for me is to flush out extent report after test execution.

Can i have a guidance on this

[SabotageAndi]

@ramgkn It should work with the latest 3.1 packages. If not, please provide a project where we can reproduce the issue.

[ramgkn]

@SabotageAndi - as you can see I used the 3.1 version (precisely 3.1.32) ; is there a specific version that i should be using. If you can let me know which version of 3.1 should i try - that would be helpful

Tried with the latest version of 3.1 - with AfterTestRun not being triggered.

Packages Used:

Project is also attached along with this.
Test_Sol.zip

[ramgkn]

Team - did we get a chance to look onto this; my deliverable is sidelined.

Please suggest a fix.

[SabotageAndi]

@ramgkn Please don't forget that SpecFlow is open source and free, so there are limits to how much time we can invest in tackling issues, especially given that we are a very small team.

I had now a look at the project. I had to add the NUnit3TestAdapter (<PackageReference Include="NUnit3TestAdapter" Version="3.15.1" />) package to be able to execute the tests within Visual Studio/dotnet test. After that there was no problem with the execution of the AfterTestRun hook. The report was created and I added some code to check if it is executed. That happened.
So I can't reproduce your issue that the AfterTestRun hook is not called.

[Meligy]

I think this sort of issues is test runner specific. In my case the issue is with XUnit.

[Meligy]

I tried to create a repro for BeforeTestRun and it sure works fine. Sorry for falsely flagging it.

In my actual project the issue was that the Binding class also had a Scope attribute. The idea was that is should run only if there're any features that have a certain tag. I'd love to be able to do so as a feature request, but it's not the same issue anymore. So I'll stop polluting it.

[Meligy]

Also AfterTestRun seems to work, tested with version 3.1.43-beta, which just happened to be the latest when I created my repro project.

This is how I tested it:

using System;
using System.Threading.Tasks;
using TechTalk.SpecFlow;

namespace specstart.Lifecycles
{
    [Binding]
    public class Lifecycles
    {
        [AfterTestRun]
        public static async Task AfterTests()
        {
            await Task.Delay(TimeSpan.FromSeconds(1));
            throw new Exception("I ran");
        }
    }
}

The error was shown in the test runner, confirming it did run, and waited for the delay I gave it (so it works fine with async).

[santhosh-James]

when 3.1 versions will be released?

[SabotageAndi]

@santhosh-James Preview versions are already available on NuGet.org. For the final release we are working on the last issue (https://github.com/techtalk/SpecFlow/issues/1760) we want to fix before it.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.