Specflow: MSB4062 when running build on Azure DevOps Hosted Ubuntu 16.04

[tistocks]

/home/vsts/.nuget/packages/specflow.tools.msbuild.generation/3.1.21-beta/build/SpecFlow.Tools.MsBuild.Generation.targets(78,5): error MSB4062: The "SpecFlow.Tools.MsBuild.Generation.GenerateFeatureFileCodeBehindTask" task could not be loaded from the assembly /home/vsts/.nuget/packages/specflow.tools.msbuild.generation/3.1.21-beta/build/../tasks/netcoreapp2.0/SpecFlow.Tools.MsBuild.Generation.dll. Assembly with same name is already loaded Confirm that the declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask.

SpecFlow Version:

• [X] 3.0
• [ ] 2.4
• [ ] 2.3
• [ ] 2.2
• [ ] 2.1
• [ ] 2.0
• [ ] 1.9

Used Test Runner

• [ ] SpecFlow+Runner
• [X] MSTest
• [ ] NUnit
• [ ] Xunit

Version number:

==============================================================================
Task         : .NET Core
Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command
Version      : 2.156.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
==============================================================================

Project Format of the SpecFlow project

• [ ] Classic project format using packages.config
• [ ] Classic project format using <PackageReference> tags
• [ ] Sdk-style project format

.feature.cs files are generated using

• [X] SpecFlow.Tools.MsBuild.Generation NuGet package
• [ ] SpecFlowSingleFileGenerator custom tool

Visual Studio Version

• [ ] VS 2019
• [ ] VS 2017
• [ ] VS 2015
• [X] NA

Enable SpecFlowSingleFileGenerator Custom Tool option in Visual Studio extension settings

• [ ] Enabled
• [X] Disabled

Are the latest Visual Studio updates installed?

• [X] Yes
• [ ] No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

• [ ] >= .NET 4.5
• [ ] before .NET 4.5
• [X] .NET Core 2.0
• [ ] .NET Core 2.1
• [ ] .NET Core 2.2
• [ ] .NET Core 3.0

Test Execution Method:

• [ ] Visual Studio Test Explorer
• [ ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
• [X] Command line – PLEASE SPECIFY THE FULL COMMAND LINE
  [command] /opt/hostedtoolcache/dncs/2.2.103/x64/dotnet test /home/vsts/work/1/s/Landorphan.Ioc.ServiceLocation.XPlat.sln --logger trx --results-directory /home/vsts/work/_temp

<SpecFlow> Section in app.config or content of specflow.json

N/A

Repro Project

Source: https://github.com/landorphan/service-location/tree/tistocks-Instrumentation-NextSteps
Build Pipeline Result: https://dev.azure.com/landorphan/Service-Location/_build/results?buildId=759

Issue Description

Steps to Reproduce

run dotnet test with project specified off of the prescribed branch on Azure DevOps hosted Ubuntu 16.04 build agent.

[tistocks]

Windows and OSX builds work fine.

[tistocks]

Is anyone looking into this? It's been sitting dormant for almost a month.

[SabotageAndi]

We worked on building SpecFlow itself on Linux and MacOS.
Work is not yet complete, but the Linux build is working.

PR is here: https://github.com/techtalk/SpecFlow/pull/1709

We are using the MSBuild task itself in our build of SpecFlow.

[zplan]

We see this error a lot lately on our Windows build machines.
I think it got worse when we switched Test Targetframework from FullFramework to dotnet Core.

Is the root cause known or is there a workaround?

[SabotageAndi]

@zplan We never saw this error on Windows yet. This was only happening on Linux.

[zplan]

@SabotageAndi: We have this on our TFS2017 build agents and I think I saw it on the "Azure Pipelines" for Specflow when I did my last PR, there also the Windows build was failing because of this issue.

[SabotageAndi]

@zplan there are some builds failing at the moment for us, but this is because of build order problem. There a file is missing. This error about that an assembly is loaded twice.

[zplan]

Ok.

But here are two examples from us from two different builds, running on Windows build agents:

1. example
   2020-02-05T01:50:45.3661496Z C:\Users\abc\.nuget\packages\specflow.mstest\3.1.67\build\SpecFlow.MsTest.targets(34,5): error MSB4062: The "SpecFlow.Tools.MsBuild.Generation.ReplaceTokenInFileTask" task could not be loaded from the assembly C:\Users\abc\.nuget\packages\specflow.tools.msbuild.generation\3.1.67\build\..\tasks\netcoreapp2.0\SpecFlow.Tools.MsBuild.Generation.dll. Assembly with same name is already loaded Confirm that the <UsingTask> declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask. [Z:\w\184\s\Tests\IntegrationTests.csproj]
2. example
   2020-02-05T02:21:07.9412350Z C:\Users\abc\.nuget\packages\specflow.mstest\3.1.82\build\SpecFlow.MsTest.targets(36,5): error MSB4062: The "SpecFlow.Tools.MsBuild.Generation.ReplaceTokenInFileTask" task could not be loaded from the assembly C:\Users\abc\.nuget\packages\specflow.tools.msbuild.generation\3.1.82\build\..\tasks\netcoreapp2.0\SpecFlow.Tools.MsBuild.Generation.dll. Assembly with same name is already loaded Confirm that the <UsingTask> declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask. [Z:\w\98\s\Tests\AcceptanceTests.csproj]

[SabotageAndi]

@zplan do you have 2 projects in that solution that have the SpecFlow.Tools.MsBuild.Generation NuGet package added?

[zplan]

@SabotageAndi : Yes, that is the case, I have multiple test projects in one solution. each of them contains:

<PackageReference Include="SpecFlow" Version="3.1.82" />
<PackageReference Include="SpecFlow.MsTest" Version="3.1.82" />
<PackageReference Include="SpecFlow.Tools.MsBuild.Generation" Version="3.1.82" />

[SabotageAndi]

Ok, I think I found the issue.
We are importing the MSBuild tasks in your case multiple times: https://github.com/SpecFlowOSS/SpecFlow/blob/master/SpecFlow.Tools.MsBuild.Generation/build/SpecFlow.Tools.MsBuild.Generation.props#L75

[zplan]

Can this be fixed on SpecFlow side or is there a possible workaround?

[SabotageAndi]

I think setting a PropertyGroup in the tasks file and checking this on the import with a condition is all that is needed.
Happy to review PRs to fix this. It's probably faster.

We (SpecFlow Team) can take a look at it earliest in 2-3 weeks and put it in the next sprint.

[zplan]

We (SpecFlow Team) can take a look at it earliest in 2-3 weeks and put it in the next sprint.

Timeline is fine for me

[zplan]

We (SpecFlow Team) can take a look at it earliest in 2-3 weeks and put it in the next sprint.

Is this issue still on the roadmap/backlog?
I face this issue a lot on the build agents.

[SabotageAndi]

@zplan I couldn't reproduce the issue, but I now made the changes that would fix it. If the release pipeline runs through smoothly, I will release this change to nuget.org and you can try it yourself.

[SabotageAndi]

@zplan Version with the changes are now on nuget.org. Please try them and report them.

[zplan]

@SabotageAndi : Thanks for the info.

Unfortunately the problem is not solved. I got this issue from our TFS build server after the update:

2020-03-18T01:31:12.8063871Z C:\Users\xyz\.nuget\packages\specflow.mstest\3.1.89\build\SpecFlow.MsTest.targets(36,5): error MSB4062: The "SpecFlow.Tools.MsBuild.Generation.ReplaceTokenInFileTask" task could not be loaded from the assembly C:\Users\xyz\.nuget\packages\specflow.tools.msbuild.generation\3.1.89\build\..\tasks\netcoreapp2.0\SpecFlow.Tools.MsBuild.Generation.dll. Assembly with same name is already loaded Confirm that the <UsingTask> declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask. [Z:\w\35\s\Tests\MyTests.csproj]

[SabotageAndi]

WTH????!?!?

Which .NET Core SDK version are you using? I am asking because of #1912.

[zplan]

WTH????!?!?

Which .NET Core SDK version are you using? I am asking because of #1912.

3.0.100

1912.

Figured that also out yesterday :)

[andrebires]

The same error is occurring on Windows when using the 3.1.203 SDK version. The error doesn't occur in 3.1.100.

Looks like the 3.1.200+ versions changed the behavior or MSBuild:

MSBuild now loads tasks with AssemblyLoadContext isolation, allowing tasks that have conflicting dependencies to be loaded in the same process microsoft/msbuild#4916.
.NET Core SDK 3.1.200 can build C++ Common Language Infrastructure (CLI) libraries.

https://github.com/dotnet/core/blob/master/release-notes/3.1/3.1.2/3.1.200-sdk.md#notable-changes-in-31200-sdk

[SabotageAndi]

@andrebires yes, the MSBuild behavior changed. That's what is written in https://github.com/SpecFlowOSS/SpecFlow/issues/1912

[andrebires]

Ops, I'm sorry, I didn't saw the related issue.

[Ukrainis]

Today started to get same error on 2.1.403 NetCore. On TeamCity CI server. Last SpecFlow version. Build happens on Windows machine.
P.S. Fixed by updating SpecFlow and related packages to last beta version. Thanks.

[zplan]

After the update to version 3.3.30 I see the following error randomly which seems related to this issue:

.nuget\packages\specflow.mstest\3.3.30\build\SpecFlow.MsTest.targets(36,5):
error MSB4062: The "SpecFlow.Tools.MsBuild.Generation.ReplaceTokenInFileTask" task could not be loaded from the assembly .nuget\packages\specflow.tools.msbuild.generation\3.3.30\build..tasks\netcoreapp2.1SpecFlow.Tools.MsBuild.Generation.dll
Assembly with same name is already loaded Confirm that the declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask. [\MyTestProject.csproj]

[AndresC46]

Seeing this issue also on VS 2019 for an C# NUnit SpecFlow + Runner project. I made the mistake of forgetting to install the SpecFlow.Tools.MsBuild.Generation until my first compile. After installing it I consistently get the below error. Any work around to fix this? I might try to uninstall all NuGet packages and then add back

Severity Code Description Project File Line Suppression State
Error MSB4018 The "GenerateFeatureFileCodeBehindTask" task failed unexpectedly.
System.Exception: Unit test Provider already set.
at TechTalk.SpecFlow.UnitTestProvider.UnitTestProviderConfiguration.UseUnitTestProvider(String unitTestProviderName)
at TechTalk.SpecFlow.NUnit.Generator.SpecFlowPlugin.GeneratorPlugin.Initialize(GeneratorPluginEvents generatorPluginEvents, GeneratorPluginParameters generatorPluginParameters, UnitTestProviderConfiguration unitTestProviderConfiguration) in D:\a\1\s\Plugins\TechTalk.SpecFlow.NUnit.Generator.SpecFlowPlugin\GeneratorPlugin.cs:line 14
at TechTalk.SpecFlow.Generator.GeneratorContainerBuilder.LoadPlugins(ObjectContainer container, GeneratorPluginEvents generatorPluginEvents, UnitTestProviderConfiguration unitTestProviderConfiguration, IEnumerable1 generatorPlugins) at TechTalk.SpecFlow.Generator.GeneratorContainerBuilder.CreateContainer(SpecFlowConfigurationHolder configurationHolder, ProjectSettings projectSettings, IEnumerable1 generatorPluginInfos, IObjectContainer parentObjectContainer)
at SpecFlow.Tools.MsBuild.Generation.WrappedGeneratorContainerBuilder.BuildGeneratorContainer(SpecFlowConfigurationHolder specFlowConfigurationHolder, ProjectSettings projectSettings, IReadOnlyCollection`1 generatorPluginInfos, IObjectContainer rootObjectContainer)
at SpecFlow.Tools.MsBuild.Generation.GenerateFeatureFileCodeBehindTaskExecutor.Execute()
at SpecFlow.Tools.MsBuild.Generation.GenerateFeatureFileCodeBehindTask.Execute()
at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
at Microsoft.Build.BackEnd.TaskBuilder.d__26.MoveNext() NUnitTestMongo C:\Users\Andres.nuget\packages\specflow.tools.msbuild.generation\3.3.30\buildSpecFlow.Tools.MsBuild.Generation.targets 93

[gbirchmeier]

I also see @zplan's error, when I upgrade SpecFlow packages past 3.1.52-beta.

• I see it when I build with dotnet build, but not when I build inside VS 2019.

• The relevant packages are: SpecFlow, SpecFlow.NUnit, SpecFlow.Tools.MsBuild.Generation

  • version 3.1.52-beta - I DO NOT see the error, everything works
  • version 3.1.62 (the next published nuget) - I see the error
  • version 3.3.57 (the latest nuget at this time) - I see the error

So that seems to imply that this bug was introduced between 3.1.52 and 3.1.62.