Azure-pipelines-tasks: vstest.console.exe failed with return code: 4294967295

Created on 5 May 2017  路  77Comments  路  Source: microsoft/azure-pipelines-tasks

I'm getting the following error intermittently (approx 10-20% of CI build/tests) with the Visual Studio test runner (version 2.0.30):

Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe failed with return code: 4294967295

My solution is VS2017 with .Net Core and Xunit test projects. I'm using the Hosted VS2017 agent.

Not only is it intermittent, but when it does fail, it is not in the same test project. Never are there failed tests.

What is return code 4294967295?

6_Test Assemblies .Tests.dll.txt

Test

Most helpful comment

We finally got a repro which captured the offending process from the event log line above. We know exactly what is killing msbuild/vstest now. We're following up right now with an image update. It takes awhile to rollout but good news.

This one was especially tough because the size, length and volatility of our tool would protect it from repro'ing. Sorry for the inconvenience.

All 77 comments

I also experience this lately, nothing to note with debug logging on, it just quits.
I also have it with some msbuild tasks (same idea, .net core in solution, xunit, hosted vs2017 agent).

Started happening since Tuesday

Same here. Just randomly started happening. It is a major blocker/annoyance for us.

@AbhitejJohn can you check

Tagging @smadala . Can someone share the diagnostic log too please(with System.Debug "true")?

Two attached, both debug set to true, same project, failed at different location with no 'reason'. Couldn't find anything significant.

logs_failed_msbuild.zip
logs_failed_vstest.zip

@janpieterz, @daniefvh To run .net core unit tests choose vstest.console.exe from "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe".
in vstest task (Execution options -> Specific location). In "Other console options" pass /diag:log.txt for diag information. please attach log.txt and log.*.host.txt.

@marcwmiller Please enable /diag:log.txt and attach log.txt and log.*.host.txt.

Seems for vstest to have something to do with code coverage. Where do I get those log.txt files from?
Also any thoughts about the msbuild one?

logs_639.zip

Seems for vstest to have something to do with code coverage.

/EnableCodeCoverage yet to support in .net core. Tracking here.

Pass /Framework:FrameworkCore10 arg in "Other console options" for running .net core tests.
log.txt is relative path from current working directory, you can pass absolute path too.

No vstest task or tests running in msbuild one. It was failing in Build solution, @acesiddhu Can you please tag Build solution task folks?

We are adding respective folks to investigate into this issue. Seems like both msbuild and vstest are failing with same error code.

@smadala - I have enabled /diag:log.txt as requested and will attach log files after next failure.

@smadala - As luck would have it (I guess) I got the error first time running CI build after enabling diagnostic logs. Please find log files attached.
6_Test Assemblies .Tests.dll.txt
log.host.17-05-08_12-07-31_43223_4.txt
log.host.17-05-08_12-07-55_11017_4.txt
log.host.17-05-08_12-07-57_53449_4.txt
log.host.17-05-08_12-08-02_33688_4.txt
log.host.17-05-08_12-08-07_38614_9.txt

@marcwmiller

DefaultEngineInvoker: ParentProcess '1244' Exited in log.host.17-05-08_12-08-07_38614_9.txt

Looks vstest.console.exe exiting abnormally, Can please share log.txt which will be in current directory of vs test task command runs.
vstest.console.exe full dump by following instructions here.

@smadala - I think I've correctly updated the Copy Files task to capture the log.txt file (we'll have to wait for another failure).

But I'm not sure about your last request for logs - I'm using the Hosted VS2017 agent and cannot perform instructions provided in the link. Please advise.

This seems widespread, we have this error in approx. every 3rd build or so for several projects. Sometimes consecutive test runs have this, then it works for a couple of builds without any apparent change on our side. This happens only in VSO i haven't been able to repro locally.

Solution is VS 2017 .Net 4.6, Nunit 3.7, Hosted VS2017 agent

@marcwmiller are you able to get log.txt file?
@nigurr Is there a way we can get dmp from Hosted server?

Just a note that we are not running .NET core. I will try getting logs for you.

@marcwmiller can you please send your VSTS account and build details to devops_tools at microsoft dot com
I will be investigating into this asap.

FYI, we have the same problem with .net 4.6, VS2017, 2017 host, MsTest v2. And it happens to the build step as well, about as frequent.

Hi,

We have created a Live Site Incident for the same and actively working on this.

@nigurr - I am unable to email the information requested. My email is returned with the following error:

The groups only accept messages from people in their organization or on their allowed senders list, and your email address isn't on the lists

@marcwmiller replied to you

does everyone on this thread have .NET Core projects in their solution? Also can you all post your account URLs?

@chrisrpatterson - My solution contains .NET Core projects. I have shared account URL with @nigurr and can email to you if you'd like.

@marcwmiller can you simply ping your account and tell me what it resolves to.

Jup, contains .NET core, account url is intreba.visualstudio.com, one of affected projects Arke.
@nigurr has access to this already.

@chrisrpatterson tfsprodcus1.app.visualstudio.com [104.43.203.175]

Another question, what about TypeScript?

@chrisrpatterson - nope.

Jup, all failing builds have TypeScript projects

Typescript compile should be disable though, at least on the msbuild

failing projec

Met vriendelijke groet,

Jan-Pieter Zoutewelle

@janpieterz did you recently change or update the typescript version?

@chrisrpatterson wouldn't say recently, moved to 2.2.2 the 9th and 19th of April, but to be honest the history seems to stop after a bit (probably a setting) and I can't trace back when these errors started.

Maybe something that changed is that I've added GitVersionTask to both solutions recently, though I haven't found anything in the logs suggesting that fails (though neither anything else).

@janpieterz How are you referencing TypeScript? If you are using NuGet can see check your project files to see if you happen to be importing targets from both the MSBuildExtensions location

and the NuGet

And remove the one from MSBuildExtensions.

I'm using npm, no reference to typescript in csproj or in packages. Only references in csproj to Typescript are:

<TypeScriptEnabled>false</TypeScriptEnabled>
<TypeScriptCompileBlocked>true</TypeScriptCompileBlocked>

@janpieterz sorry I am working on a couple of issues at the moment. Are you also seeing random failures in MSBuild with a return code of -1 or are you only seeing the vstest.console failure?

No worries! I am seeing both, so msbuild sometimes fails with -1, sometimes vstest.console fails with some number.

@janpieterz which version of the GitVersion task are you using?

4.0.0-beta0011
Test projects don't reference this though

but the build projects do. This feels like some sort of assembly mismatch error that is not bubbling through to me. I am just to exactly sure where, Thanks for the information.

Interesting, I did have some issue with System.Runtime being assembly redirect, on runtime, which was caused by GitVersionTask. Test project doesn't reference this, but who knows.

I'm more than happy to give you access and you can kick off as many builds as you want (it's night time here). If so, you might want to disable the npm/webpack tasks when doing since they slow it down a lot.

sure is your account MSA or AAD backed? if MSA backed you can add [email protected]. If AAD you will probably need to invite me to your tenant so it would be [email protected].

The System.Runtime redirect I have seen before. In that case it seemed adding some Roslyn analyzers caused it.

@janpieterz can you make me a PCA so I could register an agent on a debug pool please

AAD backed, invited you but doesn't seem to want to add you as a PCA as of yet
EDIT: Seems you're PCA now!

Just a quick note that we partially resolved this problem by not using GitVersion during MSBuild but rather moving it to an extra Build Task in VSTS. The Build passes more regularly now, but we are still seeing occasional failures.

If I had to place a bet on it, I'd say the Agent is running out of resources during the build.

@daniefvh It is not the agent that is failing, it is MSBuild. The machines have 7GB of memory so resources seem unlikely but possible. There is an option on the task to switch to 64bit MSBuild. Which version of GitVersion where you using?

@janpieterz thanks I will see what I can figure out.

@chrisrpatterson we tried upgrading to the latest version and that still failed.

@chrisrpatterson Should I look into the (new) _TF209010_ error and/or create an issue with GitVersion or here? Seems that is unrelated to what's going on here (though also pretty weird).

Any thoughts about the msbuild/vstest failures?

@janpieterz take a look at the clone of your definition in the project. I made a small change and it potentially points to a logger we hook up. For now you could potentially unblock yourself by adding a similar change. Now we have not changed that logger in a couple of years so it appears their might be some new interaction with MSBuild we need to look into.

@janpieterz while I am working through debugging this issue I am going to temporarily increase the number of hosted agents in your account so I don't block your normal work

@chrisrpatterson I see, interesting. I'll wait for you to finish, only need to build correctly once for this security advisory.

@janpieterz you have 5 agents right now so you should be able to build.

Jup, I noticed when both ran at the same time. Cheers!

We don't have any .NET Core / TypeScript stuff.

We went back to the non-2017 host and all is well. We can't open it up for experimenting with builds since we have external dependencies (database for end2end tests).

Like I said in a previous post, we see builds failing on either the build step _or_ the test step (not both).

@janpieterz thank you for your patience. We are working with the MSBuild team to figure out what additional diagnostics we can gather. It seems the failure rate is about 20% and the MSBuild failure happens at a different place in the project every time.

@chrisrpatterson no worries, happy to help! If you want to speed up your build test, you could disable webpack and npm, they shouldn't bother the msbuild and test!

If you'd like me to add any other MS employee just send me an email or ping me here, besides that I'm enjoying the notifications for the builds in our Slack 馃槃

@janpieterz we have a repro from a basic solution https://github.com/sgrebnov/Windows-appsample-coloringbook now. You can remove me from your account if you want. For a temporary work around if you add a second vsbuild task right after the first one and set the first to continue on error it seems to complete successfully 100% of the time.

@chrisrpatterson Great to hear, just removed you. In case you need to get back in just let me know, otherwise good luck hinting, and cheers for that temp workaround!

What do I do with those agents btw?

@janpieterz the extra hosted agents should disappear. You can also delete the agent in the default pool.

we are also seeing about 50% of builds fail on running tests with error

Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 4294967295

similar to @nvankesteren we are using the Hosted VS2017 agent, due to this we also needed to change the build to use the Version 2.* preview version of the Visual Studio Test step

The current situation is intolerable, but not sure what to do, is the error we are seeing the same issue? is this a known issue that is being fixed or do we back out and stop using the Hosted VS2017 agent? we started using the VS2017 agent because the default one started giving us c# compilation errors randomly due to new c# language features.

We are actively working very hard on it. We apologize for the inconvenience this is causing. What we know is the process does not crash - both msbuild and vstest simply exits with -1 or maxint. It only happens on the first run on a fresh OS (that is the hosted build scenario since every build is a new machine reimage). For example, you can put a second task of build or test in the same job with the first being continue on error and it will always succeed.

The workaround is to use a private agent (your own VM / machine).

A new image which fixes/mitigates the issue should be rolling across scale units throughout the day. Chime in throughout the weekend and early in the week if it's addressed for you. We will be keeping an eye on it.

@bryanmacfarlane - sweet. Thanks to all involved with issue resolution.

got another failure just now
Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 4294967295

@cob999, can you please send your VSTS account and build details to devops_tools at microsoft dot com
and send your build log?

@nigurr I've sent that on, got a bounce message saying Couldn't deliver to the following groups: but i'm hoping the original email got to you

@nigurr - I've received the error twice today :-(

Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe failed with return code: 4294967295
Vstest exited with return code: 1.

we're copying a new image with gflags to figure out what's wacking the process (it's not crashing, it's not exiting). the previous mitigation appears to reduce the likelyhood but still hapenning ... we're on it.

A new image is rolled out with additional logging enabled to determine more information. When msbuild.exe/vstest.console.exe exits, info will be written to the application event log with the exit code and if another process killed it, the name of the process that killed msbuild.exe/vstest.console.exe.

Please add the below inline powershell to grab the info from the application event log once. Please make sure you select "Always Run" in the task so that any previous errors in the build will still continue to execute this and dumps the logs.

Get-EventLog -LogName Application -After ([datetime]::now.addhours(-1)) -Source 'Process Exit Monitor' | % { "EventID: $($_.EventID)" ; "TimeGenerated: $($_.TimeGenerated)" ; "Message: $($_.Message)" ; "" }

Can MS repro this without outside projects? My team no longer has the resources to keep testing this and have stepped back a version. I'm watching this space to see if the issue is resolved.

We finally got a repro which captured the offending process from the event log line above. We know exactly what is killing msbuild/vstest now. We're following up right now with an image update. It takes awhile to rollout but good news.

This one was especially tough because the size, length and volatility of our tool would protect it from repro'ing. Sorry for the inconvenience.

@bryanmacfarlane - when do you anticipate the roll out to be complete? I'm still receiving error:

Ran for 3.7 minutes (Hosted VS2017), completed at Fri 05/26/2017 05:43 PM
Request Summary
Request 309 Marc Miller Completed
Summary
| Build
2 error(s), 3 warning(s)
Build - 2 error(s), 3 warning(s)
Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe failed with return code: 4294967295
Vstest exited with return code: 1.

@thechrisjohnson should this have deployed everywhere by now?

@marcwmiller please send ersciple the name of your account (at microsoft com).

@ericsciple - done.

Was this page helpful?
0 / 5 - 0 ratings