Msbuild: MSB4166: Pipe is broken when building with .net core SDK 3.0 preview 7

[hillin]

Steps to reproduce

Not quite sure, this issue occurs randomly when building netcoreapp3.0 projects with .net core 3.0 SDK preview 7.

Expected behavior

Build correctly.

Actual behavior

On a big solution (ours has 70+ projects), it's not uncommon to see one or two projects (and tens of projects depending on them) being failed to build due to a node of msbuild crashes. However, the problem occurs randomly. By random I mean how many of these errors and on which projects they occur is totally unpredictable. The build process simply stops when a node crashes, fails the projects currently being built.

The error message complains:

Child node "_node-id_" exited prematurely. Shutting down. Diagnostic information may be found in files in "%APPDATA%\Local\Temp\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.

The MSBuild_*.failure.txt files has either content of the following two:

UNHANDLED EXCEPTIONS FROM PROCESS _process-id_:
_date-time_
System.IO.IOException: Pipe is broken.
at System.IO.Pipes.PipeStream.CheckWriteOperations()
at System.IO.Pipes.PipeStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.RunReadLoop(Stream localReadPipe, Stream localWritePipe, ConcurrentQueue`1 localPacketQueue, AutoResetEvent localPacketAvailable, AutoResetEvent localTerminatePacketPump)

or

UNHANDLED EXCEPTIONS FROM PROCESS _process-id_:
_date-time_
System.IO.IOException: Pipe is broken.
at System.IO.Pipes.PipeStream.WinIOError(Int32 errorCode)
at System.IO.Pipes.PipeStream.BeginWriteCore(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback callback, Object state)
at System.IO.Pipes.PipeStream.WriteCore(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Pipes.PipeStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Microsoft.Build.BackEnd.NodeEndpointOutOfProcBase.RunReadLoop(Stream localReadPipe, Stream localWritePipe, ConcurrentQueue`1 localPacketQueue, AutoResetEvent localPacketAvailable, AutoResetEvent localTerminatePacketPump)

It's worth noting that these failure log files, with the same content, are also generated by SDK preview 6, they just weren't shown as errors.

Environment data

msbuild /version output: 16.200.19.32702

OS info: Windows 10 Version 1903

If applicable, version of the tool that invokes MSBuild (Visual Studio, dotnet CLI, etc):
.NET Core SDK (reflecting any global.json):
Version: 3.0.100-preview7-012821
Commit: 6348f1068a

[livarcocc]

Is this happening consistently to you? Any chance you could share a repro repo on github with us? These failures are usually very hard to track down.

[alexandredsimoes]

Is this happening consistently to you? Any chance you could share a repro repo on github with us? These failures are usually very hard to track down.

Same here. WPF .net core 3 project. .NET Core 3 preview 7.

[alexandredsimoes]

Is this happening consistently to you? Any chance you could share a repro repo on github with us? These failures are usually very hard to track down.

Same here. WPF .net core 3 project. .NET Core 3 preview 7.

My fault here.
One project was referencing another project that was already being copied to another directory via xcopy in PostBuildEvent.

[rainersigwald]

@alexandredsimoes We're interested in any repro of an MSBuild node crash, no matter what the build is doing. Your scenario might result in a build failure, but it should never cause this "pipe is broken" error.

[alexandredsimoes]

@alexandredsimoes We're interested in any repro of an MSBuild node crash, no matter what the build is doing. Your scenario might result in a build failure, but it should never cause this "pipe is broken" error.

In my case, project 1 references project 2 (as a "ProjectReference Include" and not as an "Reference Include"). Since project 2 is copied via xcopy in the AfterBuildEvent event, I think it was causing the problem when project 1 was built.
I changed the reference to project 2 to "Reference Include" (in the folder where xcopy sends the assembly) and everything is ok now.

Sorry for my english. :)

[rainersigwald]

I'm glad you fixed your problem. But I still want to know how to reproduce the failure.

MSB4166 should never happen. It is always a bug in MSBuild. I'd like to fix that bug, so I'd like to get more information about your case if we can.

[StefanOssendorf]

@rainersigwald I can reproduce the bug with each build at work. I'll ask my superior if we can share our solution in private (since it's our product :-D).

[alexandredsimoes]

I'm glad you fixed your problem. But I still want to know how to reproduce the failure.

MSB4166 should never happen. It is always a bug in MSBuild. I'd like to fix that bug, so I'd like to get more information about your case if we can.

I can share my repository if you want.

[hillin]

@livarcocc yes it happens consistently. I can add you to our private azure devops repo. You can write to me at [email protected] so we can discuss about that.

[livarcocc]

@alexandredsimoes yes, we would love to try your repo. Let me know if you want to share it in private and I can share an email where you can send the info to.

[alexandredsimoes]

@alexandredsimoes yes, we would love to try your repo. Let me know if you want to share it in private and I can share an email where you can send the info to.

Ok, send your e-mail address to send the repo.

[hillin]

Some more observations:

• We have many WPF projects and one ASP.NET project mixed in one solution, all of which are netcoreapp3.0 projects
• Building the solution yields _pipe is broken_
• Building the ASP.NET project alone yields a _StackOverflowException_
• Remove the ASP.NET project from solution, and the solution builds correctly.

Fortunately the ASP.NET project is optional and somewhat obsolete to us, so we removed it and negotiated peace with MSBuild. So it seems obvious that in our case the ASP.NET project is the culprit - although the underlying mechanism is yet to be discovered.

A naive attempt to put a WPF project and an ASP.NET project together did not reproduce the issue.

[rainersigwald]

@hillin that sounds a lot like you might be hitting aspnet/AspNetCore#12693. Can you try adding the ASP.NET project back to your solution and applying the workaround mentioned in that bug to confirm?

[hillin]

Thanks @rainersigwald ! That's exactly the problem, and the workaround worked for me.

[hillin]

I'm closing this issue. @StefanOssendorf @alexandredsimoes Feel free to ask me to reopen it if you were in a different scenario.

[StefanOssendorf]

Did work for us, too 👍