Azure-pipelines-tasks: msbuild.exe fails randomly with exit code -1

[rposener]

Last couple days, we are seeing random errors using Hosted VS2017 build agent tasks. Our build includes all of the following:

• Sql Proeject (dacpac)
• asp.net mvc4 project
• asp.net web forms project
• asp.net core web project

failures are random as to which it fails on, but always just seem to be abrupt immediate return of -1.

exact same source has shown the following failures (in different retries of the build):

(on the sql project)
2017-05-09T14:39:27.0162445Z PrepareForBuild: 2017-05-09T14:39:27.0542610Z Creating directory "bin\Release\". 2017-05-09T14:39:27.0552617Z Creating directory "obj\Release\". 2017-05-09T14:39:27.0562622Z CoreCompile: 2017-05-09T14:39:27.0562622Z C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Roslyn\csc.exe /noconfig /nowarn:1701,1702 /nostdlib+ /errorreport:prompt /warn:4 /define:TRACE /highentropyva+ /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\mscorlib.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Data.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Xml.dll" /debug:pdbonly /optimize+ /out:obj\Release\PracticeEngine.Database.dll /ruleset:"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Static Analysis Tools\\Rule Sets\MinimumRecommendedRules.ruleset" /subsystemversion:6.00 /target:library /warnaserror- /utf8output "C:\a\_temp\.NETFramework,Version=v4.6.1.SqlClrAttributes.cs" 2017-05-09T14:39:27.0662663Z Using shared compilation with compiler from directory: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Roslyn 2017-05-09T14:39:28.5125683Z SqlBuild: 2017-05-09T14:39:28.5125683Z Creating a model to represent the project... 2017-05-09T14:39:30.0411754Z Loading project references... 2017-05-09T14:39:31.4858210Z Loading project files... 2017-05-09T14:39:35.6346146Z ##[error]Process 'msbuild.exe' exited with code '-1'.

(on the web forms project)
017-05-09T10:44:55.3929685Z Adding file (C:\a\1\s\Releases\9.6\Classic\Websites\PracEng\obj\Release\Package\PackageTmp\bin\System.Web.Razor.dll). 2017-05-09T10:44:55.4189665Z Adding file (C:\a\1\s\Releases\9.6\Classic\Websites\PracEng\obj\Release\Package\PackageTmp\bin\System.Web.WebPages.Deployment.dll). 2017-05-09T10:44:55.4229665Z Adding file (C:\a\1\s\Releases\9.6\Classic\Websites\PracEng\obj\Release\Package\PackageTmp\bin\System.Web.WebPages.dll). 2017-05-09T10:44:55.4409651Z Adding file (C:\a\1\s\Releases\9.6\Classic\Websites\PracEng\obj\Release\Package\PackageTmp\bin\System.Web.WebPages.Razor.dll). 2017-05-09T10:44:55.4439648Z Adding file (C:\a\1\s\Releases\9.6\Classic\Websites\PracEng\obj\Release\Package\PackageTmp\bin\Telerik.Web.UI.dll). 2017-05-09T10:44:56.7588905Z Adding file (C:\a\1\s\Releases\9.6\Classic\Websites\PracEng\obj\Release\Package\PackageTmp\bin\Telerik.Web.UI.Skins.dll). 2017-05-09T10:44:57.4008408Z ##[error]Process 'msbuild.exe' exited with code '-1'.

(on the mvc 4 project)
2017-05-09T10:50:10.0370603Z Done Building Project "C:\a\1\s\Releases\9.6\PE.Storage.Interfaces\PE.Storage.Interfaces.csproj" (default targets). 2017-05-09T10:50:11.1260927Z CoreResGen: 2017-05-09T10:50:11.1260927Z "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.1 Tools\resgen.exe" /useSourcePath /r:C:\a\1\s\Releases\9.6\packages\AutoMapper.4.1.1\lib\net45\AutoMapper.dll /r:C:\a\1\s\Releases\9.6\packages\CsvHelper.2.13.5.0\lib\net40-client\CsvHelper.dll /r:C:\a\1\s\Releases\9.6\packages\EntityFramework.6.1.2\lib\net45\EntityFramework.dll /r:C:\a\1\s\Releases\9.6\packages\EntityFramework.6.1.2\lib\net45\EntityFramework.SqlServer.dll /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Microsoft.CSharp.dll" /r:C:\a\1\s\Releases\9.6\packages\Unity.3.5.1404.0\lib\net45\Microsoft.Practices.Unity.Configuration.dll /r:C:\a\1\s\Releases\9.6\packages\Unity.3.5.1404.0\lib\net45\Microsoft.Practices.Unity.dll /r:C:\a\1\s\Releases\9.6\packages\Unity.3.5.1404.0\lib\net45\Microsoft.Practices.Unity.RegistrationByConvention.dll /r:C:\a\1\s\Releases\9.6\packages\Microsoft.Web.Infrastructure.1.0.0.0\lib\net40\Microsoft.Web.Infrastructure.dll /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\mscorlib.dll" /r:C:\a\1\s\Releases\9.6\packages\Newtonsoft.Json.7.0.1\lib\net45\Newtonsoft.Json.dll /r:C:\a\1\s\Releases\9.6\PE.CacheProvider.Interfaces\bin\Release\PE.CacheProvider.Interfaces.dll /r:C:\a\1\s\Releases\9.6\PE.Dal.Interfaces\bin\Release\PE.Dal.Interfaces.dll /r:C:\a\1\s\Releases\9.6\PE.Data\bin\Release\PE.Data.dll /r:C:\a\1\s\Releases\9.6\PE.DoubleMetaphone\bin\Release\PE.DoubleMetaphone.dll /r:C:\a\1\s\Releases\9.6\PE.References\bin\Release\PE.References.dll /r:C:\a\1\s\Releases\9.6\PE.ReportEngine.Interfaces\bin\Release\PE.ReportEngine.Interfaces.dll /r:C:\a\1\s\Releases\9.6\PE.Services.Interfaces\bin\Release\PE.Services.Interfaces.dll /r:C:\a\1\s\Releases\9.6\PE.Storage.Interfaces\bin\Release\PE.Storage.Interfaces.dll /r:C:\a\1\s\Releases\9.6\PE.TaskRunner.Interfaces\bin\Release\PE.TaskRunner.Interfaces.dll /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.ComponentModel.DataAnnotations.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Configuration.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Core.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Data.DataSetExtensions.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Data.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Drawing.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Web.ApplicationServices.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Web.dll" /r:C:\a\1\s\Releases\9.6\packages\Microsoft.AspNet.WebPages.3.2.2\lib\net45\System.Web.Helpers.dll /r:C:\a\1\s\Releases\9.6\packages\Microsoft.AspNet.Razor.3.2.2\lib\net45\System.Web.Razor.dll /r:C:\a\1\s\Releases\9.6\packages\Microsoft.AspNet.WebPages.3.2.2\lib\net45\System.Web.WebPages.Deployment.dll /r:C:\a\1\s\Releases\9.6\packages\Microsoft.AspNet.WebPages.3.2.2\lib\net45\System.Web.WebPages.dll /r:C:\a\1\s\Releases\9.6\packages\Microsoft.AspNet.WebPages.3.2.2\lib\net45\System.Web.WebPages.Razor.dll /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Windows.Forms.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Xml.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Xml.Linq.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Collections.Concurrent.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Collections.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ComponentModel.Annotations.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ComponentModel.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ComponentModel.EventBasedAsync.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Diagnostics.Contracts.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Diagnostics.Debug.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Diagnostics.Tools.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Diagnostics.Tracing.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Dynamic.Runtime.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Globalization.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.IO.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Linq.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Linq.Expressions.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Linq.Parallel.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Linq.Queryable.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Net.NetworkInformation.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Net.Primitives.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Net.Requests.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Net.WebHeaderCollection.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ObjectModel.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.Emit.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.Emit.ILGeneration.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.Emit.Lightweight.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.Extensions.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.Primitives.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Resources.ResourceManager.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Extensions.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Handles.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.InteropServices.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.InteropServices.WindowsRuntime.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Numerics.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Serialization.Json.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Serialization.Primitives.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Serialization.Xml.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Security.Principal.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ServiceModel.Duplex.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ServiceModel.Http.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ServiceModel.NetTcp.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ServiceModel.Primitives.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ServiceModel.Security.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Text.Encoding.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Text.Encoding.Extensions.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Text.RegularExpressions.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Threading.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Threading.Tasks.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Threading.Tasks.Parallel.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Threading.Timer.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Xml.ReaderWriter.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Xml.XDocument.dll" /r:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Xml.XmlSerializer.dll" /compile Base64Types.resx,obj\Release\PE.Services.Base64Types.resources 2017-05-09T10:50:11.1290971Z Processing resource file "Base64Types.resx" into "obj\Release\PE.Services.Base64Types.resources". 2017-05-09T10:50:11.1440899Z GenerateTargetFrameworkMonikerAttribute: 2017-05-09T10:50:11.1450900Z Skipping target "GenerateTargetFrameworkMonikerAttribute" because all output files are up-to-date with respect to the input files. 2017-05-09T10:50:11.1500903Z CoreCompile: 2017-05-09T10:50:11.1510897Z C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Roslyn\csc.exe /noconfig /nowarn:1701,1702 /nostdlib+ /errorreport:prompt /warn:4 /define:TRACE /highentropyva+ /reference:C:\a\1\s\Releases\9.6\packages\AutoMapper.4.1.1\lib\net45\AutoMapper.dll /reference:C:\a\1\s\Releases\9.6\packages\CsvHelper.2.13.5.0\lib\net40-client\CsvHelper.dll /reference:C:\a\1\s\Releases\9.6\packages\EntityFramework.6.1.2\lib\net45\EntityFramework.dll /reference:C:\a\1\s\Releases\9.6\packages\EntityFramework.6.1.2\lib\net45\EntityFramework.SqlServer.dll /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Microsoft.CSharp.dll" /reference:C:\a\1\s\Releases\9.6\packages\Unity.3.5.1404.0\lib\net45\Microsoft.Practices.Unity.Configuration.dll /reference:C:\a\1\s\Releases\9.6\packages\Unity.3.5.1404.0\lib\net45\Microsoft.Practices.Unity.dll /reference:C:\a\1\s\Releases\9.6\packages\Unity.3.5.1404.0\lib\net45\Microsoft.Practices.Unity.RegistrationByConvention.dll /reference:C:\a\1\s\Releases\9.6\packages\Microsoft.Web.Infrastructure.1.0.0.0\lib\net40\Microsoft.Web.Infrastructure.dll /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\mscorlib.dll" /reference:C:\a\1\s\Releases\9.6\packages\Newtonsoft.Json.7.0.1\lib\net45\Newtonsoft.Json.dll /reference:C:\a\1\s\Releases\9.6\PE.CacheProvider.Interfaces\bin\Release\PE.CacheProvider.Interfaces.dll /reference:C:\a\1\s\Releases\9.6\PE.Dal.Interfaces\bin\Release\PE.Dal.Interfaces.dll /reference:C:\a\1\s\Releases\9.6\PE.Data\bin\Release\PE.Data.dll /reference:C:\a\1\s\Releases\9.6\PE.DoubleMetaphone\bin\Release\PE.DoubleMetaphone.dll /reference:C:\a\1\s\Releases\9.6\PE.References\bin\Release\PE.References.dll /reference:C:\a\1\s\Releases\9.6\PE.ReportEngine.Interfaces\bin\Release\PE.ReportEngine.Interfaces.dll /reference:C:\a\1\s\Releases\9.6\PE.Services.Interfaces\bin\Release\PE.Services.Interfaces.dll /reference:C:\a\1\s\Releases\9.6\PE.Storage.Interfaces\bin\Release\PE.Storage.Interfaces.dll /reference:C:\a\1\s\Releases\9.6\PE.TaskRunner.Interfaces\bin\Release\PE.TaskRunner.Interfaces.dll /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.ComponentModel.DataAnnotations.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Configuration.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Core.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Data.DataSetExtensions.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Data.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Drawing.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Web.ApplicationServices.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Web.dll" /reference:C:\a\1\s\Releases\9.6\packages\Microsoft.AspNet.WebPages.3.2.2\lib\net45\System.Web.Helpers.dll /reference:C:\a\1\s\Releases\9.6\packages\Microsoft.AspNet.Razor.3.2.2\lib\net45\System.Web.Razor.dll /reference:C:\a\1\s\Releases\9.6\packages\Microsoft.AspNet.WebPages.3.2.2\lib\net45\System.Web.WebPages.Deployment.dll /reference:C:\a\1\s\Releases\9.6\packages\Microsoft.AspNet.WebPages.3.2.2\lib\net45\System.Web.WebPages.dll /reference:C:\a\1\s\Releases\9.6\packages\Microsoft.AspNet.WebPages.3.2.2\lib\net45\System.Web.WebPages.Razor.dll /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Windows.Forms.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Xml.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\System.Xml.Linq.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Collections.Concurrent.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Collections.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ComponentModel.Annotations.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ComponentModel.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ComponentModel.EventBasedAsync.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Diagnostics.Contracts.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Diagnostics.Debug.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Diagnostics.Tools.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Diagnostics.Tracing.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Dynamic.Runtime.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Globalization.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.IO.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Linq.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Linq.Expressions.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Linq.Parallel.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Linq.Queryable.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Net.NetworkInformation.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Net.Primitives.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Net.Requests.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Net.WebHeaderCollection.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ObjectModel.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.Emit.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.Emit.ILGeneration.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.Emit.Lightweight.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.Extensions.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Reflection.Primitives.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Resources.ResourceManager.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Extensions.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Handles.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.InteropServices.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.InteropServices.WindowsRuntime.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Numerics.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Serialization.Json.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Serialization.Primitives.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Runtime.Serialization.Xml.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Security.Principal.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ServiceModel.Duplex.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ServiceModel.Http.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ServiceModel.NetTcp.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ServiceModel.Primitives.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.ServiceModel.Security.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Text.Encoding.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Text.Encoding.Extensions.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Text.RegularExpressions.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Threading.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Threading.Tasks.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Threading.Tasks.Parallel.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Threading.Timer.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Xml.ReaderWriter.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Xml.XDocument.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6.1\Facades\System.Xml.XmlSerializer.dll" /debug:pdbonly /filealign:512 /optimize+ /out:obj\Release\PE.Services.dll /ruleset:"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Static Analysis Tools\\Rule Sets\MinimumRecommendedRules.ruleset" /subsystemversion:6.00 /resource:obj\Release\PE.Services.Base64Types.resources /target:library /utf8output Admin\AdminPasswordService.cs Admin\SystemOptionService.cs Clients\ClientOriginationService.cs Engine\BlobService.cs Expenses\AllocationService.cs Expenses\DisbursementsGridService.cs Expenses\ClaimsGridService.cs Expenses\DisbursementBatchService.cs Expenses\ExpenseClaimService.cs Expenses\ReceiptsService.cs Integration\IntegrationImportStatusService.cs Integration\IntegrationExportStatusService.cs MappingProfiles\AdminMappingProfile.cs MappingProfiles\ClientMappingProfile.cs MappingProfiles\ExpenseMappingProfile.cs MappingProfiles\ReportEngineMappingProfile.cs MappingProfiles\StaffMappingProfile.cs MappingProfiles\TimesheetMappingProfile.cs MaskingService.cs Email\EmailStatusService.cs PESystem\TaggingService.cs Properties\AssemblyInfo.cs AttributeServiceBase.cs GridServiceOfT.cs Admin\AlertService.cs Admin\AdminService.cs Assurance\AssuranceContactAttributeService.cs Assurance\AssuranceClientAttributeService.cs Assurance\AssuranceDetailService.cs Assurance\AssuranceQuestionnaireService.cs Assurance\AssuranceRequestService.cs Clients\ClientServiceService.cs Clients\ClientsService.cs Contacts\ContactsService.cs Dashlets\DashletService.cs Fees\FeeService.cs Jobs\JobDetailsService.cs Jobs\JobsService.cs Jobs\JobTaxService.cs Email\SendMailService.cs Engine\ClientService.cs Engine\ContactService.cs Engine\LookupService.cs Engine\StaffLoginService.cs Lockout\LockoutService.cs MappingProfiles\AssuranceMappingProfile.cs MappingProfiles\JobMappingProfile.cs Prospects\ProspectAttributeService.cs Prospects\ProspectsService.cs Questionnaires\QuestionnaireService.cs ReportEngine\ReportEngineService.cs ReportEngine\ReportService.cs Schedules\ScheduleService.cs Security\DataSecurityService.cs Security\PermissionsService.cs Security\PEWebSecurity.cs Security\RolesWrapper.cs Security\TicketService.cs Security\UserInfoService.cs Staff\StaffFavouritesService.cs Staff\StaffService.cs PESystem\GradeDefaultService.cs PESystem\LicenceService.cs PESystem\StoredProcedureService.cs PESystem\DashboardService.cs PESystem\MenuService.cs Timesheets\TimesheetApprovalService.cs Timesheets\TimesheetJobGridService.cs Timesheets\TimesheetEntryGridService.cs Timesheets\TimesheetHeaderAdminGridService.cs Timesheets\TimesheetAdminService.cs Timesheets\TimesheetExtenderService.cs Timesheets\TimesheetPostingGridService.cs Timesheets\TimesheetService.cs Timesheets\TimesheetWeeklyGridService.cs Utils\ErrorLogService.cs Utils\ErrorLogGridService.cs Utils\MimeTypeMappingService.cs Utils\ThumbnailCreatorService.cs "C:\a\_temp\.NETFramework,Version=v4.6.1.AssemblyAttributes.cs" 2017-05-09T10:50:11.1540893Z Using shared compilation with compiler from directory: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Roslyn 2017-05-09T10:50:11.6400930Z ##[error]Process 'msbuild.exe' exited with code '-1'.

Not sure if it's a task issue or an environment issue, but it's a major problem for us at the moment!

[ericsciple]

One thing you can try is isolating whether this is related to the custom logger or not. You can try running the command directly (without the logger attached) to see whether you still get a repro. The msbuild command-line that the task runs is printed in log.

[rposener]

Not sure I'm following @ericsciple - the build fails on VSTS Hosted VS2017 build. Runs fine when building locally. How can I change the command being run on the Hosted environment?

[nigurr]

This issue and https://github.com/Microsoft/vsts-tasks/issues/4256 are related. we are investigating possible causes

[ericsciple]

How are you running msbuild? Are you using the MSBuild task or VSBuild task?

[rposener]

Thanks @nigurr - if I can provide logs or other details let me know.

[rposener]

System.Debug revealed this:
2017-05-09T15:34:29.7142022Z CopyFilesToOutputDirectory:
2017-05-09T15:34:29.7232492Z Copying file from "obj\Release\PE.Unity.dll" to "bin\Release\PE.Unity.dll".
2017-05-09T15:34:29.7232492Z PE.Unity -> C:\a\1\s\Releases\9.6\PE.Unity\bin\Release\PE.Unity.dll
2017-05-09T15:34:29.7232492Z Copying file from "obj\Release\PE.Unity.pdb" to "bin\Release\PE.Unity.pdb".
2017-05-09T15:34:29.7252596Z ##[debug]Processed: ##vso[task.logdetail id=8583d3f7-8597-4326-80d7-00577f65439a;parentid=894025d7-d80d-4f60-8107-3e465b9f4505;type=Build;result=Succeeded;finishtime=2017-05-09T15:34:29.7252596Z;progress=100;state=Completed;parentid=894025d7-d80d-4f60-8107-3e465b9f4505;name=;]
2017-05-09T15:34:29.7252596Z Done Building Project "C:\a\1\s\Releases\9.6\PE.Unity\PE.Unity.csproj" (default targets).
2017-05-09T15:34:30.3062617Z PreComputeCompileTypeScriptWithTSConfig:
2017-05-09T15:34:30.3062617Z C:\Program Files (x86)\Microsoft SDKs\TypeScript\2.1\tsc.exe --project "C:\a\1\s\Releases\9.6\PE.Web.Main\tsconfig.json" --listEmittedFiles
2017-05-09T15:34:30.3444585Z CompileTypeScriptWithTSConfig:
2017-05-09T15:34:30.3454637Z C:\Program Files (x86)\Microsoft SDKs\TypeScript\2.1\tsc.exe --project "C:\a\1\s\Releases\9.6\PE.Web.Main\tsconfig.json" --listEmittedFiles
2017-05-09T15:34:39.8001608Z ##[debug]Exit code: -1
2017-05-09T15:34:39.8120167Z ##[debug]Error record:
2017-05-09T15:34:39.8999480Z ##[debug]Invoke-VstsTool : Process 'msbuild.exe' exited with code '-1'.
2017-05-09T15:34:39.9048893Z ##[debug]At C:\a_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-affca411ecda\1.113.0\ps_modules\MSBuildHelpers\InvokeFunctions.ps1:111 char:13
2017-05-09T15:34:39.9058779Z ##[debug]+ Invoke-VstsTool -FileName $MSBuildPath -Arguments $argume ...
2017-05-09T15:34:39.9078536Z ##[debug]+ ~~~~~~~~~~~~~
2017-05-09T15:34:39.9088417Z ##[debug] + CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
2017-05-09T15:34:39.9098296Z ##[debug] + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-VstsTool
2017-05-09T15:34:39.9118055Z ##[debug]
2017-05-09T15:34:39.9167455Z ##[debug]Script stack trace:
2017-05-09T15:34:39.9177338Z ##[debug]at Invoke-Tool, C:\a_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-affca411ecda\1.113.0\ps_modules\VstsTaskSdk\ToolFunctions.ps1: line 86
2017-05-09T15:34:39.9187216Z ##[debug]at Invoke-MSBuild, C:\a_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-affca411ecda\1.113.0\ps_modules\MSBuildHelpers\InvokeFunctions.ps1: line 111
2017-05-09T15:34:39.9208011Z ##[debug]at Invoke-BuildTools, C:\a_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-affca411ecda\1.113.0\ps_modules\MSBuildHelpers\InvokeFunctions.ps1: line 36
2017-05-09T15:34:39.9216965Z ##[debug]at , C:\a_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-affca411ecda\1.113.0\VSBuild.ps1: line 63
2017-05-09T15:34:39.9226735Z ##[debug]at , : line 1
2017-05-09T15:34:39.9236613Z ##[debug]at , : line 22
2017-05-09T15:34:39.9256463Z ##[debug]at , : line 18
2017-05-09T15:34:39.9513240Z ##[debug]at , : line 1
2017-05-09T15:34:39.9898572Z ##[debug]Exception:
2017-05-09T15:34:40.0698849Z ##[debug]Microsoft.PowerShell.Commands.WriteErrorException: Process 'msbuild.exe' exited with code '-1'.
2017-05-09T15:34:40.1756007Z ##[error]Process 'msbuild.exe' exited with code '-1'.
2017-05-09T15:34:40.1756007Z ##[debug]Processed: ##vso[task.logissue type=error]Process 'msbuild.exe' exited with code '-1'.
2017-05-09T15:34:40.1785655Z ##[debug]Leaving Invoke-VstsTool.
2017-05-09T15:34:40.1864694Z ##[debug]Processed: ##vso[task.complete result=Failed]
2017-05-09T15:34:40.1963488Z ##[debug]Processed: ##vso[task.logdetail finishtime=2017-05-09T15:34:40.1864694Z;id=90fc947a-4c32-4925-9b7f-92139835f718;progress=100;state=Completed;result=Failed]
2017-05-09T15:34:40.2131458Z ##[debug]Leaving Invoke-MSBuild.
2017-05-09T15:34:40.3020657Z ##[debug]Leaving Invoke-BuildTools.
2017-05-09T15:34:40.3030534Z ##[debug]Leaving C:\a_tasks\VSBuild_71a9a2d3-a98a-4caa-96ab-affca411ecda\1.113.0\VSBuild.ps1.
2017-05-09T15:34:41.2689348Z ##[section]Finishing: Build solution $/Practice Engine 9/Releases/9.6/PE.Web.Main 9.6.sln

[ericsciple]

@nigurr is it clear the issue you referenced is related? The issue you referenced is for vstest.console.exe. This issue is msbuild.exe.

[ericsciple]

@rposener look at the top of the log for the VSBuild task and you will see the msbuild.exe command line that the task runs. What you can try is adding a Command Line task instead of running the VSBuild task, and remove the custom logger from the command line. That will help to rule in/out the custom logger as the culprit.

[rposener]

It appears the issue is tsc.exe is an older version (2.1), which is run by the VSBuild task. I did not realize there was a nuget package Microsoft.TypeScript.MSBuild that would provide this functionality with upgrade. I believe this is our issue- have added that to the project now, and I am waiting for build to complete. If this does not solve it, will try your suggestion of removing custom logger. (see also: https://developercommunity.visualstudio.com/content/problem/52426/random-vsts-build-errors-process-msbuildexe-exited.html)

[nigurr]

@rposener Can you please share your account name?

[chrispat]

@nigurr his url is https://praceng.visualstudio.com

[rposener]

New error with typescript package - not sure if this helps:
C:\a\1\s\Releases\9.6\packages\Microsoft.TypeScript.MSBuild.2.3.1\tools\net45\Microsoft.TypeScript.targets(182,8): error MSB4064: The "IsFileSystemCaseSensitive" parameter is not supported by the "FindConfigFiles" task. Verify the parameter exists on the task, and it is a settable public instance property. [C:\a\1\s\Releases\9.6\PE.Web.Main\PE.Web.Main.csproj]
C:\a\1\s\Releases\9.6\packages\Microsoft.TypeScript.MSBuild.2.3.1\tools\net45\Microsoft.TypeScript.targets(179,5): error MSB4063: The "FindConfigFiles" task could not be initialized with its input parameters. [C:\a\1\s\Releases\9.6\PE.Web.Main\PE.Web.Main.csproj]

9 Warning(s)
2 Error(s)

Time Elapsed 00:01:04.09
Process 'msbuild.exe' exited with code '1'.

[rposener]

Another with same: https://github.com/Microsoft/TypeScript/issues/15536

[rposener]

Oddly, we just had a successful build from another dev's checkin. Keeps with the randomness.

[rposener]

Possibly related (Roslyn move) https://github.com/Microsoft/msbuild/issues/1542

[chrispat]

That is interesting. I wonder if that was related at all.

[rposener]

I have tried everything (adding Microsoft.TypeScript.MSBuild package) and (adding System.IO.FileSystem package) - neither seems to work. Could not figure out how/where to add the property to the .csproj noted as a workaround in the related msbuild issue.

[chrispat]

@rposener would you be willing to send me your proj files? [email protected]

[chrispat]

@rposener Can you remove all of the lines in your project files that refer to typescript props and targets that do not come from the NuGet package. In the project you sent me that would be line 4 and line 1786. Looking at those should give you a pattern for taking those out of other project files as needed. Also update the TypeScriptToolsVersion element to 2.3 from 2.1

[rposener]

@chrisrpatterson the version change alone did not change anything. Am trying to remove the other bits and will give it another go. Probably last thing I can try today - have to catch flight to build.

[rposener]

@chrisrpatterson no dice on that - different error, looks like it tried to compile ts this time, but failed (presumably due to lack of definitions provided by the targets somehow)?

  C:\a\16\s\main\PE.Web.Main\modules\Notifications.d.ts(13,5): error TS2687: Build:All declarations of 'body' must have identical modifiers. [C:\a\16\s\main\PE.Web.Main\PE.Web.Main.csproj]
  C:\a\16\s\main\PE.Web.Main\modules\Notifications.d.ts(20,5): error TS2687: Build:All declarations of 'dir' must have identical modifiers. [C:\a\16\s\main\PE.Web.Main\PE.Web.Main.csproj]
  C:\a\16\s\main\PE.Web.Main\modules\Notifications.d.ts(26,5): error TS2687: Build:All declarations of 'icon' must have identical modifiers. [C:\a\16\s\main\PE.Web.Main\PE.Web.Main.csproj]
  C:\a\16\s\main\PE.Web.Main\modules\Notifications.d.ts(32,5): error TS2687: Build:All declarations of 'title' must have identical modifiers. [C:\a\16\s\main\PE.Web.Main\PE.Web.Main.csproj]
  C:\a\16\s\main\PE.Web.Main\modules\Notifications.d.ts(38,5): error TS2687: Build:All declarations of 'lang' must have identical modifiers. [C:\a\16\s\main\PE.Web.Main\PE.Web.Main.csproj]
  C:\a\16\s\main\PE.Web.Main\modules\Notifications.d.ts(44,5): error TS2687: Build:All declarations of 'tag' must have identical modifiers. [C:\a\16\s\main\PE.Web.Main\PE.Web.Main.csproj]

[chrispat]

ok but that is a typescript compilation error and something that you should be able to fix.

[rposener]

Yeah, just kind of noticing that... Web Notifications - they must be included, but only partially....

[chrispat]

Not sure what web notifications are. Where you are you downloading that type definition from?

[rposener]

See this: https://github.com/Microsoft/TypeScript/issues/3111

[chrispat]

According to the dev on the TypeScript team you can remove the Notification.d.ts references now with 2.3 because it is included in lib.d.ts and that is added automatically.

[bryanmacfarlane]

This issue started off with an msbuild repro of msbuild returning -1, meandered into a vstest console issue, meandered into a typescript valid compilation failure (return code 1) and then something about web notifications.

I don't want this thread to turn into any build issues with VS2017. Should we close this and create a topic focusing on the msbuild -1 issue?

[bryanmacfarlane]

@rposener - the msbuild return of 1 in the middle of this thread is a real compilation failure (msbuild returns 1 on valid failure)

For the -1 case, you noted that you believed it may have been nuget. I've been wondering the same whether some nuget package got updated and it has a intermittent failure (timing). This is msbuild failing and it's useful to note that the image, the tasks and our service were not updated around the time of this (and our tasks simply run msbuild).

We will work with you to try and narrow. We can setup your project on the exact image we use on an azure VM to try and repro. The solution starts with a repro on a machine that doesn't get reclaimed immediately.

For the VS task, under advanced, there's an option for "Record Project Details". Can you try with that off?

[rposener]

Hi Guys - finally have a minute to look at this again - here is the latest failure. I think we are getting close, and I believe the msbuild.exe failure is a result of the change in the .targets references Chris had me remove. Here is the latest failure:

2017-05-10T17:55:25.5494824Z "d:\a\16\s\Releases\9.6\PE.Web.Main\PE.Web.Main.csproj" (default target) (2) ->
2017-05-10T17:55:25.5494824Z (FindConfigFiles target) -> 
2017-05-10T17:55:25.5494824Z   d:\a\16\s\Releases\9.6\packages\Microsoft.TypeScript.MSBuild.2.3.1\tools\net45\Microsoft.TypeScript.targets(182,8): error MSB4064: The "IsFileSystemCaseSensitive" parameter is not supported by the "FindConfigFiles" task. Verify the parameter exists on the task, and it is a settable public instance property. [d:\a\16\s\Releases\9.6\PE.Web.Main\PE.Web.Main.csproj]
2017-05-10T17:55:25.5494824Z   d:\a\16\s\Releases\9.6\packages\Microsoft.TypeScript.MSBuild.2.3.1\tools\net45\Microsoft.TypeScript.targets(179,5): error MSB4063: The "FindConfigFiles" task could not be initialized with its input parameters.  [d:\a\16\s\Releases\9.6\PE.Web.Main\PE.Web.Main.csproj]
2017-05-10T17:55:25.5504847Z 
2017-05-10T17:55:25.5504847Z     9 Warning(s)
2017-05-10T17:55:25.5504847Z     2 Error(s)
2017-05-10T17:55:25.5504847Z 
2017-05-10T17:55:25.5514880Z Time Elapsed 00:00:44.37
2017-05-10T17:55:25.7298992Z ##[error]Process 'msbuild.exe' exited with code '1'.

[chrispat]

@yuit can you help with this TypeScript issue?

[yuit]

@rposener could you share with me the build error and csproj you used for this build? Just to be clear this is same as original issue here even after all the modification suggested by @chrisrpatterson ?

[rposener]

So 1 of our builds are successfully building again. Just made same changes in our release branch and waiting results of it's build (looks good so far - should know soon). Issue was using TypeScript 2.3 (via nuget package) and the .csproj still had reference to TypeScript MSBuild targets and props files which were in the msbuild win32 extensions folder. As I understand things, when that is included it causes failure related to the Roslyn move which causes one of the tasks defined by TypeScript targets to error, thus the failure with msbuild.

[rposener]

Build succeeded - this can be closed - see last note about how to fix if others have the same issue. Thank you @chrisrpatterson !

[bryanmacfarlane]

Let us know if you hit it again. My gut says the changes lost the -1 repro and the msbuild 1 failure is a valid red herring for this issue. But we have a couple other repros we're tracking on.

[rposener]

I'm afraid you are right @bryanmacfarlane - just failed again on same source:

2017-05-11T03:41:02.8684796Z PreComputeCompileTypeScriptWithTSConfig:
2017-05-11T03:41:02.8754792Z   d:\a\16\s\Releases\9.6\packages\Microsoft.TypeScript.MSBuild.2.3.1\tools\net45\..\tsc\tsc.exe --project "d:\a\16\s\Releases\9.6\PE.Web.Main\tsconfig.json" --listEmittedFiles
2017-05-11T03:41:02.8944803Z CompileTypeScriptWithTSConfig:
2017-05-11T03:41:02.8944803Z   d:\a\16\s\Releases\9.6\packages\Microsoft.TypeScript.MSBuild.2.3.1\tools\net45\..\tsc\tsc.exe --project "d:\a\16\s\Releases\9.6\PE.Web.Main\tsconfig.json" --listEmittedFiles
2017-05-11T03:41:07.1585888Z ##[error]Process 'msbuild.exe' exited with code '-1'.
2017-05-11T03:41:10.6601092Z ##[section]Finishing: Build solution $/Practice Engine 9/Releases/9.6/PE.Web.Main 9.6.sln

Please let me know what next steps are.

[rposener]

Just was re-reading this and noticed @bryanmacfarlane comment about turning off "Record Project Details". We have 6 separate build tasks on the build definition - and it seems to randomly fail on one of those most of the time. I am turning that option off on all 6, and will try a few builds. If it looks good, I will leave/encourage our team to check in and trigger some builds to see if that makes any difference.

[rposener]

no dice on that change @bryanmacfarlane - still getting a msbuild.exe failure 1 at a random point.

[sgrebnov]

I see the same random Process 'msbuild.exe' exited with code '-1'.error when building UWP project (Visual Studio Build task). The error happens in different build phases (the exact same VSTS configuration) and occurs randomly.

##[section]Initializing Hosted Agent
Requesting an Agent from the Hosted Agent Pool
Hosted Agent successfully assigned.
Configuring connection settings for Hosted Agent
Starting Hosted Agent
Initializing connection to Hosted Agent
##[section]Starting: Build
Current agent version: '2.116.1'
##[section]Starting: Initialize Job

...
  C:\Program Files (x86)\Windows Kits\10\bin\x64\MakeAppx.exe pack /l /h sha256 /m d:\a\1\s\subfolder\ColoringBook\bin\ARM\Release\x86\ilc\ForBundle\AppxManifest.xml /f obj\x86\Release\main.map.txt /o /p d:\a\1\s\subfolder\ColoringBook\bin\ARM\Release\x86\ColoringBook\ColoringBook_1.0.0.0_x86.appx  
  ColoringBook -> d:\a\1\s\subfolder\ColoringBook\bin\ARM\Release\x86\ColoringBook\ColoringBook_1.0.0.0_x86.appx
_CreateMainUploadPackage:
  C:\Program Files (x86)\Windows Kits\10\bin\x64\MakeAppx.exe pack /l /h sha256 /m d:\a\1\s\subfolder\ColoringBook\bin\ARM\Release\x86\ilc\ForUploadBundle\AppxManifest.xml /f obj\x86\Release\Upload\main.map.txt /o /p d:\a\1\s\subfolder\ColoringBook\bin\ARM\Release\x86\ColoringBook\Upload\ColoringBook_1.0.0.0_x86.appx  
##[error]Process 'msbuild.exe' exited with code '-1'.
##[section]Finishing: Build UWP sln
##[section]Starting: Post Job Cleanup

Full build trace: https://gist.github.com/sgrebnov/036ac36b3fa15d086900d7adad93dc07

Here is log file from VSTS with enabled debug mode (different run):
https://microsoft-my.sharepoint.com/personal/v-segr_microsoft_com/_layouts/15/guestaccess.aspx?guestaccesstoken=kqIvNhr9lMIThhL7t98jGGwP%2fSTRfeWOmoFxGaOL1ZM%3d&docid=2_0f0a74936096041a6ae5dfb9683aaec71&rev=1

It started to happen in agent version 2.115.0, but I also saw this in 2.116.1 as well.

Are there any recommendations or instructions how to debug/profile such issue?

[bryanmacfarlane]

@sgrebnov - thanks for the tight repro.

@rposener - note that msbuild 1 is a real build failure (regardless of how flaky). msbuild completed execution and there was a build failure.

The issue we're tracking here is returning -1 which indicates the msbuild process terminated prematurely and didn't finish. I want to keep the conversation focused to that.

If anyone else has this issue, please post details or email me - bryanmac and that's at microsoft.

[rposener]

@bryanmacfarlane - sounds good. We are definitely still seeing the -1 reported. This was our result today:

2017-05-12T10:48:24.5194114Z   Adding file (d:\a\16\s\main\PETaskJob\obj\Release\Package\PackageTmp\bin\ru\Microsoft.Data.Edm.resources.dll).
2017-05-12T10:48:24.5244155Z   Adding file (d:\a\16\s\main\PETaskJob\obj\Release\Package\PackageTmp\bin\ru\Microsoft.Data.OData.resources.dll).
2017-05-12T10:48:24.5484383Z   Adding file (d:\a\16\s\main\PETaskJob\obj\Release\Package\PackageTmp\bin\ru\Microsoft.Data.Services.Client.resources.dll).
2017-05-12T10:48:24.7596233Z ##[error]Process 'msbuild.exe' exited with code '-1'.
2017-05-12T10:48:24.7946529Z ##[section]Finishing: Build solution $/Practice Engine 9/main/PE.Web.Main.sln

Is it sufficient to add a build at beginning that continues on error ?

[rposener]

As a work-around that does seem to be working (just added a build of a small solution in the first position which continues on error) the build does complete. So until it can be resolved properly, that does at least appear to work-around the issue.

[rposener]

It is interesting that none of the builds fail now (not even the first one that continues on error). Looking back through our builds that failed with -1, the builds all failed just after 60 seconds (between 60 - 65 seconds for that task run). When successful the task that we used to have first runs for about 80 seconds. It's like the first run gets it's process killed if running more than 60 seconds. The new 'extra' build I added runs for only 13 seconds.

[bryanmacfarlane]

We're rolling an image right now that fixes/mitigates the issue. Not sure if it's in your scale unit yet but that may be why. We believe the issue should go away sometime today.

[chrispat]

The updated image should be deployed to all of the pools. Please let us know here if you are still seeing issues.

[sgrebnov]

Confirm that I don't see msbuild error anymore - I've re-run build 10 times and all completed successfully.

The only thing I see is that agent version is still '2.116.1', I expected it to be a new one, please confirm that 2.116.1 is expected.

[bryanmacfarlane]

@sgrebnov - thanks for confirming

The issue wasnt in the agent ( application that runs your build ). There's multiple parts - there's the OS image, the tools installed on the OS image (VS etc) and our build agent we pull onto that image which hooks up to your account and runs the build.

[sgrebnov]

@bryanmacfarlane - makes sense, thanks for details

[rposener]

so far our builds all look good over the weekend - albeit not as many as we would see during the week. Will close this tomorrow if we don't find any other issues. Thanks for the help @chrisrpatterson and @bryanmacfarlane.

[rposener]

Looks solid to me - closing the issue.

[bryanmacfarlane]

we're copying a new image with gflags to figure out what's wacking the process (it's not crashing, it's not exiting). the previous mitigation appears to reduce the likelyhood but still hapenning ... we're on it.

[rposener]

Just an update - again today - the same 2 builds failed on the first go for the day. One of them failed also on the second build task during the second run of the definition. All 3 failures report '-1' status at the end apparently mid-step of the build. All builds after those 3 have been successful today (about 6 of them across the 2 definitions). If there is more I can do to log/help let me know.

[bryanmacfarlane]

We have an image about to rollout which should help but more importantly we've added some gflags debug tooling to the image so if it happens to you again, you can add something to the def to output what is killing your msbuild process. We had repros but in our test instance, it doesn't repro so we might need your help tomorrow after it rolls out. We will follow up on what to add to your definition

[sgrebnov]

Encountered with msbuild issue again, please let me know whether new image is here and how I can activate new debug/tracing functionality

##[error]Process 'msbuild.exe' exited with code '-1'.

[ericsciple]

A new image is closed to being rolled out with additional logging enabled to determine more information. When msbuild.exe/vstest.console.exe exits, info will be written to the application event log with the exit code and if another process killed it, the name of the process that killed msbuild.exe/vstest.console.exe.

To be ready when the new image finishes rolling out, you can go ahead and add the below inline powershell to grab the info from the application event log once it starts getting published. You will want to modify the control options on the step, so it runs even on failure/cancel.

Get-EventLog -LogName Application -After ([datetime]::now.addhours(-1)) -Source 'Process Exit Monitor' | % { "EventID: $($_.EventID)" ; "TimeGenerated: $($_.TimeGenerated)" ; "Message: $($_.Message)" ; "" }

[rposener]

Just added this, still seeing errors on first run each day. Will report with details as soon as we capture one of the failures (probably tomorrow).

[chrispat]

@rposener any luck getting a repro with logs?

[rposener]

@chrisrpatterson no failures today. Yesterday we had an odd outage - nuget was failing with a socket error across all builds on the first nuget task for a good hour or more. Any chance these issues are related?

[rposener]

Here are the nuget error details we were seeing, just incase this helps anyone.

2017-05-22T15:32:14.4126804Z ##[section]Starting: NuGet restore $/Practice Engine 9/main/PE.Web.Main.sln
2017-05-22T15:32:14.4136802Z ==============================================================================
2017-05-22T15:32:14.4136802Z Task         : NuGet Restore
2017-05-22T15:32:14.4136802Z Description  : Restores NuGet packages in preparation for a Visual Studio Build step.
2017-05-22T15:32:14.4136802Z Version      : 1.0.0
2017-05-22T15:32:14.4136802Z Author       : Microsoft Corporation
2017-05-22T15:32:14.4136802Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613747)
2017-05-22T15:32:14.4136802Z ==============================================================================
2017-05-22T15:32:15.5476442Z [command]C:\Windows\system32\chcp.com 65001
2017-05-22T15:32:15.6306423Z Active code page: 65001
2017-05-22T15:32:16.0596255Z Downloading: https://dist.nuget.org/win-x86-commandline/v4.0.0/nuget.exe
2017-05-22T15:33:50.2041302Z ##[error]socket hang up
2017-05-22T15:33:50.2121299Z ##[section]Finishing: NuGet restore $/Practice Engine 9/main/PE.Web.Main.sln

[rposener]

Just reviewing the logs - our last '-1' error was last Friday (May 19). We didn't have any builds triggered over the weekend, and yesterday we were getting the nuget error on the first builds of the day, when we have been getting the '-1'.

[bryanmacfarlane]

We finally got a repro which captured the offending process from the event log line above. We know exactly what is killing msbuild/vstest now. We're following up right now with an image update. It takes awhile to rollout but good news.

This one was especially tough because the size, length and volatility of our tool would protect it from repro'ing. Sorry for the inconvenience.

[chrispat]

@rposener can you create a seperate issue for the NuGet question.

[rposener]

No worries things look good once again on my end. Thanks for the work. Will close this niw.