Sdk: `dotnet build` fails with "Cannot find project info" when P2P build fails, hiding the original error

[AArnott]

This build break only occurs on dotnet CLI builds that occur on Ubuntu 14.04. It is quite minimal, requiring only two projects with a reference between them, and the referenced project including a PackageReference to CodeGeneration.Roslyn.BuildTime, v0.3.11-g806e2e697e.
In these circumstances the build fails with:

/usr/share/dotnet/sdk/1.0.1/Sdks/Microsoft.NET.Sdk/build/Microsoft.NET.Sdk.targets(92,5): error : Cannot find project info for '/home/travis/build/AArnott/TravisCINetSDK/lib/lib.csproj'. This can indicate a missing project reference. [/home/travis/build/AArnott/TravisCINetSDK/test/test.csproj]

Repro project
Travis CI build log

This works fine on Ubuntu 16 and on Ubuntu 14 that is included with Windows 10.

[AArnott]

Package restore results appear to be identical in repro Ubuntu 14 and non-repro Windows cases. So it appears to be a bug in dotnet build itself.

[livarcocc]

@eerhardt I remember you helping someone with a similar issue not long ago. Can you quickly see if that is the same problem now as it was then.

[AArnott]

It looks like the "Cannot find project info" error is but a red herring. The original error (only visible in more verbose logged builds) is "Version for package dotnet-codegen could not be resolved." even though package restore had occurred.

[eerhardt]

The first thing I tried was using a recent .NET Core SDK v2.0, and I get the following error during restore:

/home/eerhardt/git/cli/.dotnet_stage0/x64/sdk/2.0.0-preview1-005720/NuGet.targets(97,5): error : Package System.CommandLine 1.0.0.1 is not compatible with netcoreapp1.0 (.NETCoreApp,Version=v1.0). Package System.CommandLine 1.0.0.1 supports: net35 (.NETFramework,Version=v3.5) [/home/eerhardt/git/traviscinetsdk/test/test.csproj]
/home/eerhardt/git/cli/.dotnet_stage0/x64/sdk/2.0.0-preview1-005720/NuGet.targets(97,5): error : Package TypeUtils 1.0.0.2 is not compatible with netcoreapp1.0 (.NETCoreApp,Version=v1.0). Package TypeUtils 1.0.0.2 supports: net35 (.NETFramework,Version=v3.5) [/home/eerhardt/git/traviscinetsdk/test/test.csproj]
/home/eerhardt/git/cli/.dotnet_stage0/x64/sdk/2.0.0-preview1-005720/NuGet.targets(97,5): error : One or more packages are incompatible with .NETCoreApp,Version=v1.0. [/home/eerhardt/git/traviscinetsdk/test/test.csproj]

However, using v1.0.1 of the .NET Core SDK, the restore succeeds.

I'm not sure where these packages are coming from. @AArnott - can you tell me? Are they hosted on https://ci.appveyor.com/nuget/codegeneration-roslyn?

[eerhardt]

In my build /v:diag trace, I see

                     /usr/bin/dotnet codegen -r /home/eerhardt/.nuget/packages/microsoft.win32.primitives/4.3.0/ref/netstandard1.3/Microsoft.Win32.Primitives.dll -r /home/eerhardt/.nuget/packages/system.appcontext/4.3.0/ref/netstandard1.3/System.AppContext.dll -r /home/eerhardt/.nuget/packages/system.collections.concurrent/4.3.0/ref/netstandard1.3/System.Collections.Concurrent.dll -r /home/eerhardt/.nuget/packages/system.collections/4.3.0/ref/netstandard1.3/System.Collections.dll -r /home/eerhardt/.nuget/packages/system.console/4.3.0/ref/netstandard1.3/System.Console.dll -r /home/eerhardt/.nuget/packages/system.diagnostics.debug/4.3.0/ref/netstandard1.3/System.Diagnostics.Debug.dll -r /home/eerhardt/.nuget/packages/system.diagnostics.tools/4.3.0/ref/netstandard1.0/System.Diagnostics.Tools.dll -r /home/eerhardt/.nuget/packages/system.diagnostics.tracing/4.3.0/ref/netstandard1.3/System.Diagnostics.Tracing.dll -r /home/eerhardt/.nuget/packages/system.globalization.calendars/4.3.0/ref/netstandard1.3/System.Globalization.Calendars.dll -r /home/eerhardt/.nuget/packages/system.globalization/4.3.0/ref/netstandard1.3/System.Globalization.dll -r /home/eerhardt/.nuget/packages/system.io.compression/4.3.0/ref/netstandard1.3/System.IO.Compression.dll -r /home/eerhardt/.nuget/packages/system.io.compression.zipfile/4.3.0/ref/netstandard1.3/System.IO.Compression.ZipFile.dll -r /home/eerhardt/.nuget/packages/system.io/4.3.0/ref/netstandard1.3/System.IO.dll -r /home/eerhardt/.nuget/packages/system.io.filesystem/4.3.0/ref/netstandard1.3/System.IO.FileSystem.dll -r /home/eerhardt/.nuget/packages/system.io.filesystem.primitives/4.3.0/ref/netstandard1.3/System.IO.FileSystem.Primitives.dll -r /home/eerhardt/.nuget/packages/system.linq/4.3.0/ref/netstandard1.0/System.Linq.dll -r /home/eerhardt/.nuget/packages/system.linq.expressions/4.3.0/ref/netstandard1.3/System.Linq.Expressions.dll -r /home/eerhardt/.nuget/packages/system.net.http/4.3.0/ref/netstandard1.3/System.Net.Http.dll -r /home/eerhardt/.nuget/packages/system.net.primitives/4.3.0/ref/netstandard1.3/System.Net.Primitives.dll -r /home/eerhardt/.nuget/packages/system.net.sockets/4.3.0/ref/netstandard1.3/System.Net.Sockets.dll -r /home/eerhardt/.nuget/packages/system.objectmodel/4.3.0/ref/netstandard1.3/System.ObjectModel.dll -r /home/eerhardt/.nuget/packages/system.reflection/4.3.0/ref/netstandard1.3/System.Reflection.dll -r /home/eerhardt/.nuget/packages/system.reflection.extensions/4.3.0/ref/netstandard1.0/System.Reflection.Extensions.dll -r /home/eerhardt/.nuget/packages/system.reflection.primitives/4.3.0/ref/netstandard1.0/System.Reflection.Primitives.dll -r /home/eerhardt/.nuget/packages/system.resources.resourcemanager/4.3.0/ref/netstandard1.0/System.Resources.ResourceManager.dll -r /home/eerhardt/.nuget/packages/system.runtime/4.3.0/ref/netstandard1.3/System.Runtime.dll -r /home/eerhardt/.nuget/packages/system.runtime.extensions/4.3.0/ref/netstandard1.3/System.Runtime.Extensions.dll -r /home/eerhardt/.nuget/packages/system.runtime.handles/4.3.0/ref/netstandard1.3/System.Runtime.Handles.dll -r /home/eerhardt/.nuget/packages/system.runtime.interopservices/4.3.0/ref/netstandard1.3/System.Runtime.InteropServices.dll -r /home/eerhardt/.nuget/packages/system.runtime.interopservices.runtimeinformation/4.3.0/ref/netstandard1.1/System.Runtime.InteropServices.RuntimeInformation.dll -r /home/eerhardt/.nuget/packages/system.runtime.numerics/4.3.0/ref/netstandard1.1/System.Runtime.Numerics.dll -r /home/eerhardt/.nuget/packages/system.security.cryptography.algorithms/4.3.0/ref/netstandard1.3/System.Security.Cryptography.Algorithms.dll -r /home/eerhardt/.nuget/packages/system.security.cryptography.encoding/4.3.0/ref/netstandard1.3/System.Security.Cryptography.Encoding.dll -r /home/eerhardt/.nuget/packages/system.security.cryptography.primitives/4.3.0/ref/netstandard1.3/System.Security.Cryptography.Primitives.dll -r /home/eerhardt/.nuget/packages/system.security.cryptography.x509certificates/4.3.0/ref/netstandard1.3/System.Security.Cryptography.X509Certificates.dll -r /home/eerhardt/.nuget/packages/system.text.encoding/4.3.0/ref/netstandard1.3/System.Text.Encoding.dll -r /home/eerhardt/.nuget/packages/system.text.encoding.extensions/4.3.0/ref/netstandard1.3/System.Text.Encoding.Extensions.dll -r /home/eerhardt/.nuget/packages/system.text.regularexpressions/4.3.0/ref/netstandard1.3/System.Text.RegularExpressions.dll -r /home/eerhardt/.nuget/packages/system.threading/4.3.0/ref/netstandard1.3/System.Threading.dll -r /home/eerhardt/.nuget/packages/system.threading.tasks/4.3.0/ref/netstandard1.3/System.Threading.Tasks.dll -r /home/eerhardt/.nuget/packages/system.threading.timer/4.3.0/ref/netstandard1.2/System.Threading.Timer.dll -r /home/eerhardt/.nuget/packages/system.xml.readerwriter/4.3.0/ref/netstandard1.3/System.Xml.ReaderWriter.dll -r /home/eerhardt/.nuget/packages/system.xml.xdocument/4.3.0/ref/netstandard1.3/System.Xml.XDocument.dll --out obj/Debug/netstandard1.3/ --generatedFilesList obj/Debug/netstandard1.3/hbvk2h2g.udp -- Class1.cs  (TaskId:26)
                      (TaskId:26)
                     Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'System.CommandLine, Version=0.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified. (TaskId:26)
                      (TaskId:26)

Note I'm getting the same build error you are.

So I'm guessing there is something wrong with the dotnet-codegen tool.

[eerhardt]

I'm not sure why the build doesn't fail after this error, I see the error:

15:14:15.811   2:2>/home/eerhardt/.nuget/packages/codegeneration.roslyn.buildtime/0.3.11-g806e2e697e/build/CodeGeneration.Roslyn.BuildTime.targets(20,5): error MSB6006: "dotnet" exited with code 134. [/home/eerhardt/git/traviscinetsdk/lib/lib.csproj]

I would have thought that should:
a) be written out
b) fail the build

[AArnott]

The workaround seems to be to explicitly dotnet restore lib before dotnet restore test. Evidently, dotnet restore does not restore DotNetCliToolReference items for P2P projects on Ubuntu 14.04 (but does everywhere else).

[eerhardt]

@emgarten @rohit21agrawal - any thoughts?

[eerhardt]

When I dotnet restore lib, I get the above restore errors:

/usr/share/dotnet/sdk/1.0.1/NuGet.targets(97,5): error : Package System.CommandLine 1.0.0.1 is not compatible with netcoreapp1.0 (.NETCoreApp,Version=v1.0). Package System.CommandLine 1.0.0.1 supports: net35 (.NETFramework,Version=v3.5) [/home/eerhardt/git/traviscinetsdk/lib/lib.csproj]
/usr/share/dotnet/sdk/1.0.1/NuGet.targets(97,5): error : Package TypeUtils 1.0.0.2 is not compatible with netcoreapp1.0 (.NETCoreApp,Version=v1.0). Package TypeUtils 1.0.0.2 supports: net35 (.NETFramework,Version=v3.5) [/home/eerhardt/git/traviscinetsdk/lib/lib.csproj]
/usr/share/dotnet/sdk/1.0.1/NuGet.targets(97,5): error : One or more packages are incompatible with .NETCoreApp,Version=v1.0. [/home/eerhardt/git/traviscinetsdk/lib/lib.csproj]

[AArnott]

@eerhardt What commit are you running that on? Try it on the workaround2 branch, where I added an entry to nuget.config.

[eerhardt]

I was on aceb047ca657269b9004c323b1d4a646da7e4895

[AArnott]

I just pushed a commit to master so that nuget.config has the missing entry as well.

[eerhardt]

Even on the tip of master (40cdb6bc05479c64a7480c45176cc72baeefe39e), I'm getting an error when I:

$ dotnet restore lib
$ dotnet build lib

/home/eerhardt/.nuget/packages/codegeneration.roslyn.buildtime/0.3.11-g806e2e697e/build/CodeGeneration.Roslyn.BuildTime.targets(20,5): error MSB6006: "dotnet" exited with code 134. [/home/eerhardt/git/traviscinetsdk/lib/lib.csproj]

It seems odd that this error gets eaten when building a P2P reference (test -> lib). When I dotnet build test, this error doesn't show and the test project keeps trying to build and fails later since it can't find the lib project info.

[AArnott]

Yes, I had to use /v:detailed on the build of the test project before I finally saw the original dotnet exit code and error. Did you see an error before your exit code line that would tell you what exit code 134 means?

As the latest Travis build from master shows, the build of the test project still fails if all one does is dotnet restore test then dotnet build test. But if I just add a dotnet restore lib before dotnet restore test, the build succeeds.

I'm beginning to wonder if this isn't an Ubuntu 14-only error after all though. I wonder if dotnet restore simply never restores packages for DotNetCliToolReference items across P2P references and we only see it repro on Ubuntu 14 because other test machines either have the package restored already or were restoring the 'sln' file, which would restore each project individually.

[eerhardt]

Did you see an error before your exit code line that would tell you what exit code 134 means?

Yes, I see

Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'System.CommandLine, Version=0.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified. (TaskId:13)

In general, when you see exit code 134 on Unix it means "unhandled exception".

I think the issue is that system.commandline/1.0.0.1 doesn't support netcoreapp1.0. It only supports net35. The dotnet-codegen tool has a reference to it:

 "dotnet-codegen/0.3.11-g806e2e697e": {
        "dependencies": {
          "CodeGeneration.Roslyn": "0.3.11-g806e2e697e",
          "System.CommandLine": "1.0.0.1",
          "System.Runtime.Loader": "4.3.0"
        },
        "runtime": {
          "lib/netcoreapp1.0/dotnet-codegen.dll": {}
        }
      },
      "system.commandline/1.0.0.1": {
        "dependencies": {
          "TypeUtils": "1.0.0.2"
        }
      },

Notice that system.commandline doesn't have any "runtime" assets (i.e. a .dll to load). Thus when running the dotnet codegen command, a FileNotFoundException is being raised for System.CommandLine, Version=0.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60.

When I call

$ ls -al /home/eerhardt/.nuget/packages/system.commandline/1.0.0.1/lib
total 12
drwxrwxr-x 3 eerhardt eerhardt 4096 Apr 15 14:59 .
drwxrwxr-x 3 eerhardt eerhardt 4096 Apr 15 14:59 ..
drwxrwxr-x 2 eerhardt eerhardt 4096 Apr 15 14:59 net35

I only see a lib for net35.

@AArnott - were you guys using project.json before and now converting to MSBuild/.csproj? If so, there is a take-back in the CliToolReference that you can't use imports anymore. Cli tools (like dotnet-codegen) need to have all their dependencies support netcoreapp1.0.

[AArnott]

@eerhardt You're looking at the wrong version of the System.Commandline package. 1.0.0.1 shows up in the error message if you haven't got the added entry to nuget.config because nuget resolves the 0.1.0-* version that it can't find to 1.0.0.1 which it can find, but it's a totally different package.
Once you have the nuget.config file entry added as I've pushed to my master branch recently, you'll see that package restore gets the right version. And you don't get netcoreapp1.0 restore errors any more.

[AArnott]

The missing System.Commandline package error from the detailed build log that you're seeing is exactly the evidence I found that led me to realize that dotnet restore doesn't restore DotNetCliToolReference items across P2P references. I don't know yet whether that limitation is unique to Ubuntu 14.04.

[AArnott]

It's not an Ubuntu 14.04 only limitation. Evidently it even happens on Windows -- I guess we just never saw it because everyone involved in this investigation on the libgit2sharp side had previously done a dotnet restore on the sln or the lib project such that the package cache had all the required dependencies. And appveyor was restoring the sln each time. It was only the travis CI script that was restoring just the test project itself.

So this distills to two (alleged) issues:

1. dotnet build hides build errors in P2P refs and only reports "missing project info" unless you use a detailed build log. This makes investigations very slow and was misleading to me for a long time.
2. As some MSBuild extensions include CodeGeneration.Roslyn and SourceLink are built around invoking the DotNetCliToolReference command from MSBuild, dotnet restore some.proj is insufficient because it won't restore the CLI tools along with the other package dependencies.

[AArnott]

I presume issue no. 2 above should be moved to the nuget/home repo, yes?

[eerhardt]

You're right. The CLI was caching the deps.json file for the codegen tool when I restored it previously with the wrong version. Cleaning out my ~/.nuget/packages/.tools directory and restoring again allowed me to build lib.csproj.

Now, when I clean, restore, and build /v:diag test I see the error you have above:

/usr/bin/dotnet codegen -r /home/eerhardt/.nuget/packages/microsoft.win32.primitives/4.3.0/ref/netstandard1.3/Microsoft.Win32.Primitives.dll -r /home/eerhardt/.nuget/packages/system.appcontext/4.3.0/ref/netstandard1.3/System.AppContext.dll -r /home/eerhardt/.nuget/packages/system.collections.concurrent/4.3.0/ref/netstandard1.3/System.Collections.Concurrent.dll -r /home/eerhardt/.nuget/packages/system.collections/4.3.0/ref/netstandard1.3/System.Collections.dll -r /home/eerhardt/.nuget/packages/system.console/4.3.0/ref/netstandard1.3/System.Console.dll -r /home/eerhardt/.nuget/packages/system.diagnostics.debug/4.3.0/ref/netstandard1.3/System.Diagnostics.Debug.dll -r /home/eerhardt/.nuget/packages/system.diagnostics.tools/4.3.0/ref/netstandard1.0/System.Diagnostics.Tools.dll -r /home/eerhardt/.nuget/packages/system.diagnostics.tracing/4.3.0/ref/netstandard1.3/System.Diagnostics.Tracing.dll -r /home/eerhardt/.nuget/packages/system.globalization.calendars/4.3.0/ref/netstandard1.3/System.Globalization.Calendars.dll -r /home/eerhardt/.nuget/packages/system.globalization/4.3.0/ref/netstandard1.3/System.Globalization.dll -r /home/eerhardt/.nuget/packages/system.io.compression/4.3.0/ref/netstandard1.3/System.IO.Compression.dll -r /home/eerhardt/.nuget/packages/system.io.compression.zipfile/4.3.0/ref/netstandard1.3/System.IO.Compression.ZipFile.dll -r /home/eerhardt/.nuget/packages/system.io/4.3.0/ref/netstandard1.3/System.IO.dll -r /home/eerhardt/.nuget/packages/system.io.filesystem/4.3.0/ref/netstandard1.3/System.IO.FileSystem.dll -r /home/eerhardt/.nuget/packages/system.io.filesystem.primitives/4.3.0/ref/netstandard1.3/System.IO.FileSystem.Primitives.dll -r /home/eerhardt/.nuget/packages/system.linq/4.3.0/ref/netstandard1.0/System.Linq.dll -r /home/eerhardt/.nuget/packages/system.linq.expressions/4.3.0/ref/netstandard1.3/System.Linq.Expressions.dll -r /home/eerhardt/.nuget/packages/system.net.http/4.3.0/ref/netstandard1.3/System.Net.Http.dll -r /home/eerhardt/.nuget/packages/system.net.primitives/4.3.0/ref/netstandard1.3/System.Net.Primitives.dll -r /home/eerhardt/.nuget/packages/system.net.sockets/4.3.0/ref/netstandard1.3/System.Net.Sockets.dll -r /home/eerhardt/.nuget/packages/system.objectmodel/4.3.0/ref/netstandard1.3/System.ObjectModel.dll -r /home/eerhardt/.nuget/packages/system.reflection/4.3.0/ref/netstandard1.3/System.Reflection.dll -r /home/eerhardt/.nuget/packages/system.reflection.extensions/4.3.0/ref/netstandard1.0/System.Reflection.Extensions.dll -r /home/eerhardt/.nuget/packages/system.reflection.primitives/4.3.0/ref/netstandard1.0/System.Reflection.Primitives.dll -r /home/eerhardt/.nuget/packages/system.resources.resourcemanager/4.3.0/ref/netstandard1.0/System.Resources.ResourceManager.dll -r /home/eerhardt/.nuget/packages/system.runtime/4.3.0/ref/netstandard1.3/System.Runtime.dll -r /home/eerhardt/.nuget/packages/system.runtime.extensions/4.3.0/ref/netstandard1.3/System.Runtime.Extensions.dll -r /home/eerhardt/.nuget/packages/system.runtime.handles/4.3.0/ref/netstandard1.3/System.Runtime.Handles.dll -r /home/eerhardt/.nuget/packages/system.runtime.interopservices/4.3.0/ref/netstandard1.3/System.Runtime.InteropServices.dll -r /home/eerhardt/.nuget/packages/system.runtime.interopservices.runtimeinformation/4.3.0/ref/netstandard1.1/System.Runtime.InteropServices.RuntimeInformation.dll -r /home/eerhardt/.nuget/packages/system.runtime.numerics/4.3.0/ref/netstandard1.1/System.Runtime.Numerics.dll -r /home/eerhardt/.nuget/packages/system.security.cryptography.algorithms/4.3.0/ref/netstandard1.3/System.Security.Cryptography.Algorithms.dll -r /home/eerhardt/.nuget/packages/system.security.cryptography.encoding/4.3.0/ref/netstandard1.3/System.Security.Cryptography.Encoding.dll -r /home/eerhardt/.nuget/packages/system.security.cryptography.primitives/4.3.0/ref/netstandard1.3/System.Security.Cryptography.Primitives.dll -r /home/eerhardt/.nuget/packages/system.security.cryptography.x509certificates/4.3.0/ref/netstandard1.3/System.Security.Cryptography.X509Certificates.dll -r /home/eerhardt/.nuget/packages/system.text.encoding/4.3.0/ref/netstandard1.3/System.Text.Encoding.dll -r /home/eerhardt/.nuget/packages/system.text.encoding.extensions/4.3.0/ref/netstandard1.3/System.Text.Encoding.Extensions.dll -r /home/eerhardt/.nuget/packages/system.text.regularexpressions/4.3.0/ref/netstandard1.3/System.Text.RegularExpressions.dll -r /home/eerhardt/.nuget/packages/system.threading/4.3.0/ref/netstandard1.3/System.Threading.dll -r /home/eerhardt/.nuget/packages/system.threading.tasks/4.3.0/ref/netstandard1.3/System.Threading.Tasks.dll -r /home/eerhardt/.nuget/packages/system.threading.timer/4.3.0/ref/netstandard1.2/System.Threading.Timer.dll -r /home/eerhardt/.nuget/packages/system.xml.readerwriter/4.3.0/ref/netstandard1.3/System.Xml.ReaderWriter.dll -r /home/eerhardt/.nuget/packages/system.xml.xdocument/4.3.0/ref/netstandard1.3/System.Xml.XDocument.dll --out obj/Debug/netstandard1.3/ --generatedFilesList obj/Debug/netstandard1.3/ls1jf2s2.3qb -- Class1.cs  (TaskId:26)
Version for package `dotnet-codegen` could not be resolved. (TaskId:26)
15:56:50.551   2:2>/home/eerhardt/.nuget/packages/codegeneration.roslyn.buildtime/0.3.11-g806e2e697e/build/CodeGeneration.Roslyn.BuildTime.targets(20,5): error MSB6006: "dotnet" exited with code 1. [/home/eerhardt/git/traviscinetsdk/lib/lib.csproj]
                     Output Item(s): GeneratedCodeFromAttribute_Compile= (TaskId:26)
                     Output Item(s): FileWrites= (TaskId:26)
                     Output Item(s): FileWrites= (TaskId:26)
                   Done executing task "GenerateCodeFromAttributes" -- FAILED. (TaskId:26)
15:56:50.551   2:2>Done building target "GenerateCodeFromAttributes" in project "lib.csproj" -- FAILED.: (TargetId:60)

I think there are 2 bugs at play here:

1. (as you mentioned) CliToolReference must not be getting restored across P2Ps
2. When a Task raises an error in a P2P, it is not writing the error and failing the build correctly.

[eerhardt]

I presume issue no. 2 above should be moved to the nuget/home repo, yes?

Our numbers are switched around 😄, but yes the CliToolReference restore issue should be moved to nuget.

The other issue for P2P builds not failing correctly should be moved to https://github.com/microsoft/msbuild

/cc @rainersigwald @jeffkl @AndyGerlicher - FYI

[AArnott]

OK. I'll take care of creating new issues. Thanks for your help, @eerhardt.

[AArnott]

FYI @ctaggart so he knows SourceLink may fail this way as well given its DotNetCliToolReference dependency in the build.