Protobuf: Google.Protobuf.Tools proto compilation is only supported by default in a C# project (extension .csproj)

[jozefizso]

What version of protobuf and what language are you using?
Version: v3.7.0
Language: C#, .NET 4.7.2
Grpc.Tools: v1.18.0, v2.23.1

What operating system (Linux, Windows, ...) and version?
Windows, Visual Studio 2017 and 2019

What runtime / compiler are you using (e.g., python version or gcc version)
.NET Framework 4.7.2 + msbuild

What did you do?
Steps to reproduce the behavior:

1. Clone project https://github.com/jozefizso/Protobuf_SanityCheck
2. open it in Visual Studio 2017 and/or 2019
3. Restore nuget packages
4. Run Build > Build Solution
5. Build fails with error Google.Protobuf.Tools proto compilation is only supported by default in a C# project (extension .csproj)

What did you do?
Steps to reproduce the behavior:

1. Use the Protobuf_SanityCheck project from previous steps
2. Change the Service1.proto file and include a new rpc message
3. Run Build > Build Solution
4. Build fails with error Google.Protobuf.Tools proto compilation is only supported by default in a C# project (extension .csproj)

What did you expect to see
Project should build cleanly

What did you see instead?

msbuild error: Google.Protobuf.Tools proto compilation is only supported by default in a C# project (extension .csproj)

Full build log: https://gist.github.com/jozefizso/383d6d1346a0ffcd8dde5e5955384070

Anything else we should know about your project / environment
We hit this issue on a build servers which do clean builds of our solution. After upgrading to grpc 1.18 (from 1.14) there are random msbuild errors due to incorrect detection of C# projects.

I was able to reproduce this on a simple .NET project created from VS 2017 and doing clean checkout of it.

[jozefizso]

[jozefizso]

I can reproduce this with Grpc.Tools v2.23.1 as well.

https://github.com/jozefizso/Protobuf_SanityCheck/tree/grpc_1.23.1
https://gist.github.com/jozefizso/383d6d1346a0ffcd8dde5e5955384070#file-build_grpc-tools-v1-23-1-log

[jtattermusch]

@JunTaoLuo can you take a look? It seems that the same problem is impacting both Google.Protobuf.Tools and Grpc.Tools.

CC @kkm000

[kkm000]

I think it's our message.

[kkm000]

Yup. This:

https://github.com/grpc/grpc/blob/f92c283a5d6b77d77ae3d879ac13adb4e19a3a94/src/csharp/Grpc.Tools/build/_protobuf/Google.Protobuf.Tools.targets#L4-L8

and this:

https://github.com/grpc/grpc/blob/f92c283a5d6b77d77ae3d879ac13adb4e19a3a94/src/csharp/Grpc.Tools/build/_protobuf/Google.Protobuf.Tools.targets#L50-L54

I'll take a look.

[kkm000]

Oh, and this line in the build log is especially intriguing:

1>Language = C#

@jozefizso,

After upgrading to grpc 1.18 (from 1.14) there are random msbuild errors due to incorrect detection of C# projects.

How random is random? The same build fails on the same server? Are you doing test builds in containers? I'm wondering where the randnomness would have come from. Can it be a particular server or a group of servers?

For one, MSBuild can sometimes do, ehm, let's put it like significantly less than transparent things. E.g., if you set an environment variable Language, I think it will take precedence over project-level globals. When I was combing through MSBuild issues, I've seen an open issue that lists this as potentially unhermetizing the build, and the team's discussion how to get out of the conundrum w/o breaking build scripts in the whole world... That was a while ago, but issues like that are tremendously complex to solve, so I believe the priority of environment may still be the case.

[jozefizso]

I saw behavior like this:

1. Edit .proto file
2. Run Build - fails
3. Run Build - fails
4. Run Build - build succeeded
5. Run build several times again - it mostly works
6. Run Clean
7. Run Build - it fails again

I did not set Language explicitly in the project file https://github.com/jozefizso/Protobuf_SanityCheck/blob/master/ConsoleApp1/ConsoleApp1.csproj and I checked my environment settings and there's no Language variable there.

[jozefizso]

Environment variables:
https://gist.github.com/jozefizso/383d6d1346a0ffcd8dde5e5955384070#file-build-log-L7-L88

Initial Properties: https://gist.github.com/jozefizso/383d6d1346a0ffcd8dde5e5955384070#file-build-log-L90-L876

So Language was set by default script for .NET projects.

[kkm000]

I see an interesting pattern. If the package ./packages/Grpc.Tools.1.18.0/ exists, then build succeeds. A certain recipe to reproduce the failure is clean thoroughly (git clean -fdx). Then, as this is a "classic" using old package.config style, it does not restore packages during build; I have to invoke "Restore NuGet packages" from the Solution node context menu. The packages are indeed restored into the ./package/ subdirectory, but build fails to pick them up on the first build.

So MSBuild NuGet tooling is not involved; the restore is done by the NuGet VS extension. It's too easy to think that some of the latest versions of the extension do fail to mark the solution dirty, so that VS does not rebuild the project copy in memory, but then the first build after the restore consistently fails, the second consistently succeeds, and then... yeah, it looks random. This is VS16; I did not try VS15 yet.

I'm wondering what commands are you running on the build servers? If we could remove the whole VS from the repro, that would a tad simplify the problem... :)

[jozefizso]

On build server we use nuget restore and msbuild /t:Rebuild /p:Configuration=Release over the solution file. The restore happens before running msbuild.exe.

I hit the issue once again on a clean checkout code - as you described, I had to manually run Restore NuGet packages and restart Visual Studio before it could compile the .NET project from a clean state.

[mariusherzog]

I have the same problem. It works after I manually clean the project.

[apuch]

I see the same problem (Grpc.Tools2.26.0, .Net 4.7.1). On a fresh clone (never build a project with gRPC before), the build fails with that exact error message.

a) Is there a known workaround? We try to replace our current RPC solution with gRPC and this would be a major blocker.
b) Can I somehow help to diagnose this problem? Is more info required? What info do you need?

[jozefizso]

We still hit this issue on developer machines and we have no workaround for it.

[ArveSystad]

Deleting the .vs folder seems to fix it - it did for me.

• Old style .csproj
• VS 2019 (16.3.4)

EDIT: Added some more info.

[kkm000]

@ArveSystad: interesting. Too bad only one log has been posted, and it's not quite informative. VS caches a lot of things in this directory. The format also has changed between VS 2017 ans 2019.

There are many variables at play:

• VS version
• Package management style: “old”, with the "_solution_/package" directory, or new, when all packages are in the user directory.
• Project style: classic vs SDK. The thing is that the new NuGet package management can be used with classic project.

In short, it's a mess, and there is no common pattern that could be inferred. But what you did id sensible. Also, with the "new" package management style, the contents of the obj/ directory (excluding its subdirectories) matters: there are a few files that are not deleted when switching branches. Nobody reported if they are doing that or not. Also, for reasons I do not understand, the NuGet tooling is designed to keep this directory on project cleanup. Again, this applies only to project using the "new" NuGet tooling, with the central package storage (<PackageReference ...> items present in the project.

If this is reproducible using MSBuild instead of VS, that would be very helpful. If msbuild /t:Build /bl:somename.binlog;ProjectImports=embed (note the last switch), this binlog file would tell a whole lot to me. It will be less chatty if you runt /t:Clean separately without it, and only do /t:Build with it. It can accidentally a lot of information about your machine, so best of all share it with me privately, if you are ok with that. I'll certainly use it for as short a time as possible, shred it carefully, and I am encrypting all my drives.

If this happens only inside VS, there is also an option to save a .binlog from it, but that one is much more compact, and sometimes does not contain the required info. I'll explain how to get it, id MSBuild command-line build does not repro the issue.

I want to put this issue to rest. It pops up in so many different guises, that I'm not even sure if this is one problem.

---

By the way, switching to the new package management is the right thing to do, even if you want to keep "classic" csproj format. It's much more precise in dependency tracking, and keeps the graph of dependencies as graph, unlike the old, hierarchy-flattening package.config format, which is a simple transitive closure of the dependency DAG. The biggest and the most annoying problems it has been designed to solve are

• if main.exe project depends on library A, which in turn depends on B, and A needs package X that depends on W, while B depends on W explicitly, W must be a <Reference ... of main.exe project, or it will fail to compile, and listed in its package.config. After you remove dependencies, or if packages dependency change with an upgrade, there is no way to tell _why_ W in in main.exe's packages.config: does it really use W directly, or is this an artifact of the flattening dependency graph closure into a set?. Finding these becomes a nightmare over time.
• The modification of package.config files is handled by a VSIX extension in VS. <PackageReference ...>-style does not, and managed entirely by NuGet MSBuild tasks. You (a) cannot add a dependency without VS with the former, and (b) need to run nuget restore to set up packages in the headless build machine, which normally should not have VS, only the
  free Build Tools package. The new style has a /t:Restore MSBuild target, and this is even implicitly invoked
• The <Reference>s updated by the VSIX extension depend on the location of the project within a solution. So if a project is referred from two solutions (E.g., ./main.sln and ./core/core.sln, all references are off. This was a huge pain point; we had to run Git filters on the server that rejected PRs with relative paths like ../packages/xxx or ../../packages/xxx, and had to modify them to $(SolutionDir)packages/xxx, which was gladly reverted into ../(../)+ every time you do anything in the VS package management tool, e.g. upgrade a package.

The rework is manual (remove package.config, all s to directly packaged be auto-sensed by VS, and manually add <PackageReference ...> for direct dependencies only. To us, it took time to rework paid off tremendously. An important thing to keep in mind that after checking out different styles of package management it's crucial to delete all obj/ directories.