Protobuf: Include MS Build task for proto to C# generation

[jskeet]

Currently running the codegen for C# is a manual step. We should include an MS Build task in the NuGet package if we can. That will require some manual intervention to use in project files, but it's better than nothing.

Ccs:
@igorgatis, @csmith, @mmdriley who have been interested in making the experience better

@kkm000 and @jtattermusch who have been thinking about this for GRPC: https://github.com/grpc/grpc/issues/4805

Advice welcome :)

[jskeet]

Okay, I've been thinking about this for a few days now, and the sheer breadth of possible options is a bit overwhelming.

Some aspects to think about:

Platform support

Ideally, we'd want to make it easy to build applications using protos in:

• Visual Studio using msbuild-based solutions
• MonoDevelop / Xamarin Studio
• project.json-based solutions (regardless of IDE)

I suspect that a single solution won't fit all of those; having an msbuild task would be better than nothing.

File locations and input/output

• Some developers may want to have their proto files alongside their C# code; others may have a separate directory containing the protos. (This directory might not even be within the solution - particularly if it's a cross-platform app.)
• Some developers may want the generated code to be visible and even checked in. (That's my preference, for example - it means that anyone who wants to build the protobuf runtime library is able to just clone it and build it, without having to get a working protoc.) Others may want it to be generated as a temporary build artifact.
• For efficiency, it would be useful to be able to pass _all_ the .proto files into protoc in a single run, otherwise we can end up with an ~O(n^2) situation where protoc needs to _parse_ (almost) all the files several times (for validity) but only outputs one C# file at a time. It's not clear to me how this should be done, although it may be possible. (I know very little about msbuild, but obviously the C# compiler gets all its files in one go...)
• We need to have the ability to include other directories on the proto path. If this can be done entirely within Visual Studio, so much the better.
• As a twist on the two above points, it's possible that different proto files will want different paths - which may even have different messages with the same name. So we'd actually want to call protoc as many times as we have distinct paths. (Ick.)
• Each input .proto file creates a single output .cs file, but predicting the name basically requires writing code (equivalent to this
• Additionally, there are protoc options to determine the output directory for each file automatically based on the package name and a "base" namespace.

GRPC

We'd want to be able to autogenerate the GRPC service as well, which will be a separate .cs file. We can detect the presence of the grpc tools package, which is helpful - but this adds another layer of complexity on the "predicting the output file" part of the above.

[jskeet]

// cc @mgravell for commentary on what protobuf-net does for this.

[mgravell]

Protobuf-net doesn't have any magic wands here. At the moment, it provides
a VS code-gen tool in the IDE (not an MSBuild task), and has command-line
tools as well, but: behind the scenes this unpacks protoc (embedded),
executes it to generate a parsed protobuf-format schema, and processes that
schema via deserialization. Not very pretty.

I would be reluctant to invest time in an MSBuild task at this point, since
that doesn't play with .net core ("dnu build", or "dotnet build"). I am
also painfully aware of the OS dependence that protoc imposes.

My ideal scenario would be a library (perhaps shared - I'd happily chip in)
that offers parsing services into an object model, pure .net, ideally
core-clr compatible. This could then be used for library-based parsing, as
well as a basis for console exes, msbuild tasks, and IDE codegen tools.

I have previously looked at existing general purpose .net parsers - "
Irony" looked promising, but I haven't got very far. I got swamped by
various other things before I got it off the ground.

On Wed, 3 Feb 2016 12:14 Jon Skeet [email protected] wrote:

// cc @mgravell https://github.com/mgravell for commentary on what
protobuf-net does for this.

—
Reply to this email directly or view it on GitHub
https://github.com/google/protobuf/issues/1193#issuecomment-179192372.

[gambrose]

I would vote for a msbuild targets file shipped with the tools nuget package.
Nuget does a good job of including linking in these extra msbuild targets making the experience quite seamless. See StyleCop.MSBuild package for example of including a tool into the build via nuget. This is a lot less painful than having to install a vs 'custom tool'.

The target file could compile proto items using sensible defaults. To make it easy for a user to customize these defaults then could try and import a well known target file in the project dir. This is how web publishing customization works http://www.asp.net/mvc/overview/deployment/visual-studio-web-deployment/deploying-extra-files

[jskeet]

As MSBuild is now going to be the eventual project system for .NET Core, it makes more sense to revisit this. Thanks for the suggestion of the configuration file, @gambrose - will have a look when I get more time. I think it's very unlikely that we'll get anything in here in time for the GA release, but it can be added later.

[xfxyjwf]

@jskeet what's status of this issue? If it's not fixed already, we probably wouldn't be able to include it in GA release.

[jskeet]

@xfxyjwf: Definitely not for GA. May come back to it later, but it's somewhat fraught with trickiness.

[listepo]

@jskeet what's status of this issue? more than a year since the last discussion

[jskeet]

There's a lot of discussion in gRPC, which would affect this: grpc/grpc#13207

[chrisdunelm]

I'd also find this helpful. I'm not using gRPC, only protobuf, so a protobuf-only build task would be great.

[kkm000]

@chrisdunelm, please follow grpc/grpc#13207. My goal is to release the Grpc.Tools package (with Protobuf support available without gRPC, you should be able to use it as is, or disable gRPC in a more advanced usage); then split into Protobuf-only and Grpc package depending on Protobuf.

The only current disadvantage is a slightly larger package size.

What is your exact use case? I. e. what is the target language, how large is the project, do you keep protoc outputs for any reason, or only interested in compiling them into a dll, and such.

[dejanstojanovic]

I did something similar, but it is a part of csproj file, so I do not think it will be picked up by msbuild, but you can maybe try to re-use it. Works fine for me from Visual Studio https://dejanstojanovic.net/aspnet/2018/june/generate-c-models-from-protobuf-proto-files-directly-from-visual-studio/

[mgravell]

@dejanstojanovic interesting, thanks. Since you are mentioning protobuf-net, a minor correction:

which means we need to got to GitHub protobuf-project release page, download the latest release and build it with Visual Studio

Actually it is available as a "global tool":

dotnet tool install --global protobuf-net.Protogen

(then just type protogen - works on Linux, Windows, etc)

Or as a zip download here: https://protogen.marcgravell.com

Would a nuget of the windows binaries be useful as a third option? Not sure of the best way to package them in a nuspec for that scenario.

[dejanstojanovic]

Thanks @mgravell didn't know about that, I guess I learn the hard way :)

[resnyanskiy]

Workaround using custom target in .csproj. It executes protoc for all .proto files in Protos directory of a project, which creates .g.cs files in Contract directory. You need to install Google.Protobuf.Tools package.

  <Target Name="Protoc" BeforeTargets="BeforeBuild">
    <ItemGroup>
      <Protos Include="$(ProjectDir)Protos\*.proto" />
    </ItemGroup>
    <Exec Command="$(NugetPackageRoot)google.protobuf.tools\3.6.0\tools\windows_x86\protoc.exe --proto_path=$(ProjectDir)Protos\ --csharp_out=$(ProjectDir)Contract\ --csharp_opt=internal_access,file_extension=.g.cs @(Protos, ' ')" />
  </Target>

[kkm000]

That is how everyone does it. Hardcoded paths, and additionally machine specific, so it works on my machine but breaks on yours.

If my changes are ever and accepted reviewed, we'll have a nice fully integrated system. Too bad it takes months on end to each of us for each step, but we are all overloaded, sorry.

[NN---]

Any updates here ?
Grpc.Tools MSBuild has been already merged: https://github.com/grpc/grpc/pull/13207

[kkm000]

@NN---, our plans are to go through at least one minor cycle (i. e. up to gRPC 1.8), maybe more, and receive feedback on the tools from out there. The reason is it is easier to change when one project is involved. When the code stabilizes, we are planning to split it into protoc and gRPC tooling NuGet packages that would recognize each other when installed together; all code is already written this way with split points prepared and marked in MSBuild scripts. When Grpc.Tools 1.7.1 is out, you can use it for proto generation alone (the integration readme mentions how to disable gRPC codegen entirely).

[hanabi1224]

@jskeet @kkm000 I have recently created a Protobuf.CodeGen.BuildTask package to achieve this. Wonder if it's possible to merge it into this official repo

[NN---]

Any update ?
When an MSBuild task will be official ?

Currently I use this trick

  <Target Name="GenerateFiles" BeforeTargets="PreBuildEvent" Inputs="$(ProjectDir)my.proto" Outputs="my.pb.g.cs">
    <CreateProperty Value="%(PackageReference.Version)" Condition="%(PackageReference.Identity) == 'Google.Protobuf.Tools'">
      <Output TaskParameter="Value" PropertyName="ProtocCompilerVersion" />
    </CreateProperty>

    <PropertyGroup>
      <ProtocCompiler Condition="'$(OS)' == 'Windows_NT'">$(NuGetPackageRoot)Google.Protobuf.Tools/$(ProtocCompilerVersion)/tools/windows_x64/protoc.exe</ProtocCompiler>
      <ProtocCompiler Condition="'$([System.Runtime.InteropServices.RuntimeInformation]::IsOSPlatform($([System.Runtime.InteropServices.OSPlatform]::OSX)))'">$(NuGetPackageRoot)google.protobuf.tools/$(ProtocCompilerVersion)/tools/macosx_x64/protoc</ProtocCompiler>
      <ProtocCompiler Condition="'$([System.Runtime.InteropServices.RuntimeInformation]::IsOSPlatform($([System.Runtime.InteropServices.OSPlatform]::Linux)))'">$(NuGetPackageRoot)google.protobuf.tools/$(ProtocCompilerVersion)/tools/linux_x64/protoc</ProtocCompiler>
    </PropertyGroup>

    <Exec Command="&quot;$(ProtocCompiler)&quot; --proto_path=&quot;$(ProjectDir)&quot; --csharp_out=&quot;$(ProjectDir)\&quot; --csharp_opt=file_extension=.pb.g.cs &quot;$(ProjectDir)my.proto&quot;" Outputs="my.pb.g.cs" />

    <ItemGroup>
      <Compile Include="**\*.g.*" Exclude="@(Compile)" />
    </ItemGroup>
  </Target>

[jtattermusch]

@NN--- you can use the Grpc.Tools package from grpc which has had this feature for long time already.
https://github.com/grpc/grpc/blob/master/src/csharp/BUILD-INTEGRATION.md

[NN---]

It works.
Thanks.