Protobuf: How to learn to use protoc in 21+ easily infuriating steps

[Ark-kun]

I was trying to generate C# code from a very small .proto file. Here is my console log. In reality it was twice as long due to duplicate commands issued during this stochastic goal search. To be hones, I'm appalled at the quality of the program command-line interface.

>protoc --version
libprotoc 3.2.0

>protoc
Missing input file.

>protoc a:\proto_dir\CNTK.proto
Missing output directives.

>protoc a:\proto_dir\CNTK.proto --csharp_out A:\Output directory\cntk.proto.cs
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I).  You must specify a --proto_path which encompasses this file.  Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).

>protoc a:\proto_dir\CNTK.proto --csharp_out A:\Output directory\cntk.proto.cs --proto_path a:\proto_dir\
directory: No such file or directory

>protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory\cntk.proto.cs" --proto_path a:\proto_dir\
A:\Output directory\cntk.proto.cs/: No such file or directory

>protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I).  You must specify a --proto_path which encompasses this file.  Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).

>protoc CNTK.proto --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\
CNTK.proto: No such file or directory

>protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory\" --proto_path a:\proto_dir
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I).  You must specify a --proto_path which encompasses this file.  Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).

>protoc --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\ a:\proto_dir\CNTK.proto
Missing input file.

>protoc --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\ CNTK.proto
Missing input file.

>protoc --csharp_out "A:\Output directory\CNTK.proto.cs" a:\proto_dir\CNTK.proto
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I).  You must specify a --proto_path which encompasses this file.  Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).

>protoc --csharp_out "A:\Output directory\CNTK.proto.cs" --proto_path=a:\proto_dir\ a:\proto_dir\CNTK.proto
A:\Output directory\CNTK.proto.cs/: No such file or directory

>protoc --csharp_out "A:\Output directory\" --proto_path=a:\proto_dir\ a:\proto_dir\CNTK.proto
Missing input file.

>protoc --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir\" a:\proto_dir\CNTK.proto
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I).  You must specify a --proto_path which encompasses this file.  Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).

>protoc --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir\" "a:\proto_dir\CNTK.proto"
a:\proto_dir\CNTK.proto: File does not reside within any path specified using --proto_path (or -I).  You must specify a --proto_path which encompasses this file.  Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).

>protoc --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir" "a:\proto_dir\CNTK.proto"
Missing input file.

>cd A:\proto_dir

A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir" CNTK.proto
Missing input file.

A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" --proto_path=. CNTK.proto
Missing input file.

A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" --proto_path=./ CNTK.proto
Missing input file.

A:\proto_dir>protoc.exe --csharp_out=. CNTK.proto

A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" CNTK.proto
Missing input file.

A:\proto_dir>protoc.exe --csharp_out="A:\" CNTK.proto
Missing input file.

A:\proto_dir>protoc.exe --csharp_out="A:/" CNTK.proto

A:\proto_dir>protoc.exe --csharp_out="A:/Output directory/" --proto_path=./ CNTK.proto

[xfxyjwf]

It might be easier to start with following the tutorials first:
https://developers.google.com/protocol-buffers/docs/csharptutorial#compiling-your-protocol-buffers

[mihajloZg]

@Ark-kun
I second your criticism. This is just horrible. Instead of actual software devolopment I have to play this guessing games with command line :/

[SephVelut]

The cli really is badly designed. It could use more helpful messages when outputting errors.

[jjpepper]

I take particular issue with "File does not reside within any path specified using --proto_path (or -I). You must specify a --proto_path which encompasses this file. Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think)"

Firstly, how do they know what I think. Secondly, it's not hard. Not even what I would remotely call a "hard" problem. Let's have a try, what's the normal steps to solve a problem like this.

1. Google it. I tried googling "how to test if paths are equivalent C++ filesystem"
   the top result was this.
2. Oh wait, step 1 solved it. Actually it was easier than I thought.

[dfontenot]

I have yet to find a way to toggle the verbosity of protoc. Is this possible? That would really help to double-check that the CLI arguments were set correctly.

[xfxyjwf]

@dfontenot There isn't a verbosity mode in protoc. Inside Google we rarely need to invoke protoc directly (the internal version of bazel takes care of everything) so not much attention has been given to the command line interface. If you have suggestions on how to improve the current error messages, feel free to send us pull requests.

[xfxyjwf]

I went through the list in https://github.com/google/protobuf/issues/3028#issue-225299414 and believe many of the commands should have worked. Created https://github.com/google/protobuf/issues/4993 to fix protoc on windows.

[Ark-kun]

@xfxyjwf With all due respect the documentation link you've provided does not provide much help.
Here is the single line mentioning protoc there:

protoc -I=$SRC_DIR --csharp_out=$DST_DIR $SRC_DIR/addressbook.proto

I have 17+ lines that conform to that documentation, yet fail to work:

protoc a:\proto_dir\CNTK.proto --csharp_out A:\Output directory\cntk.proto.cs --proto_path a:\proto_dir\
protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory\cntk.proto.cs" --proto_path a:\proto_dir\
protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\
protoc CNTK.proto --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\
protoc a:\proto_dir\CNTK.proto --csharp_out "A:\Output directory\" --proto_path a:\proto_dir
protoc --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\ a:\proto_dir\CNTK.proto
protoc --csharp_out "A:\Output directory\" --proto_path a:\proto_dir\ CNTK.proto
protoc --csharp_out "A:\Output directory\CNTK.proto.cs" a:\proto_dir\CNTK.proto
protoc --csharp_out "A:\Output directory\CNTK.proto.cs" --proto_path=a:\proto_dir\ a:\proto_dir\CNTK.proto
protoc --csharp_out "A:\Output directory\" --proto_path=a:\proto_dir\ a:\proto_dir\CNTK.proto
protoc --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir\" a:\proto_dir\CNTK.proto
protoc --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir\" "a:\proto_dir\CNTK.proto"
protoc --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir" "a:\proto_dir\CNTK.proto"
A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" --proto_path="a:\proto_dir" CNTK.proto
A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" --proto_path=. CNTK.proto
A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" --proto_path=./ CNTK.proto
A:\proto_dir>protoc.exe --csharp_out="A:\Output directory\" CNTK.proto

[jjpepper]

Hey @TeBoring I saw you added this to ToDO in "Weekly Fixit". I've gone and had a look at the path resolution code. There is a comment from "kenton" stating that in essence the current behaviour of disallowing ".." relative paths is to support some obscure setup where people have symbolic links in their source code. Symbolic links are there to hack around problems of inflexible software that hard codes paths. This is a perfect example of some hypothetical few being favoured over the many that would have much more hair if they were allowed to use relative paths to proto files. Please allow me to specify relative paths like normal software does.

[amelvill]

It seems like no matter how you specify the path to the .proto file, protoc only works if the .proto file is lower in the folder hierarchy than where you called protoc. Your .proto file seems to need to be in a subfolder of the folder you call protoc in.

For example, you can never do this,
protoc --cpp_out=. ../myfile.proto

Even more strange, you can't even use an absolute path instead of ..:
protoc --cpp_out=. /path/to/myfile.proto

Using proto_path seems to have no effect.

However this will work:
protoc --cpp_out=. somefolder/myfile.proto