Protobuf: No way to use custom options in Ruby

[kailan]

I created a custom option extension that I use in my proto files, like this:

service Fishtank {
  rpc GetFish(FishQuery) returns (Fish) {
    option (protapi.http).get = "/fish/{id}";
  }
}

It doesn't seem that once it is compiled into Ruby code, I can access the value of the protapi.http option. I believe this can be done in other languages by accessing the descriptor.

Am I missing something, or is this not possible right now?

[haberman]

You are correct that Ruby does not yet support custom options. Hopefully we can add this before too long.

[kailan]

@haberman Awesome! It'll really help out a project of mine.

Any way I can help out with implementing this?

[skipperguy12]

This would be helpful for me as well

[haberman]

This is becoming a more and more common request. I have been thinking through how the implementation ought to work on this. It's slightly tricky to modify the DSL for this, because custom options let you declare the option and use it both in the same file. This is a challenge to accommodate in a clean way.

[kailan]

Any progress on this? @haberman

[haberman]

Ok there are a couple parts to making this work.

First of all we need to support proto2 field presence. Right now the code generator fails immediately if it sees a file with syntax="proto2";. Step 1 would be a PR that supports proto2 files that don't contain any extensions or custom options. Proto2 "optional" fields should be supported by adding hasbits to the in-memory representation to indicate whether a field is present or not. I'm happy to explain this in more detail for anyone who wants to implement this.

Secondly we need to support proto2 extensions. This means extending the DSL with something like add_extension and making it so you can read and write extension fields of extended messages with something like msg[my_extension] = 5.

Finally we need a way of extending the DSL to support custom options. We can talk more about this part once #1 and #2 are done.

I'd be happy to see pull requests for any/all of these, and I'd be happy to help guide people who want to implement this to the right places in the code.

[kailan]

I'd love to work on this but I have one question: Why would proto2 need to be supported? Surely implementing this for proto3 would be enough.

[haberman]

Custom options are defined as extensions of messages descriptor.proto, which is a syntax="proto2"; file. Extensions don't exist at all in proto3.

So even if your messages are only proto3, we need proto2 support to be able to represent the options themselves.

[kailan]

Ah, makes sense. I'll give it a shot later on :+1:

[nerdrew]

Any status updates on proto2 support?

[djudd-stripe]

Any progress since last year? I might have a few days to push this forward.

[nerdrew]

Funny. We were thinking of looking at this soon too. cc @zanker

[zanker]

Hah, @djudd-stripe I was actually talking to @nerdrew about this last week and we assumed that Stripe didn't use protobufs. Are you using the "protobuf" gem currently?

I was able to hack something to support defaults in a few days, so I don't think it's too hard to at least get basic protobuf2 support at least. It looks like most of the work is building out the Ruby <-> C bridges, as UPB handles everything already.

I'm still working on proving this out internally to make sure we can feasibly swap to this, but right now we're interested in adding at least basic synta2 support.

@haberman it sounds like on your end, you would be interested in merging syntax2 support to the Ruby gem, as long as it meets code standards and is done properly?

[djudd-stripe]

Not a lot of Protobuf at Stripe right now but we are exploring adding more. We're using this code so far but are interested in custom annotations.

[djudd-stripe]

@zanker it sounds like you have Proto2 support work in-progress? Anything more useful I can do than wait for you to get it merged, then maybe add custom option support on top (unless you've included that already)?

[zanker]

Thanks for the offer, I don't think there's anything right now. I was able to get a more proper one (using hasbits) working, I need a few days to sort it out and clean/test things and I'll submit a PR.

I'm just starting with the syntax2/default parts initially. If you wanted to add custom option support on top of that then 👍 . Will end up doing it otherwise.

[zanker]

@djudd-stripe This is still very much WIP (hence the random debug statements), but for reference: https://github.com/google/protobuf/compare/master...zanker:zanker/syntax2 is what things are looking like. I don't think wiring in extensions would be _too_ hard overall.

[pcj]

Any progress on this? AFAIK there is no way to work with descriptor.proto in ruby.

~protobuf
syntax = "proto3";
package foo;
import "google/protobuf/descriptor.proto";
~

:cry:

[zanker]

Sorry! I left Square back in October and the team who was going to take it over got reshuffled so they're not going to have time to pick it back up.

The PR is at https://github.com/google/protobuf/pull/3606, and all the proto side code is pretty solid from the testing I did. The blocker was the proto syntax, which wasn't narrowed down until after I had already left.

If you want to pick up that part of the PR, I'd be happy to walk you through on whatever part of the PR isn't clear.

[pechorin]

Is where any plans to implement custom options in ruby? Custom options looks great for expressions some meta logic and it is so sad what ruby does not include this abilitites.

My primary interest is extending enum options.

[ebenoist]

It looks like there _is_ support in upb.c. What is the thinking around what the syntax would look like? If it's straightforward I can put up a PR.

[pechorin]

Custom options currently cannot be compiled via protoc2, but can be compiled via protoc3 but options will be skipped in compiled ruby pb file. So it will be cool to provide meta options like this in ruby output file:

syntax = "proto2";

import "google/protobuf/descriptor.proto";

extend google.protobuf.EnumValueOptions {
   optional string name = 1001;
}
message Operation {

  enum ClassName {
    OPERATION = 0 [(name) = 'Operation'];
    PUSH_OPERATION = 1  [(name) = 'PushOperation'];
  }
}

So, you ask about how it will be implemented in ruby generated classes? Am i understand you right?

[ebenoist]

@pechorin Thats correct, just looking for guidance on the Ruby syntax/methods for the generated classes. It seems like upb.c already has support for reading out the options, they would just need to be exposed.