Protobuf: Allow defining constant values to be added

[mitar]

We are using protobuf through gRPC and would like to include in the message definition a value which is provided in the .proto file and not by the program. For example, something like that:

message SessionRequest {
    string version = 1 [constant="0.1.0"];
}

The purpose of such version is to make debugging easier: if communication fails, we can log the versions and use that to debug further.

[liujisi]

Have you tried with the default value with proto2 syntax? In proto3, we deliberated removed the support on default values to simplify the language semantics. If you need the default value semantic, you can always fall back to use proto2. -- you can have a specific proto2 message just for defaults, and use the proto2 message in your other proto3 messages.

[mitar]

Hm, we are trying to use proto3 for gRPC. For proto2 I have seen the approach with defaults.

[liujisi]

This is essentially default values in proto2. Main downside:

• Different users of the proto may want different default values. It shouldn't be configured at proto level.
• If you change the defaults in the proto file, then the behavior/meaning of the proto message changes.
• With field presence removed, default cannot be implemented - you need the concept of not-set, then we bring back the confusion about empty vs not set.

I don't see we will bring it back in proto3. As mentioned in the previous comment, you could always use a specific configuration proto2 message, or use a different method for storing default values. e.g. have a text configuration (in json or text format) and load it into the program as the default value message.

[mitar]

So, no. This is pretty different from default values. It should be a value hard-coded in the protocol spec and simply send as it is with the message it is defined in. Users should not be able to override it. Defaults can work for that, but users can still (by accident) override it. So defaults are a workaround, but what I am asking are not defaults, but a constant value defined in the spec users do not have to specify, or even cannot specify. Simply, a constant in the message.

The important part is that this value is in the spec, so that users do not have to worry to be out of the sync with the spec. Existence of the value should not change between versions (like is normal in proto buffers), but values can change. So there are not changes in behavior/meaning of the proto message as values change.

Currently, I implemented this with the following workaround.

import "google/protobuf/descriptor.proto";

extend google.protobuf.FileOptions {
    string protocol_version = 51000;
}

option (protocol_version) = "0.1.0";

message SessionRequest {
    string version = 1;
}

The idea is that users of the protocol should always provide version what they get from the option extension:

import test_service_pb2
version = test_service_pb2.DESCRIPTOR.GetOptions().Extensions[test_service_pb2.protocol_version]

So if the user of the spec updates the spec file to the new version, then version value in messages will be updated automatically.

This is of course not ideal. You still have to rely on users to pass the value from the option extension and not pass some other value, but at least users do not have to worry about changing the version when they update the spec.

[drew-512]

It seems messed up that proto3 has no ability for even string or int constants. This seems to be an obvious use case for a proto file. Guys, this isn't asking for a lot given all the stuff pbs already do, am I wrong? Please consider it!?

Protobufs is an AMAZING tool and it's such a lovely gift to be working with! Kindly add consts and that solidifies protobufs to the next level.

Thanks heros!!

[4nte]

Ability to specify constants would bring so much value and practicality to everyone, yet it's so trivial and simple to implement. Can we talk about this?

[akashagrahari-mt]

I was shocked as well, that I cannot define constants on the proto level. Is this being taken up? Defining constant literals is something that almost every language supports, and is vital to avoid repetition of data in code

[ModischFabrications]

I feel the same way. I came here looking for a way to set a program readable constant in proto files, similar to #define in C, LABEL in Dockerfiles or const static values in Java.

[Gr33nbl00d]

fully agree this should be possible

[lemonlinger]

+1

[cheatmenot]

yep I agree, like @mitar said, it can be specified on the spec to avoid sync problems. Doing this, will open up new possibilities and use cases for protobuf. Thanks! Hope you will add this request.

[gabrielps]

I too would like to see a mechanism to define constants, e.g. API error codes.

[kurt-cb]

here here

[cheatmenot]

+1

[yinthewind]

+1

[tsc1989]

+1

[Gaspa79]

+1

Honestly why was this closed??

[mickep76]

+1

Seems like there is a disconnect here, trying to keep things simple by removing it from version 2 but making it less usable in the process doesn't seem like the right direction to me?

[sne4ky]

+1

This feature would be very useful, please take it into consideration

[anzellai]

+1

There are genuine use cases and no acceptable alternatives so far being suggested.

Can this issue be re-opened, or at least explain why this is considered closed?

[iulian0512]

+1
Very useful feature.

[adrianschlatter]

+1

I need to store the version of the proto spec in the encoded data. Defaults are not encoded on the wire and therefore no alternative.

[mouradmourafiq]

Definitely a very useful feature.

[cheatmenot]

+1 very useful feature

[XiaohuaRen]

+1 very very useful feature

[4nte]

Can we get this issue Opened?
There seems to be a lot of interest for this feature, it would do community a favor if it was marked as open.

@mitar

[wawade]

It would also also benefit me if the feature was added.
So +1

[kurt-cb]

+1

I solved this a different way, but it is a band-aid and only works for C++ and python. Basically, I wrote a python tool that parses a simple c++ header and generates python constants.

See: common/manifest/h2py.py (.h converter).
A make rule reads the manifest_dir.hh and creates the manifest_dir_st.hh… then includes that in the pybind section of the C lib.

Again, a real HACK. This solution would be cleaner.

Kurt

From: wawade [e-mail]
Reply-To: protocolbuffers/protobuf [e-mail]
Date: Monday, February 24, 2020 at 4:02 AM
To: protocolbuffers/protobuf [e-mail]
Cc: Kurt Godwin [e-mail], Comment [e-mail]
Subject: Re: [protocolbuffers/protobuf] Allow defining constant values to be added (#3520)

It would also also benefit me if the feature was added.
So +1

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/protocolbuffers/protobuf/issues/3520?email_source=notifications&email_token=ALVJPF4NQBAF57EVWMEDIZ3REOZMXA5CNFSM4DXM3ZLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMXRK2I#issuecomment-590288233, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALVJPF5PCV7L6ORFKRQJ4UTREOZMXANCNFSM4DXM3ZLA.

[mshekhar]

+1

[rajan-chari]

+1

[swjain]

+1
This is must have. Please consider this request.

[cheatmenot]

+1

[promethe42]

+1

I don't understand how a "protocol" can exist without utility constant values that can be shared by all peers.
Having to define those constants in language specific separate source files defeats the whole purpose of Protobuf.

[acozzette]

It sounds like there are a couple different feature requests here. One is the ability to define a constant in a .proto file. That is already doable with custom options as some have already pointed out above. The other idea is going beyond this and always serializing the constant inside the message, similar to a default value except that it is always set and can't be overridden by code. I can see the motivation for this as a version number to help with debugging, but I don't think it makes sense to implement within protobuf. First of all, even if it's a simple idea, there is a large cost to it given that it has to be implemented separately for every language. Second, it's hard to conceptualize what exactly the constant means. In a distributed system, a protobuf message can be passed around and modified by many different entities with different schema versions. Does the constant get set by the first entity to create the message, or the last one to view/modify it? For example, if a proxy examines a protobuf message and then passes it along unchanged to another server, should it update the constant based on its view of the schema? I don't think there is any easy way to make this work reliably without surprises.

[noel-yap]

https://stackoverflow.com/questions/11474416/define-dictionary-in-protocol-buffer/64416797#64416797 another extension-based solution (using enums).

[virtumonde]

Please consider adding constants.

[JohnKarel]

+1