Protobuf: Decimal Type?

[simplesteph]

It'd be amazing to have a Decimal type as a Well Known Type that can be leveraged across all the languages. I have no preference around the back-end implementation of it, as an end user I just think it'd greatly simplify codebases using Decimals

[xfxyjwf]

By "Decimal", I guess you mean something like java.math.BigDecimal in Java that can represent arbitrary precision. I think we should only introduce it if such a decimal type is supported as a built-in type or standard library type by all programming languages that we officially support. That probably won't happen any time soon though.

[simplesteph]

Yes exactly !

[bric3]

The java implementation could understand how to serialize BigDecimal or BigInteger to/from strings. I think this is a missed opportunity. Without it this causes developers to use floating pointing types instead, and use these type for calculation which may be bad. Leveraging protobuf to promote these types would help to use consistent and safe way to handle precision numbers.

[bugproof]

I'm currently struggling with it myself (I'm using C#). I think you can't avoid having to map(convert) to decimal manually everytime you want to use decimal.

Some people pointed to https://github.com/googleapis/googleapis/blob/master/google/type/money.proto but there's no information on how to convert from nanos and to nanos.

And it gives you the same result as just storing it as a string.

I wanted to rewrite my code to use gRPC but the state of things with decimal is really putting me off.

[TeBoring]

Maybe a util method from protobuf's supported type (e.g., string) to the decimal type in the specific language?

[simplesteph]

@TeBoring that'd be the easier one. I'd rather have this than Avro's bytes method, as it destroys the user experience, but I'm sure their choices would be important to understand https://avro.apache.org/docs/1.8.1/spec.html#Decimal https://issues.apache.org/jira/browse/AVRO-1402

[TeBoring]

@anandolee, this seems to be c# specific. Could you follow on that?

[tephe]

We are also using decimals quite heavily in our application and this is currently an issue for the migration to protobuf. Is there any update on this?

[mrunesson]

@TeBoring I would say this is not C# specific. We have:

• BigDecimal in Java https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/math/BigDecimal.html
• decimal in Python https://docs.python.org/3/library/decimal.html
  It is really important for all systems handling finance data.
  I also think a util function converting from string would be bad from a performance perspective. Better to use something like the money type, without currency, as input to the util function. Or look on Avros Logical type for decimal https://avro.apache.org/docs/1.8.0/spec.html#Decimal. Both ways would be more efficient than string.

[GrigorievNick]

https://avro.apache.org/docs/1.8.2/spec.html#Decimal
When we talk about 'Decimal' most of us mean type that can be keep 'precision' and 'scale' value for float values. Avro is good example, they use unify serialisations of decimal to Bytes + keep two additional properties (scale, precision).
Why its important:
All use same serialisations process
Previously some serialised it to string, some to Byte Array with Ascending Order, some With Descending, etc
Now every one use same bytes, with same order, and keep metadata in same fields.

[maurodelazeri]

decimal would be very handy, I'm currently converting to a byte array base-256 digit, I guess is the most efficient way

[mikev]

"While suitable for many purposes, binary floating-point arithmetic should not be used for financial, commercial, and user-centric applications"

Yes, Google we need a Decimal type! This is an example of the issue and probably demonstrates the issue in any language:
bool result = ((0.1 + 0.2) == 0.3); // result is false

The following link provides a detailed explanation and examples to explain the motivation for the "Decimal" data type, which is available in Python, Java and C#.
http://speleotrove.com/decimal/

[jtattermusch]

Update: I'm currently driving internal proposal for the "decimal" type and I published the proposal document externally, so that people can chime in as needed.

See https://github.com/protocolbuffers/protobuf/pull/7039

[ritaly]

There's class BigDecimal in Ruby. So strong +1 to add Decimal to protobuf types 👍

I'd get rid of workaround for Google::Protobuf::TypeError: Expected number type for double field (given BigDecimal).

[vivainio]

Decimals are exceptionally popular in C#, especially with database-backed business software. Almost all the "number" types in our application (and there is a lot of them) are decimals, there are only few floats here and there. Since float is not a safe replacement, we would end up mapping decimals to strings which is sort of embarrassing (and unsafe since not every string is a valid decimal).

Since it's a missing fundamental type, it should be in WellKnownTypes (as the suggested FixedDecimal). Alternatively, there should be another source of these "well known types" to avoid everyone copy-pasting the FixedDecimal to their codebases (in different namespaces)

[LiamMorrow]

This doesn't solve this in the WellKnownTypes land, but here we use:

message BigDecimal {
  string value = 1; // feel free to substitute for a variant of precision/scale/unscaled
}

Then in the csharp land we can define some implicit/explicit casting operators or conversion methods via extension of the partial class:

public partial class BigDecimal
{
    public static implicit operator BigDecimal(decimal value)
    {
        return new BigDecimal
        {
            Value = value.ToString(CultureInfo.InvariantCulture),
        };
    }

    public static implicit operator decimal(BigDecimal value)
    {
        return decimal.Parse(value.Value, CultureInfo.InvariantCulture);
    }
}

[csmorley]

Are there any preview ports of this? Writing a finance app with millions of messages per second, non-native support and casting each time is a non-starter for gRPC currently without this

[jufemaiz]

For those working with decimals, can I confirm that there are two approaches that we are left with while we await movement on #7039 :

1. Use of string marshalling for decimals
2. Use of a custom protobuf message with either units and nanos (ala Google's Money.proto example) or an coefficient, exponent and sign (ala big.js)

Then hand off the expected marshalling of such messages to internal implementations of each application.

If this is the case, can people chime in with their preferred approach to date?

[csmorley]

For those working with decimals, can I confirm that there are two approaches that we are left with while we await movement on #7039 :

1. Use of string marshalling for decimals
2. Use of a custom protobuf message with either units and nanos (ala Google's Money.proto example) or an coefficient, exponent and sign (ala big.js)

Then hand off the expected marshalling of such messages to internal implementations of each application.

If this is the case, can people chime in with their preferred approach to date?

Many thanks! Personally, would prefer custom profobuf message (this is what i have been doing to date). If there are any one-time/optimised marshalling that can be implemented as part of the codegen to prevent repeated casting that would be my main concern

[mtrtm]

There's a reference to this and a workaround here for wcf/recode, but I can't believe this is still an open issue
https://visualrecode.com/blog/csharp-decimals-in-grpc/

[D3MaxT]

Likewise. Decimal is an absolutely critical type for handling money in C#. Please add this to WellKnownTypes.

Thank you!