Protobuf: There is no Java API for adding arrays of primitives to repeated fields without boxing

[oridb]

I would like to have an API for primitive types of repeated fields, in order to efficiently set them. The reason I need this is that I am transferring a large array of floats (representing the pixels of a thermal image), and would be happier if there was no need to box each float before adding it to the protobuf.

Ideally, the protobuf would just provide a function addAllBuffer(float[] data) in addition to the current iterator based APIs. Currently, all of the APIs for adding an array involve boxing the data at some point or another.

The system setup is a little bit odd, since the grpc server is actually running on an android device, but I think this request applies in general.

Should this be an issue in the gRPC issue tracker?

This is a new feature request. Also, I was asked to file an issue by @ejona86.

What version of gRPC and what language are you using?

I am using Java. The desired API is not present in all versions, as far as I am aware.

What operating system (Linux, Windows, …) and version?

Android (Linux), targeting Android 6.

Yes, I would also like to switch to a newer version.

What runtime / compiler are you using (e.g. python version or version of gcc)

Android Studio, with current versions of the toolchain.

What did you expect to see?

An efficient, boxing-free, method of adding a large array of primitives to a protobuf.

What did you see instead?

http://cdn3.meme.am/cache/instances/folder113/500x/63531113/x-x-everywhere-boxing-boxing-everywhere.jpg

[xfxyjwf]

There is already an unboxed accessor addBuffer(float value), but I guess the problem is that we are storing repeated fields in List<Float> internally and that requires boxing regardless how the API looks like.

[oridb]

Yes, I'm already using that API.

As you mentioned, it still boxes values. And it's just a bit clunky compared to just adding everything in one go. It's not a deal breaker, but improving performance on large primitive buffers would be nice. And having a simple API for it would be a pretty good bonus.

[noguespi]

I have the same issue.

We are returning a message of "repeated int32". Protobuf isn't using primitive and the boxing of each element generate a lot of gc. It starts to have a noticable impact when we reach a big amount of request per second.

Do we have a hotfix/alternative ?

[bolerio]

+1

[paolieri]

+1

[rbroggi]

Hi all,
I have the same problem and end up turning down the adoption due to this issue. I've made all the developments to migrate from an RMI style communication to a GRPC and the memory usage of my service jumped from ~20Gb to ~60Gb. doubles[] are much more concise than List. I'm a big +1 here...
I am a big fan of protobuf hopefully we can achieve this requirement.

[smovva]

+1

[trams]

I see you did a big work on integrating collection of unboxed primitives to generated protobuf messages. That is great! But Builder are still using boxed types. So I am +1 for this

[hitsmaxft]

we meet performance issue while receiving large response from big data services, such as ranking service, output large amount of scores (storage as List ) with PB protocol.

Boxed primitives with collection cause high memory and cpu overhead.