Apm: Protocol Buffers encoding for metadata and events

[axw]

Is your feature request related to a problem? Please describe.

We would like to start sending profiling data from agents to the server (https://github.com/elastic/apm/issues/121). Along with the profiling data, we need to be able to send the standard metadata preamble so we can add metadata to the resulting Elasticsearch documents.

Describe the solution you'd like

For Go at least, the most natural way to send profiling data to the server will be in its native pprof format, in compressed protocol buffers (aka protobuf). Since we would need to decode protocol buffers data on the server anyway, it makes sense to also encode the metadata in protocol buffers.

On top of supporting the immediate use case of transmitting profiles as binary data over plain old HTTP, this would also:

• build the foundation for encoding and transmitting all events (transactions, spans, errors, metrics) in a more efficient binary format
• move us in a direction that would enable us to use gRPC (see Gil's PoC)

Agents will use protoc (protocol compiler) to generate code from a schema.

Schema

We will need to define a protocol buffers schema in the proto3 language. This would live alongside the JSON-Schema definition. Down the line we might be able to generate JSON-Schema from proto3, e.g. using https://github.com/chrusty/protoc-gen-jsonschema or something we build.

Fields are typed, so that's part of the validation job done. We're still left with higher-level validations that are specific to our model, such as enforcing maximum string lengths, required fields, etc. One option would be to use https://github.com/envoyproxy/protoc-gen-validate. This would require us to duplicate our validation logic in JSON-Schema and protobuf. Also, protoc-gen-validate is labelled alpha, and unstable.

A couple of other options for validation are:

• duplicate the JSON-Schema validation logic in Go code
• pass the protobuf-generated structures back through the JSON-Schema validation code

The latter is probably preferable, to avoid human errors. It would incur some overhead, but as long as it's just for metadata it's not going to be a big deal. We can improve things later if/when we expand the schema to support events.

An example proto3 file is below, including metadata message types, and an "Event" message type which is a sort of discriminated union. Requests from the agent to the server would start the body with a Metadata message, followed by a stream of Event messages.

(Example) elastic-apm.proto

syntax = "proto3";

package elastic.apm;

// Metadata holds metadata that is applied to all events in a stream.
message Metadata { 
        System system = 1;
        Process process = 2;
        Service service = 3;
        repeated Label label = 4;
}

message System { 
        message Kubernetes { 
                message Node { 
                        string name = 1;
                }
                message Pod { 
                        string name = 1;
                        string uid = 2;
                }
                string namespace = 1;
                Node node = 2;
                Pod pod = 3;
        }
        message Container { 
                string id = 1; 
        }
        string architecture = 1;
        string hostname = 2;
        string platform = 3;
        Container container = 4;
        Kubernetes kubernetes = 5;
}

message Process { 
        int64 pid = 1;
        int64 ppid = 2;
        string title = 3;
        repeated string argv = 4;
}

message Service { 
        message Agent { 
                string name = 1;
                string version = 2;
        }
        message Framework {
                string name = 1;
                string version = 2;
        }
        message Language {
                string name = 1;
                string version = 2;
        }
        message Runtime {
                string name = 1;
                string version = 2;
        }
        string name = 1;
        string version = 2;
        string environment = 3;
        Agent agent = 4;
        Framework framework = 5;
        Language language = 6;
        Runtime runtime = 7;
}

message Label {
        string name = 1;

        // Exactly one of the following fields should be specified.
        // We avoid oneOf due to inconsistencies between encoders.
        string string = 2;
        int64 int64 = 3;
        bool bool = 4;
}

message Event {
        Profile profile = 1;
}

message Profile {
        // Data holds a pprof profile, encoded as protobuf, and gzip-compressed.
        bytes Data = 1;
}

Describe alternatives you've considered

1. Encode profiles as JSON. The memory and transfer overhead would be significant. I don't think this is a realistic alternative.

2. Use an alternative binary encoding for the framing and metadata, such as CBOR, BSON, MessagePack, Thrift, Avro, ...

CBOR, BSON, and MessagePack are all schema-less, which means to encode objects their field names must be encoded as strings. The overhead (compared to protobuf, Thrift, and Avro, which use knowledge of the schema to elide the field names) is likely excessive due to the highly structured and well-defined nature of our data.

Protocol buffers, Thrift, and Avro all have language clients for all of the agents we support, and then some more. They generate similarly compressed data, but Avro does not use field IDs and so a schema must be negotiated separately between client/server to ensure upgradeability. See https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html for more details.

[axw]

TL;DR: let's define a protobuf encoding. We can start with the metadata and message framing to support profiling (https://github.com/elastic/apm/issues/121), and later down the line we can extend it to cover transactions, spans, errors, and metrics.

@elastic/apm-server @elastic/apm-agent-devs WDYT?

[roncohen]

This will mean APM Server will need to be able to receive both Protocol Buffer and JSON. Initially, JSON for all the existing data types and protobuf for the new types related to profiling + metadata

I think we need to have a conversation what the desired end state is: JSON, Protobuf, support both for everything or a mix depending on the data type. I don't think supporting both ingestion formats for everything forever is attractive right now. It means double the API design work for each new feature and double the encoding/decoding infrastructure amongst other things. There's also the case of RUM where protobuf might be problematic.

I don't see a pressing need to move to Protocol Buffers generally at the moment, but happy to have the discussion. Also, I don't think the fact that protobuf is convenient for the Go agent counts for much in itself without investigating what the situation looks like in the other agents that we expect will support profiling.

[SergeyKleyman]

Regarding

higher-level validations that are specific to our model, such as enforcing maximum string lengths, required fields, etc.

With JSON the constrains are part of the API spec which is platform independent so each agent can implement a automatic verification to ensure its JSON encoding is compliant. How will we do it if we switch to Protobuf only (no JSON schema) approach?

[Qard]

protobuf and grpc aren't very popular in Node.js. There's quite a bit of overhead to doing all that encoding in JavaScript.

[axw]

Thanks all for the feedback.

@roncohen

This will mean APM Server will need to be able to receive both Protocol Buffer and JSON. Initially, JSON for all the existing data types and protobuf for the new types related to profiling + metadata

Right. Do you see a viable alternative for the immediate use case (profiling data)? Or are you thinking we should defer the feature until we're ready to drop JSON?

Breaking the following down a bit...

I think we need to have a conversation what the desired end state is: JSON, Protobuf, support both for everything or a mix depending on the data type. I don't think supporting both ingestion formats for everything forever is attractive right now.

I thought we could defer this discussion to when we came to expanding to the remaining types, but seeing as we'd need to decode metadata in two formats I suppose you're right. My thinking was that we need _some_ kind of binary encoding for profile data, so let's define it in a way that we can _maybe_ expand on later.

If you'd asked me a year ago, I probably would have said just ditch JSON in the long term. However I expect that the ease and ~universal availability of JSON encoders has helped with the several community-developed agents that have popped up. So now I'm not so sure.

Generally I think we should optimise for the official agents, and if it's not _too_ much work to support JSON to enable more esoteric languages then do that. RUM bundle size could be an issue as you say, which might force our hand.

Say we do want to support both JSON and protocol buffers: proto3 has a canonical JSON mapping. The one thing I'm unsure of is how to deal with things like "metadata.labels", where values may have different types.

It means double the API design work for each new feature and double the encoding/decoding infrastructure amongst other things.

If we can rely on the JSON mapping, it may not that bad. I'll do a bit more investigation into this.

There's also the case of RUM where protobuf might be problematic.

According to the docs, the size of protobufjs/minimal is ~6.5kb gzipped. The agent is currently ~16KiB gzipped, so it would be a significant increase. More on the speed at the bottom.

I don't see a pressing need to move to Protocol Buffers generally at the moment, but happy to have the discussion.

Yeah, not generally for sure. I hope it didn't sound like I was suggesting that we necessarily do this; I just wanted to go in a direction that would make it possible, after on further investigation.

Also, I don't think the fact that protobuf is convenient for the Go agent counts for much in itself without investigating what the situation looks like in the other agents that we expect will support profiling.

Did you mean to write "pprof" rather than "protobuf" (understandable if so as they use a lot of the same letters :)). I don't think protobuf is more convenient for Go than the others. It's true that pprof is, since it's built into the runtime.

My proposal to use protobuf (vs. say Thrift or Avro) was based on the fact that pprof has a well-defined protobuf encoding. In my searches I've not come across an alternative profile format that has a well-defined binary encoding.

There's a parallel effort by Felix to do some investigation into transforming JFR profiles to pprof. If it turns out that it's not viable, then I suppose the alternatives would be: send one of N different formats to the server, or invent a new one. That would mean the benefit of protocol buffers vs. Thrift/Avro is moot, but has no impact on $any_binary_encoding vs. JSON.

@SergeyKleyman

With JSON the constrains are part of the API spec which is platform independent so each agent can implement a automatic verification to ensure its JSON encoding is compliant. How will we do it if we switch to Protobuf only (no JSON schema) approach?

Yeah, that's going to be an issue. As I mentioned in the description we would still get validation of the basic types and structure, but we'd lose things like string length limits. https://github.com/envoyproxy/protoc-gen-validate only supports generating validators for a limited set of languages, so we'd either have to extend it or it's not going to be a general solution.

We might have to change our approach, and rely on integration tests for validation, i.e. by the APM Server, or some stripped down version of it.

@Qard

protobuf and grpc aren't very popular in Node.js. There's quite a bit of overhead to doing all that encoding in JavaScript.

Let's put gRPC aside, as that's much more hypothetical. Is there something like protobuf that's more popular? protobuf.js (see below) is "used by" ~160K projects on GitHub, and ~2M downloads from npm per month. Seems popular enough to not be worried about it being abandoned.

Regarding overhead: I did a bit of digging and that is true of the default google-protobuf implementation, but not of the more popular protobuf.js. See their benchmarks for comparison: https://github.com/protobufjs/protobuf.js#performance

They ran on an older version of Node, I just reproduced it on 10.16.0: protobuf.js (using their specific .proto definition) is considerably faster.

It depends a lot on the definition though. I wrote a similar benchmark comparison using the metadata definition in the issue description, and the difference in speed is less pronounced between protobuf.js and json:

benchmarking encoding performance ...

json x 340,524 ops/sec ±1.76% (89 runs sampled)
google-protobuf x 173,983 ops/sec ±2.90% (85 runs sampled)
protobuf.js x 361,717 ops/sec ±3.70% (93 runs sampled)

            protobuf.js was fastest
                   json was 4.1% ops/sec slower (factor 1.0)
        google-protobuf was 51.5% ops/sec slower (factor 2.1)

In this benchmark I'm comparing the speed of encoding the metadata from an example opbeans-node document. It's also worth noting the difference in size: JSON comes in at 544 bytes, and protobuf at 214 bytes.

The story is similar with Go: the default Google implementation is pretty slow compared to the statically-generated JSON encoder for the Go agent, but they're both slower than the alternative gogo/protobuf implementation.

I'll look into the situation with other languages.

EDIT: I was reproducing the benchmark on Python and noticed I made a mistake in my benchmark above for Node.js. I had typo'd "argv" as "args", and so the json encoder was encoding more data than the protobuf. Running again with corrected code it's more favourable to JSON:

benchmarking encoding performance ...

json x 328,448 ops/sec ±3.12% (84 runs sampled)
google-protobuf x 152,768 ops/sec ±3.13% (88 runs sampled)
protobuf.js x 300,395 ops/sec ±4.30% (88 runs sampled)

                   json was fastest
            protobuf.js was 9.6% ops/sec slower (factor 1.1)
        google-protobuf was 53.5% ops/sec slower (factor 2.2)

... and the number of bytes for protobuf is 296.

However, it also occurred to me that this isn't a particularly useful comparison. Metadata would typically be dwarfed by events, so I'll run some more tests with transaction data.

[Qard]

Looks like protobuf support has improved somewhat since I last looked at it. I think the overhead is still not insignificant though.

I kind of wonder how much benefit there really is to considering protobuf encoding for the whole transport though. Ideally we should be minimizing our operational overhead within the app process. I don't see any particularly compelling reason to add this overhead to the entire transport when we could just have another route to upload arbitrary binary. 🤔

[axw]

I think the overhead is still not insignificant though.

@Qard agreed, particularly in light of the new test I just ran. I ran some more benchmarks with 1x metadata and 100x transaction objects.

Without any request/response headers, we're looking at similar performance to JSON (protobuf.js comes out better by 10% in most runs), and the size difference is pretty significant: somewhere around 40KB less for protobuf. Note that this is uncompressed though, the difference is probably not so significant after compression.

However, when adding 6 request and 15 response headers (again using a sample transaction from opbeans-node), the story is much different. We're still looking at a similar size difference, but now protobuf.js comes out 50% slower than JSON.

So it doesn't look great. This is very much a micro benchmark though, so it would be interesting to see how much this matters in relation to the overall process.

I kind of wonder how much benefit there really is to considering protobuf encoding for the whole transport though. Ideally we should be minimizing our operational overhead within the app process. I don't see any particularly compelling reason to add this overhead to the entire transport when we could just have another route to upload arbitrary binary. :thinking:

What did you have in mind when you said "we could just have another route to upload arbitrary binary"? We need a way of sending metadata with profiles, which is the primary goal behind this proposal. Extending to the remainder of the transport is not an immediate goal, it was just something I thought we could consider down the line - but in light of the above, maybe not.

[axw]

In case anyone's wondering what the performance of a JSON-encoded profile looks like:

• uncompressed protobuf: 155,457 bytes
• uncompressed JSON: 1,934,495 bytes
• gzip-compressed protobuf: 51,369 bytes
• gzip-compressed JSON: 81,927 bytes

JSON encodes ~60% slower than protobuf.js. When adding gzip-compression to both, that jumps to ~70%. This is using a CPU profile of apm-server, with 1461 samples and 987 entries in its string table; not huge, not tiny.

[axw]

Say we do want to support both JSON and protocol buffers: proto3 has a canonical JSON mapping. The one thing I'm unsure of is how to deal with things like "metadata.labels", where values may have different types.

It means double the API design work for each new feature and double the encoding/decoding infrastructure amongst other things.

If we can rely on the JSON mapping, it may not that bad. I'll do a bit more investigation into this.

I did a little more investigation. There's a special type, google.protobuf.Value, which can be used to marshal values of different types into a oneOf type structure: https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/Value

So we can define labels and custom context with the type map<string, google.protobuf.Value>, and we'll be able to accept natural JSON map representations like {"k1": "v", "k2": 123, "k3": false}.

Still, even if we wanted to support both JSON and protobuf (which is sounding increasingly likely, if we add protobuf at all), then we'll probably need to revise some of the existing schema (intake v3). e.g. timestamp as an integer, transaction/span IDs as integers, trace ID as integer pair (no 128-bit support in protobuf).

[felixbarny]

For plain encoding, I'm not sure that protobuf would be actually faster than dsl-json, the JSON encoding library we're using the Java agent. That's especially true for transactions/spans, because we'd first have to convert the agent's internal Transaction object into the protoc-generated classes*. That adds overhead and allocations.

The story is a bit different for profiling data though, as we need to convert the JFR data into a different data structure anyway. JFR stores the individual stack traces, whereas pprof aggregates similar stack traces IIUC. The protoc-generated structure may be good enough.

Another question is whether we want to gzip compress the data. Maybe we can get away with not compressing the protobuf which I would suspect saves lots of CPU cycles.

A bit off-topic but I think it makes sense to investigate alternative compression formats which are more light on the CPU. That could significantly reduce the CPU overhead of our agents.

* There might be a way to use the lower-level APIs in protobuf to manually encode the agent's internal data structures directly, without relying on protoc-generated classes.

[Qard]

@axw Didn't have a _specific_ idea in mind, but could we not encode the metadata into query parameters or something like that? Alternatively, and _potentially_ more complex, would be to submit it like a multi-part form, with separate files for the metadata and the actual file. 🤔

[axw]

could we not encode the metadata into query parameters or something like that

Maybe. Some servers and proxies put limits the length of query parameters, so we could run into trouble as the metadata size grows, though I guess we've still got a lot of room.

Alternatively, and potentially more complex, would be to submit it like a multi-part form, with separate files for the metadata and the actual file

This seems like a pretty neat, low-touch approach.

Thanks for the ideas, will investigate a bit more.

[axw]

I'd like to revisit the idea of having dual a protobuf/JSON intake later, as the memory and transfer savings can be pretty significant, and for some languages there will be a speedup. I'll close this issue for now, maybe we can reopen it later.

For the immediate needs of profiling, let's focus on the alternatives that @Qard suggested above.