Apm: [Discussion] Agree on binary format for the trace context headers

[eyalkoren]

Is your feature request related to a problem? Please describe.
The lack of a binary format is already a known limitation with regard to our OpenTracing compatibility. However, as we look at tracing Kafka clients, this becomes a requirement. Distributed tracing through Kafka will likely rely on record headers, which require a binary value.

Describe the solution you'd like
The W3c's _Distributed Tracing Working Group_ already started working on a specification draft for the binary format of the Trace Context headers. We can rely on its current state (copy and use as internal reference to make sure all agents use the same state), assuming that it will end up quite similar.

Describe alternatives you've considered
An alternative is using the simplest approach for now and keep an eye on the W3C specification until we feel it is stable enough. For example- a straightforward UTF-8 encoding of the textual representation of the headers. This means 55 bytes length for the traceparent header, instead of 29 with the binary spec.

@elastic/apm-agent-devs please leave some comments with your thoughts so we can come up with a concrete proposal.

[felixbarny]

My suggestion would be to do it similar to how we did it with the HTTP header. That is following the specification draft but don't write it to the traceparent field but the elastic-apm-traceparent field. If the dashes are problematic, maybe use a different header name like elasticapmtraceparent.

While UTF-8 can work as well adhering to the current spec draft will likely make it easier in the future to switch to the official format once it's GA.

[eyalkoren]

Closing this as we are agreed on what to do.
Opened #199 with the proposed spec.