Apm: Feature: OpenTelemetry exporter

[axw]

Is your feature request related to a problem? Please describe.

OpenTracing and OpenCensus are being merged into a new project, OpenTelemetry. The OpenTracing and OpenCensus libraries are expected to be sunset in November 2019.

Describe the solution you'd like

Implement an Elastic APM exporter for OpenTelemetry Collector. The OpenTelemetry libraries would export trace and metric data to the service, which would have the exporter plugged in. The exporter would translate from the OpenTelemetry format to Elastic APM events.

Contributions will target the contrib repository.

Describe alternatives you've considered

We could also implement the OpenTelemetry API for each of our supported languages, as the APIs are meant to be extensible.

Implementing the API for each language would be a lot of effort. That's not to say that it won't or shouldn't be done, but it probably makes sense to have an exporter in place first. Later, language-specific API implementations could enable us to "mix-and-match" OpenTelemetry spans with transactions and spans created by our own APIs, as we do with OpenTracing in at least some of the agents today: https://www.elastic.co/guide/en/apm/agent/java/current/opentracing-bridge.html#operation-modes.

[tsloughter]

It isn't official (yet) but because I really hope vendors will support the OpenTelemetry SDK (default implementation) and not simply implement the API I wanted to mention this discussion on what it will mean to "support OpenTelemetry" https://github.com/open-telemetry/opentelemetry-specification/issues/185

[cyrille-leclerc]

As discussed with @axw , I'll use this ticket to capture additional information.

According to specification/trace/semantic_conventions/span-general.md, all  network related operation should/could specify net.peer.name and net.peer.ip (here). We could leverage this for all the connections between components visualized in the service maps (database call, http call, rpc call, message bus...).

⚠️ The spec for HTTP spans indicates that net.peer.name can be omitted as the required attributes can be either

• http.url
• http.scheme, http.host, http.target
• http.scheme, net.peer.name, net.peer.port, http.target
• http.scheme, net.peer.ip, net.peer.port, http.target

Here are some examples:

MySQL

https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/trace/semantic_conventions/database.md#mysql

| Key | Value |
| :---------------------- | :----------------------------------------------------------- |
| Span name | "SELECT * FROM orders WHERE order_id = ?" |
| db.system | "mysql" |
| db.connection_string | "Server=shopdb.example.com;Database=ShopDb;Uid=billing_user;TableCache=true;UseCompression=True;MinimumPoolSize=10;MaximumPoolSize=50;" |
| db.user | "billing_user" |
| net.peer.name | "shopdb.example.com" |
| net.peer.ip | "192.0.2.12" |
| net.peer.port | 3306 |
| net.transport | "IP.TCP" |
| db.name | "ShopDb" |
| db.statement | "SELECT * FROM orders WHERE order_id = 'o4711'" |
| db.operation | not set |

Kafka

https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/trace/semantic_conventions/messaging.md#examples

Topic with multiple consumers

Given is a process P, that publishes a message to a topic T on messaging system MS, and two processes CA and CB, which both receive the message and process it.

Process P:  | Span Prod1 |
--
Process CA:              | Span CA1 |
--
Process CB:                 | Span CB1 |

| Field or Attribute | Span Prod1 | Span CA1 | Span CB1 |
|-|-|-|-|
| Span name | "T send" | "T process" | "T process" |
| Parent | | Span Prod1 | Span Prod1 |
| Links | | | |
| SpanKind | PRODUCER | CONSUMER | CONSUMER |
| Status | Ok | Ok | Ok |
| net.peer.name | "ms" | "ms" | "ms" |
| net.peer.port | 1234 | 1234 | 1234 |
| messaging.system | "kafka" | "kafka" | "kafka" |
| messaging.destination | "T" | "T" | "T" |
| messaging.destination_kind | "topic" | "topic" | "topic" |
| messaging.operation | | "process" | "process" |
| messaging.message_id | "a1" | "a1"| "a1" |

[axw]

https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/671 has been merged, OpenTelemetry data should now show up on the service map in the next opentelemetry-collector-contrib release. The PR only adds support for HTTP requests and database queries, and not messaging. That's because the Go Agent does not currently have support for messaging, and the exporter uses the Go Agent. We can add messaging in later - it's not supported across all our agents anyway.

[cyrille-leclerc]

@axw Do we have to update the changelog of the OpenTelemetry Collector Contrib project?
I'm not sure the maintainer will do the update (see https://github.com/open-telemetry/opentelemetry-collector-contrib/commit/bb120e74673bf111dea34121d990e648f74f8ee0#diff-4ac32a78649ca5bdd8e0ba38b7006a1e )

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/master/CHANGELOG.md

[axw]

@cyrille-leclerc that is part of the release process: https://github.com/open-telemetry/opentelemetry-collector/blob/master/docs/release.md#releasing-opentelemetry-core

[cyrille-leclerc]

Thanks @axw