Apm: Add support for exporting OpenTelemetry Metrics

[axw]

Is your feature request related to a problem? Please describe.

We recently introduced an Elastic exporter for OpenTelemetry Collector. This exporter currently only supports exporting traces. Users would also like to export metrics.

Describe the solution you'd like

Extend the Elastic exporter to also export metrics to Elastic APM Server. We can start with simple scalar metrics, and add support for distribution metrics once https://github.com/elastic/apm-server/issues/3195 is done.

Describe alternatives you've considered

Use the Prometheus exporter with Metricbeat.

Additional context

https://github.com/open-telemetry/opentelemetry-collector/pull/1720 was recently merged, which means we can now proceed to build metrics support without working on top of legacy code.

[axw]

@elastic/apm-server @cyrille-leclerc FYI

[axw]

@sorantis FYI also. It would be great to ensure these metrics work well with the Metrics UI. (In addition to metrics coming through APM Server from other sources.)

[sorantis]

We're currently working on the Inventory Schema that will formalize the fields and metrics across different providers (aws, azure, on-prem, etc.). The main goal with this initiative is to shift to resource centric views and ensure that anything that complies with the schema is visualized in Metrics UI.

Is this something that can be used for the OT metrics?

[sorantis]

cc: @kaiyan-sheng @exekias

[axw]

@sorantis OpenTelemetry defines "semantic conventions", which describe resources with which metrics, traces, and logs are associated: https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/resource/semantic_conventions/README.md

I won't bore you with the details, but these will end up being recorded as ECS fields in Elasticsearch in the metric docs.

OpenTelemetry does not cover specific metrics AFAIK, so we probably wouldn't end up overlapping metric names.

[exekias]

I see most of the fields for host, container, cloud, etc already map to our ECS names. Are you planning to convert the fields that don't match (ie renaming cloud.zone to cloud.availability_zone)?

[axw]

@exekias yes, the OpenTelemetry conventions will be translated to ECS. We haven't done that for the cloud.* fields yet, but we will sooner or later.

[axw]

PR adding initial support for gauge and sum (counter) metrics: https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/1173

[axw]

Update: initial metrics support has landed in opentelemetry-collector-contrib. When we have support for histograms in the server/protocol, we'll update the exporter.

[d-baranowski]

@axw I stumbled upon https://pkg.go.dev/github.com/elastic/opentelemetry-exporter but I noticed it is archived. Is there a different opentelemtry exporter I can use?

[axw]

@d-baranowski the exporter lives at https://github.com/open-telemetry/opentelemetry-collector-contrib.
You can find more details at https://www.elastic.co/guide/en/apm/get-started/current/open-telemetry-elastic.html

The repo you found was from a very early effort which we abandoned. I'll go ahead and delete that to avoid further confusion.

[axw]

Closing this as we've updated the exporter with metrics support. I've opened https://github.com/elastic/apm/issues/377 to track support for histogram metrics.