Linkerd2: Emit Proxy Spans

[grampelberg]

What problem are you trying to solve?

When viewing distributed traces, it is beneficial to see the spans associated with proxies alongside those emitted by applications. This allows users to understand what overhead proxies are adding to their applications.

As this feature has serious performance implications and requires application changes to be useful, it must be configurable and disabled by default.

How should the problem be solved?

• Implement the Trace-Context propagation format.
• Follow the OpenTelemetry specification for clients.
• Integrate with OpenTelemetry Service.
• Implement the OpenCensus Agent Protocol for exporting spans to the local agent.
• Receive sampling configuration from the agent via TraceConfig
• Configure emitting spans via inject.

Out of scope

• Installing and configuring the OpenTelemetry service.
• Exporting any metadata outside of timing.

[grampelberg]

Related to #1436.

[adleong]

It's worth noting that OpenCensus does not use the Trace-Context propagation format for gRPC requests. Instead, it uses a header called grpc-trace-bin: https://github.com/census-instrumentation/opencensus-specs/blob/master/trace/gRPC.md

This means that Linkerd must also read this header if it wishes to participate in gRPC traces.

[adleong]

If installing and configuring the OpenTelemetry service is out of scope, it becomes more difficult to send traces to the agent over TLS. The agent's trust root would need to be mounted into the proxy's container and thus the configuration gets a bit more complex.

[olix0r]

If installing and configuring the OpenTelemetry service is out of scope, it becomes more difficult to send traces to the agent over TLS. The agent's trust root would need to be mounted into the proxy's container and thus the configuration gets a bit more complex.

it probably means we aught to have the proxy support the environment variables needed to read a trust root from disk; and leave it as an exercise to the user to get that mounted in the proxy's container...

[olix0r]

Given a trace like the above, how should this change when the proxy is participating in tracing? @grampelberg

[grampelberg]

Implemented.