Opentelemetry-specification: Span's event specification

[tonyredondo]

According the specification and proto file an event can have attributes as a key:value pairs, where the keys must be strings and the values may be string, booleans or numeric type.

I'm worried about the values types, the Opentracing specification leave this value types open as an "object", allowing the creation of events like error which have an object attribute error.object with the actual exception instance of the language as seen in the semantic conventions document.

Also we can use this object type to instrument loggers supporting structured events.

So i have two questions. How can I handle this kind of usage with the OpenTelemetry api? How the OpenTracing shim is going to work on those cases?

[SergeyKanzhelev]

loggers have not been designed yet. The idea behind Event is lightweight time mark in the span. Default limit on number of events is very aggressive to use it for logs.

With the assumption that logging API will be built separately and Events are used for simple time marks - do you still have a scenario when objects needs to be supported?

[austinlparker]

I think the way the shim would handle things like this would be turning span logs into OTel logs and creating a reference between the span and the log.

[tonyredondo]

@SergeyKanzhelev The common use case is the error event as the OpenTracing semantic conventions allows. In our case we are sending the exception object in the error.object field. We are planning to send other events with custom objects linked to a span.

If this can be handle by the logging api, I would be very interested to read about that api, even participate on those discussions.
Is this api will be available in the OTel first release? Any roadmap on that?

@austinlparker When you say creating a reference between the span and the log, are you planning that the logs are not going to be inside the span but rather in another collection?

[austinlparker]

I think it'd have to be... maybe @carlosalberto has some thoughts, since I know he's been working on the shim for Java, so he's probably closer to dealing with this in practice than I am right now.

[bogdandrutu]

@tonyredondo can you tell me how the backend that you use consumes these Objects?

[yurishkuro]

I second @tonyredondo concern. In OpenTracing the tracer decides how to marshal attributes passed as objects, usually by recognizing certain common types. Limiting the API to primitive values only means that the user code must do the marshaling, which is very inefficient if only <1% of spans are sampled.

Here's an example from Jaeger where an attribute of a span.log (aka event) is an object

or a stack trace

[bogdandrutu]

@yurishkuro nice example.

The current OpenTelemetry API allows to add lazy formatted events via the addEvent(Event) API. So marshaling happens only if the Span is serialized, but indeed needs to produce primitives as the end result.

If one type is important to support I think we should add support for it in our data model and allow users to explicitly set on critical path or delayed that type.

Also there is a discussion to support Exception/Stacktraces via the Status API.

[tonyredondo]

@bogdandrutu we have a custom tracer implementation that splits the span and logs into two collections, serialize it with msgpack and send the payload to our backend, where we store the log data inside a postgresql json column type.

The common types are object exceptions and structured data from loggers (so we don't know the schema). We are planning to add others events type.

Here is an example for structured data:

Here for exceptions:

[jmacd]

Added this issue https://github.com/open-telemetry/opentelemetry-specification/issues/76 about supporting structured values, which would IMO be a better way to represent resources.

[carlosalberto]

@carlosalberto has some thoughts, since I know he's been working on the shim for Java, so he's probably closer to dealing with this in practice than I am right now.

So currently the OT Shim tries to convert to one of the base types (numerical, boolean), and fallbacks to String, which clearly is not enough, based on this issue.

I'm wondering if adding Object as a supported type in the API side is (for AttributeValue) would be enough...

[austinlparker]

I'm bumping this as another note of support for either changing the spec to accept object or something like it - as an example for what it looks like, we recently merged a PR to opentelemetry-dotnet that refactored the interface for attribute values into accepting a struct KeyValuePair<string, object> -- https://github.com/open-telemetry/opentelemetry-dotnet/blob/2183c425e80abcebd77cb2db6e1f1763d498c554/src/OpenTelemetry/Trace/Span.cs#L164

The advantage of this (in .NET at least) is that type information is preserved so we're able to use more idomatic methods of converting Attributes to whatever format we need for the exporter (see https://github.com/open-telemetry/opentelemetry-dotnet/blob/8aae2c02f1760668734d99e848ca780e7635b578/src/OpenTelemetry.Exporter.Ocagent/Implementation/SpanDataExtensions.cs#L108 for an example).

Obviously this pattern isn't going to work in every language, but supporting object would be very useful overall.

[lmolkova]

Bumping this again: Events seem like a workaround in absence of logging API. Assuming we have logging API (or there is a standard logging API for lang), I'd say all events and errors are logs with severity level and attributes.

Even without OTel, any vendor can capture logs and augment them with an implicit span context today (feel sorry for Java).

So, assuming this, and assuming OTel will provide logging API (or integration with widely adopted logging APIs) perhaps after v1, will we regret having Events?

[jmacd]

I believe the OTel position is that we would like to add a logging API but must finish the work in progress first. I don't think we'll necessarily regret having a span.AddEvent API, but users will thank us when they can write "Log" again, instead of "AddEvent". I see the span event as an ordinary log event with the addition of the span context, so span.AddEvent is equivalent to Log assuming there is a current span in the context.

There's been a lengthy discussion in the Go repository, which has an experimental "streaming" SDK which turns every API call into an event which could be logged (https://github.com/open-telemetry/opentelemetry-go/blob/master/experimental/streaming/exporter/exporter.go). It would be very easy to extend this with a logging event, it would be identical to the span event but the span context would be optional.

[Oberon00]

Note that there is also the opposite opinion by @qudongfang, namely that "Event is one of the major verticals as well as Metrics, Tracing and Logging in observability from my perspective" (#176). I feel this issue sort of duplicates #176 but looks at it from a different perspective. At least any resolution to either issue will affect the other.

[svzhl]

There is another issue with Events as logs except the one with data types specified in the subject. Events should be written to logs in a natural order. Current Go tracer implementation doesn't allow to access events as they appear. So events can be printed only when the span is finished. Exporting the following trace

Span 1
-- Event A at t1
--> Span 2
    -- Event B at t2
-- Event C at t3

to custom stdout exporter could look like

t2: Event B (comes first as Span 2 finished earlier)
t1: Event A
t3: Event C

While OT logging support is incubating I would suggest to consider an experimental feature that would allow to workaround the above issue. An additional method onEvent(Span, Event) in SpanProcessor interface similar to OnStart(Span) and OnEnd(Span) would make it possible to implement a custom processor and stream events linked to a span as logs. Looking at Go implementation it can be naturally fitted. Otherwise a whole custom tracer needs to be implemented

[bogdandrutu]

Closing this because:

1. Based on a previous comment from @tonyredondo "The common use case is the error event as the OpenTracing semantic conventions allows" we have a "RecordException" for that.
2. Attributes were extended to support Arrays, and probably will support nested attributes as well.