Kibana: [APM] Timeline: Indicate that spans are missing within transactions

[formgeist]

Following a discussion in apm-server we want to indicate in the UI that spans are missing from the transaction.

[formgeist]

Proposed design implementation where we might show a callout indicating that the current trace sample is missing spans. Span count not matching the shown spans in the Timeline. We probably have to have some documentation that might instruct users what might be happening and options to improve their instrumentation (if that's possible). Otherwise, we might as well remove the documentation link.

cc @elastic/apm-ui

[formgeist]

@simitt @felixbarny Thoughts on implementing this into the design of the Timeline?

[felixbarny]

I like it.

Notes on the implementation: the expected and actual number of spans has to be calculated via ES queries. I'm using pseudo SQL here to describe the logic of the queries.

• expected number: SELECT sum(span_count.total) FROM transaction WHERE transaction.trace_id = $traceId
• total number: SELECT count(*) FROM span WHERE span.trace_id = $traceId

The reason is that in the trace details view, we don't load all spans, but only load them on demand, based on which ones are currently expanded.

[formgeist]

@felixbarny That's a great help. Minor question, but do you think it's possible we can supply the user with some more background and possible suggestions to their instrumentation to make things better, or is that completely down to individual implementation and frameworks etc.?

[formgeist]

@felixbarny just to further clarify, perhaps adding a docs page either per agent or apm-server that helps the user out a bit.

[felixbarny]

There is usually nothing the agent can do to fix that situation. I think the most frequent source of error would be a full APM server queue, which could indicate an ES capacity problem. But it could also be the result of a validation error -> bug in the agent.

Long story short, I think it makes sense to link to a page in the APM server docs, as I don't see that it makes sense to link to an agent-specific documentation.

[sqren]

The reason is that in the trace details view, we don't load all spans, but only load them on demand, based on which ones are currently expanded.
@felixbarny

In the current tranaction details view, all spans are loaded up-front. Are you saying this will be an issue with DT, and only the spans/transactions within the viewport should be loaded?
I'm not necessarily talking about loading stack traces for all spans (which is 95% of a span), perhaps just metadata.

If my approach scales it will be possible to calculate missing spans on the client, which is very straightforward.

[felixbarny]

I don't have the issue at hand, but I thought we already discussed this somewhere.

Especially in the context of DT, the transactions/spans should be loaded lazily. The performance of the UI should still be acceptable when rendering 10k+ spans.

[sqren]

Especially in the context of DT, the transactions/spans should be loaded lazily. The performance of the UI should still be acceptable when rendering 10k+ spans.

Okay, I must have missed that. I agree that we can't draw 10k+ spans without giving the user a bad experience. This will add some complexity both to query spans in a paginated fashion, and how it's lazily loaded as the user scrolls up or down.

I can imagine it also opens up some UX issues. How does a user even navigate and find what they need in waterfall of 10k+ items?

[felixbarny]

We should probably discuss this in a separate issue.

I've added my thoughts about that here: https://github.com/elastic/kibana/issues/20660

[felixbarny]

In a large trace, it might be quite common that there is a span missing here and there. Another approach would, therefore, be to indicate for a specific transaction that there are spans missing for it. Similar to the design proposal for https://github.com/elastic/kibana/issues/22347.

[formgeist]

@felixbarny Would it be safe to just do both - show a warning at the top of the trace to inform the user, and indicate for each specific transaction that it has missing spans?

[felixbarny]

It would be safe, but might lead to some clutter and would alert users even if nothing really bad happens. If the user doesn't even navigate to the transaction which is missing spans, they would not care anyway.

[formgeist]

Perhaps incorporating the per transaction warning is better than a callout at the top of the trace as it was proposed in https://github.com/elastic/kibana/issues/22347#issuecomment-417302170. I'll update the design.

[formgeist]

Updated design proposal, moving the warning to transaction level instead of a trace header warning. Makes it more relevant for the individual transaction as suggested previously by @felixbarny

[formgeist]

Moving into implementation https://github.com/elastic/kibana/issues/24320