Kibana: [APM] Add message about dropped spans

[sqren]

The new DT timeline does not yet have the message about dropped spans, that was available pre 6.5

https://github.com/elastic/kibana/blob/7266349e33ffa608d4199da81f991c2d9e023839/x-pack/plugins/apm/public/components/app/TransactionDetails/Transaction/Spans/view.js#L131-L143

Closes: https://github.com/elastic/kibana/issues/22348

[elasticmachine]

Pinging @elastic/apm-ui

[formgeist]

There's an updated design for the new trace timeline.

• Show EuiIcon iInCircle in the Transaction row span
• Show tooltip when hovering the icon to display the following message:

This transaction is missing spans. Expected "x" but only showing "y".

[jasonrhodes]

@zubeio/apm-server (or possibly @sqren or @formgeist ?) -- can someone explain in this ticket (or point to a link where it's explained) what we should expect with this part of the Transaction (elasticsearch data) interface:

span_count?: {
  dropped?: {
    total?: number;
  };
};

A couple questions:

• Why are these optional?
• Can the span_count key be present but its value be an empty hash {}?
• Looks like we used to use this value, and default to 0 if any part of this value didn't exist, when deciding whether to notify a user about dropped spans ... is that still right?
  
  
  
  • https://github.com/elastic/kibana/blob/6.4/x-pack/plugins/apm/public/components/app/TransactionDetails/Transaction/view.js#L217-L219
  
  

[simitt]

@jasonrhodes the values are optional, as they were introduced in a minor version update and we cannot break the API by adding required attributes. There _should not_ be an entry with an empty hash in ES. It seems valid to not notify users about missing spans when this value is missing, as it basically only means we don't have any information about it. It would be wrong though to assume there are no missing spans just because the information is not there.

[jasonrhodes]

@simitt that all sounds good to me!

[jasonrhodes]

@formgeist this mockup shows a message:

This transaction is missing spans. Expected x, but only showing y.

The information we have is "number of dropped spans". If that number is, for example, 82031 ... what should this tooltip say? (That's the real number for the first one I found in a dev console query)

Would it be ok for now to say:

This transaction may be missing up to 82,031 spans.

??

[sqren]

Sorry for repeating if this is already clear.

"Missing spans" and "dropped spans" are two different things, and (might) need two different messages. The screenshot is about "missing spans".

Transaction documents may have the following:

{
  "span_count": {
    "started": 1,
    "dropped": {
      "total": 0
    }
  }
}

Missing spans
span_count.started is the number of child spans the transaction has. We can compare this to the actual number of spans we retrieved. The diff will be the missing spans. The spans might eventually show up (or they might have been lost forever. who knows)

Dropped spans
span_count.dropped.total are the number of spans the agent chose to drop. By default the Node agent will only capture the first 500 spans, and drop any subsequent spans. (Docs)

Would it be ok for now to say:
This transaction may be missing up to 82,031 spans.

The text we used to have for dropped spans was:
"{droppedSpansCount} spans dropped due to limit of {spans.data.spans.length}." And then a link to agent specific documentation:
https://github.com/elastic/kibana/blob/8efd35291256345a10fc8664a85f43278ec5d507/x-pack/plugins/apm/public/utils/documentation/agents.ts#L10-L13

[jasonrhodes]

Sorry for repeating if this is already clear.

@sqren thanks, it's actually not clear at all. :) The title of this ticket refers to "dropped spans" and so my questions are all about dropped spans. Should the title of the ticket actually be "missing spans"? Which one is the regression?

[sqren]

Dropped spans is the regression. We had it pre 6.5:
https://github.com/sqren/kibana/blob/1daa265ff82b58e54e658b8f5834aa5fe1701a9e/x-pack/plugins/apm/public/components/app/TransactionDetails/Transaction/Spans/view.js#L116-L121

"missing span" is new and I can't remember what we agreed there. There seems to be some overlap between the two concepts.

[jasonrhodes]

@sqren gotcha! @formgeist can you shine any light on what you think would make sense, design/layout-wise, for the dropped spans regression part? We should obviously wait on the new missing spans stuff.

[jasonrhodes]

The more I think about it, the more I think the warning icon is wrong here (i.e. we save it for the missing spans) -- can we just note "dropped span count" in the flyout?

[formgeist]

@jasonrhodes @sqren Been trying to find any previous issues that talk about dropped spans, and I'm pretty sure the only thing we've been going over has been the missing spans and whether to show spans without a parent transaction. Sorry to get this mixed up, but I too think we just need to display a message in the transaction flyout, rather than clutter the Timeline with unnecessary warnings. The message will only be displayed if we're cutting the spans short (due to config in the agent) afaik. I think that's the most elegant solution for now to not have to deal with a regression.

I lifted the original message out and it features a link to the agent documentation as well. Right now it's apparent that it's only Python and Node that mention dropped spans, so unsure if it's because the other agents have either forgotten to add the docs or it's because the problem can't occur since there's no special config.

[roncohen]

Søren laid out the two scenarios nicely here https://github.com/elastic/kibana/issues/24320#issuecomment-435184327, but just as a heads up, using this message wouldn't be entirely accurate:

{droppedSpansCount} spans dropped due to limit of {spans.data.spans.length}.

because we can't necessarily know the configuration setting in the agent by simply counting the spans we received. Spans could be dropped for other reasons than the config limit. For example in cases where Elasticsearch is overloaded. I suggest we simply say "Agent dropped {dropped} spans for this transaction due to a configured limit: "

We now have the started property which we could compare to spans.data.spans.length to see if we have _lost_ any spans accidentally. This should be a separate message. Something like: There are {started-spans.data.spans.length} spans missing for this transaction: <link-to-docs>. This is not a regression, so don't rush to get it in for 6.5

Looking into the future: If the timeline contains any transaction with dropped or missing spans, we could show a message to that effect to highlight it already on the timeline.

[felixbarny]

For the Java agent, dropped spans are only indirectly mentioned in the transaction_max_spans configuration option: https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-transaction-max-spans

Although it doesn't explicitly use the term dropped spans.

[felixbarny]

There are {started-spans.data.spans.length} spans missing for this transaction: <link-to-docs>.

As started should be interpreted as x or more (spans could be started after the transaction has already been reported), the message should be either There are at least {started-spans.data.spans.length} spans missing for this transaction: <link-to-docs>., or just There are spans missing for this transaction: <link-to-docs>.

[jasonrhodes]

Maybe Something happened <link-to-google>

[formgeist]

Where did we end up on the implementation here? Let's keep _missing_ spans out of the equation for now, as we're just dealing with the regression, which has to do with the dropped span message. Next release we can make it better and have shared way to display a message to the end user.

[jasonrhodes]

@formgeist thoughts on this layout/wording? Happy to tweak/adjust.

[formgeist]

@jasonrhodes I'd prefer a small EuiCallOut (info) because I think most users will quickly skim the page, and not notice the information we're providing.

As for the message, I would propose the following changes;

The APM agent that reported this transaction dropped more than 10 spans based on its configuration. Learn more.

The "at least" reads too dramatic to me, but maybe because I'm not a native speaker 🙂

[jasonrhodes]

@formgeist cool, I can update to EuiCallOut. "more than" isn't accurate because it might be exactly 10. Options are probably "at least 10 spans" or "10 or more spans".

[formgeist]

Then I prefer "10 or more spans"