Apm: Embedding logs into the APM UI

[katrin-freihofner]

Summary of the problem (If there are multiple problems or use cases, prioritize them)
Currently, we have many different actions menus. None of these gives any preview of what to expect or where this link leads to. The link list is already pretty long and therefore, a bit hard to grasp.

Example

Ideal solution (optional)
We would like to preview some information of the connected logs (same trace ID) in the APM UI. This could be the number of log lines, a few log lines... I think the result could even be multiple versions (small, with little information - big, showing log lines), depending on the screen size. From there we can link to the logs UI.

List known (technical) restrictions and requirements
This can not replace the actions menu.

There is an issue to make the action menu more consistent https://github.com/elastic/logs/issues/5, which is WIP.

[weltenwort]

We would like to show-off a few log lines [...]

Putting on my advocatus diaboli hat, would the limited benefit of showing a few log lines outweigh the cost in terms of screen estate? Have we considered not showing the log lines themselves but potentially a more condensed summary ("2342 log lines associated with this trace" or "2342 log lines from 3 hosts / 5 containers / 2 datasets")?

[katrin-freihofner]

@weltenwort good point, I should have mentioned that in the description - will update it.

[felixbarny]

Just my 2c, I think it would be cool to have a tab in the trace view for Logs, next to the Timeline and Metadata tab. Ideally, that tab would somehow embed the Logs app.

[formgeist]

I started doing some overall sketches on how it might look if we added the logs timeline to a separate tab view (as suggested ^) alongside the Timeline and Metadata in the Trace sample panel.

There are some layout/sizing considerations we need to make because we're positioning it within the same panel as the rest of the Trace content which is not full scale as the Logs app is currently. I've proposed to use inline scrolling both vertically and horizontally, but that's nothing something that is currently supported in the app.

Also not entirely sure how much functionality we want to carry over from the Logs app, if this is going to be considered a "quick" preview/embed. Do we need to be able to highlight terms? Do view customizations made in the Logs app persist onto the previews?

Happy to hear your thoughts on this 🙂

[axw]

Nice! If we're in the context of a transaction group, I think it might make sense to either filter to, or highlight, the rows with the relevant transaction.id from the trace sample. Otherwise in a complex distributed trace, the logs will probably be jumbled up.

[formgeist]

@axw Thanks for the feedback. I agree, we're missing some context around the trace and its transactions. I've given it a go and added the transaction ID in the log stream example;

Example 1 – Show transaction ID in stream

Adding the transaction ID, at least in the existing log stream, does return the stream a little intermixed, and all the IDs might be not provide the appropriate context that users are looking for.

Example 2 – Sort by the transaction ID and add service name

Based on example 1, I sorted the log stream by transaction ID and added the service name to give context to the trace itself. Since this is based off of an Opbeans example, not sure how trustworthy the data is for this particular example, but alas it's what I've put together.

Was this along the lines of what you were thinking, @axw?

[axw]

@formgeist I only meant to highlight the rows matching the transaction.id from the active trace sample, like this:

Not sure if showing the transaction.id column is useful to users. Service name probably would be. Just my 2c anyway.

[formgeist]

Right, I see what you're getting at. I've made an example of highlighting the service name (remove the transaction id's) and only highlight the selected service by the selected transaction group's id within the selected trace.

Perhaps the transaction ID could be revealed in a tooltip on the service name, so we don't need to dedicate a full column for it.

[katrin-freihofner]

@formgeist will these log messages be clickable? I'm wondering if we should add some interaction to click a specific log message and go to the logs UI from there? Let me know what's your opinion on that.

[formgeist]

@katrin-freihofner To be honest, I have kind of side-stepped the whole interaction right now, because it's not clear what we expect our users to do with it. I agree, initially it makes sense to open the log message in the Logs UI. Alternatively, we could provide the same inspect functionality where the log message details can be shown in a flyout, rather than redirecting users. We need to think carefully about what people expect here.

[axw]

@formgeist that looks good to me. My only concern is that it might not be obvious to users why those lines are highlighted. In the same vein as Katrin's question, I wonder if the service name should be clickable to change the highlight. Maybe it needs some user testing? :o)

[formgeist]

@axw Thanks for the feedback. We discussed this yesterday as well, that it's not obvious and possibly quite busy to look at from the get-go. Yeah, I think some interaction on the column cells would perhaps be a good idea as well. I'll continue sketching, but I also think we just want to start with the bare minimum and then go on from there.

[mikker]

UX thing to be aware of with tooltips like the one above is that the text is sometimes hard to interact with, eg. select and copy the transaction id. For something like this I suspect that's a very common action.

[formgeist]

@mikker Definitely! I think we would typically opt for a popover in those cases instead whenever we have links or selectable text. From a user's perspective, from this place in the trace, would you rather be able to just click the service name and be taken to the service and its corresponding transaction? We're thinking of other interactions for the log message, so you'd have to click the service name. Removing the tooltip, but possibly displaying some label on hover that explains where it will take them. Or do you think the transaction ID is important enough to always be present next to the service name and the log message? Like this;

[mikker]

I guess a direct link is better as the users wouldn't necessarily know where to bring the id. If then the plain string id is copyable on the destination page, I'm good 👍

The more I think about it, this feature will be amazing.

[mikker]

The link could be "abbreviated" to an icon, perhaps with an explanatory tooltip. This is if we want to save on the horizontal space.

[formgeist]

Yeah that might be worth exploring

[formgeist]

Design update – Jan 10, 2020

Been working on some more visual updates and creating more of the user flow in the prototype to examplify the view in context options for diving into the Logs UI from the logs widget in APM, providing users options to explore logs either around that specific log message or view the log details.

The service name legends are added to the header, since we've added the service references per log message. We might explore more ways of filtering within the logs enabling the user to get even more focused on a particular service's logs within a trace, but for now we're OK with the user having the reference.

Figma prototype link

[mikker]

This is superpowers. Exciting!

[piotrplazienski]

two cents: We have used Jaeger and migrated to Elastic APM. The feature we miss the most is being able to click span in trace view and get all logs associated with it (and only with this span).
see https://www.hawkular.org/img/blog/2017/2017-06-26-trace.png for example how this looks like.

It would be awesome if we could do the same in Elastic APM.
@formgeist do you think it would be possible to extend your approach to transaction and span details? So when we click span colored bar and we have a details popup, add logs tab there which would show logs for this particular transaction/span? Or show all logs, with clicked span highlighted in some way? This may be a bit more confusing, but provides context.

[formgeist]

@piotrplazienski Thanks for the feedback (and apologies for the late reply)...

do you think it would be possible to extend your approach to transaction and span details? So when we click span colored bar and we have a details popup, add logs tab there which would show logs for this particular transaction/span? Or show all logs, with clicked span highlighted in some way? This may be a bit more confusing, but provides context.

We will investigate later in the implementation where to place the related trace log views. Appreciate the suggestion.

[formgeist]

I'm closing this design issue as I've open an implementation issue which should have the necessary design and experience specs to continue when we prioritize https://github.com/elastic/kibana/issues/67611