Kibana: [Logs UI] Design: Pin field as column

[makwarth]

I wanted to get this feature going. It's currently targeted for 7.1.

This feature allows users to pin other fields than timestamp and message in the logging overview page.

Below are two UX proposals. Let me know what you think is best. I'll create final implementation issue once we land on a UX decision.

Option A

Expand the existing Customize window. Add a section in the Customize popover. Users will be able to search and use auto-suggestion to pick a field to pin as column. (Similar UX to the upcoming "Group by any field" feature)

To order/remove field columns, the user can click a remove icon and/or drag columns up and down (to order them left to right)

Option B

Just like in Discover. On the upcoming log event fly-out, users can click an icon next to any field name to pin that field as column.

To order/remove field columns, the user will have to use arrows and/or "x"-icon to on the pinned columns.

[elasticmachine]

Pinging @elastic/infrastructure-ui

[weltenwort]

This sounds like it has a significant overlap with the plans to make message formatting configurable. IMHO we should discuss this in a broader context, taking the technical reality of the log event ingestion into account.

[makwarth]

@weltenwort sounds good, I'll shoot you a cal invite.

[makwarth]

Spoke with @weltenwort. We decided to collaborate on writing up the roadmap description for the log formatting feature and move the discussion there. I'll close this for now.

[makwarth]

I'm reopening this issue after various discussions. @weltenwort and I discussed whether this feature should be powered by the "message formatting" feature or not. I think it needs to be a standalone feature.

Reasons:

• a user might be consuming multiple log types (modules) at the same time, so this feature needs to be independent of message formatting
• users can quickly modify columns based on ad-hoc use case (sometimes you need a column for field x, sometimes you need a column for field y) without modifying a message formatting configuration. (I think it makes sense that a user can local store his/her column configuration per source configuration)
• supports custom structured logs as is
• no engineering integration with message formatting required

[weltenwort]

a user might be consuming multiple log types (modules) at the same time, so this feature needs to be independent of message formatting

But isn't the heterogeneous nature of log entries exactly the reason why the formatting must depend on the message type?

users can quickly modify columns based on ad-hoc use case (sometimes you need a column for field x, sometimes you need a column for field y) without modifying a message formatting configuration.

But what is the message formatting configuration if not the definition of fields to display? How easily that configuration can be modified in the user interface and where that modification is persisted are independent considerations.

no engineering integration with message formatting required

Sorry, I don't understand that statement. Who will integrate it then?

Please indulge my attempt to extract the feature requests from our discussions and the above statements. According to these requests...

1. the fields of a log message should be displayed in the log stream as vertically aligned columns of text.
2. there should be a way to insert a message field that refers to a field in the document into the formatting instruction across multiple different log entry types with just one user action.
3. there should be an option to persist the formatting configurations to multiple backends (such as in-memory, local storage or server-side saved objects).

[makwarth]

But isn't the heterogeneous nature of log entries exactly the reason why the formatting must depend on the message type?

My point was that having something like column: <true/false> as part of the log type message formatting won't be useful when consuming various log types at the same time.

But what is the message formatting configuration if not the definition of fields to display?

It's a configuration of fields and order to display in the "message". My comment was about fields to pin as columns. Any field that is part of the message formatting can also be pinned as optional column

no engineering integration with message formatting required

I meant that pin as column doesn't need to be part of the message formatting specification, which doesn't add to the message formatting feature specification, and should keep both features simpler

[weltenwort]

I see. It concerns me that we are about to create parallel structures that have very similar purposes with all the downsides that such duplication brings for both code maintainability and UI intuitiveness.

Why not take the opportunity to plan this out properly and avoid taking on new tech debt? I feel we have yet to arrive on the same page as far as the intended usage and future features are concerned. How about working that out first and then deciding on the proper technical implementation?

[makwarth]

It concerns me that we are about to create parallel structures that have very similar purposes with all the downsides that such duplication brings for both code maintainability and UI intuitiveness.

Can you please elaborate on this?

To me, message formatting and pin field as column doesn't overlap. I'm suggesting an independent feature that allows users to pin any field as a column that works independently of any message formatting or any custom structured log events.

[weltenwort]

I see two levels to this:

1. On the UI level we provide two ways to configure the content of a log line that would both lead to visually very similar effects but with different persistence semantics and different configuration workflows. What would be the guidance we give to the user for choosing one configuration mechanism over the other? I can already imagine the confused questions by users about why the host name is duplicated in the message and the "pinned column" or why a certain formatting is applied to a part of the message but not the "pinned column".

2. Technically, the mechanism would still have to integrate with the formatting because the Kibana server needs to make sure that the deduplicated list of source fields it fetches includes both the ones used in the formatting instruction as well as the ones required for "pinned columns". And as indicated above we should definitely share the formatting functionality for well-known fields (e.g. dates, ips, geo coords) in order to avoid inconsistencies and duplicate maintenance of formatting instructions.

That's why I would advocate thinking through the plans both formatting mechanisms in tandem. What I could see is to have an additional "common fields" configuration that is easily accessible via a UI like the one you suggested in option A above, but still scoped to the source and can be persisted in the saved object. That way they can be quickly adjusted without clobbering the formatting of the type-specific message formatting instructions and without introducing more confusion about the persistence. These common fields would default to the timestamp but allow for more fields to be prepended. The configurability in terms of formatting (color, font, etc...) would be shared with the per-rule field formatting - hopefully even down to the reusable UI controls.

[makwarth]

Taking a step back to share my thoughts on feature prioritization:

To me, the goal of the Logging UI is to help our users consume any logs as easily as possible. This means zero configuration (hopefully not even ES index pattern sometime) to get _something_ useful. For Filebeat users, the Logging UI should provide a great experience out of the box.

I see custom user message formatting as a useful feature, but I prioritize the generic, zero-configuration features over it. In an effort to make the Logging UI useful for everyone as fast as possible, I'd therefore prioritize as follows:

• Source config UI
• Search highlighting
• Supporting custom structured log events
• Ad-hoc pin field as column
• Generic message coloring
• Message formatting for Filebeat modules
• Custom message formatting

The fact that some Filebeat modules completely parse the log message and thereby strip out the message is unfortunate, imo. If this wasn't the case, message formatting rules like the ones we've implemented per module wouldn't be so critical.

Replying to your first point inline:

On the UI level we provide two ways to configure the content of a log line that would both lead to visually very similar effects but with different persistence semantics and different configuration workflows. What would be the guidance we give to the user for choosing one configuration mechanism over the other?

Ideally, no user would need to configure anything to consume logs in a decent way. Ad-hoc functionality, like pin field as column, isn't a configuration. It's just a way of consuming logs. E.g. if your logs have an msg field instead of a message field, remove the message column and add the msg column. Or, if a certain tag is relevant for what you're debugging, add that tag as a column - and remove it again when you're done.

I can already imagine the confused questions by users about why the host name is duplicated in the message and the "pinned column" or why a certain formatting is applied to a part of the message but not the "pinned column".

The host name would only be duplicated (added as column), if the user chooses to add is a column.
As for different formatting, which formatting rules are you referring to exactly?

[weltenwort]

Thank you for that overview.

What is the difference between "Supporting custom structured log events" and "Custom message formatting"? To me, the former depends on the latter. How else would the "custom structured log events" be formatted into a useful string?

Ad-hoc functionality, like pin field as column, isn't a configuration

I would disagree with that. It is configuration, that is easily accessible in the UI and that might or might not be persisted. A good analogy might be window sizes in a stacking window manager. The user can often adjust the window size "ad-hoc", but some applications store that and restore it when the window is next opened. To me that sounds like "configuration".

I absolutely see the value in making the relevant settings easily accessible for a given workflow. All I wanted to express is that the different persistence semantics should be obvious to the user and should not lead to unnecessary tech debt.

The fact that some Filebeat modules completely parse the log message and thereby strip out the message is unfortunate, imo.

It is the reality we have to deal with. Even if we improved the situation in filebeat, we still need to support older version that don't have these improvements. And the "Supporting custom structured log events" implies we need configurable message reconstruction anyway.

Ideally, no user would need to configure anything to consume logs in a decent way.

That is exactly the goal that I had in mind when I designed the formatting rules system. We should try to supply "decent" formats for all filebeat modules. But experience and feedback shows that our user's use-case are very diverse, so providing the ability for the users to define which formats are "decent" for them makes sense.

Again, I'm not arguing against making a "pin column" functionality available in the UI. I'm just arguing in favor of a clean implementation of the various code paths that impact which fields are fetched from Elasticsearch and how they are formatted.

As for different formatting, which formatting rules are you referring to exactly?

There currently is only the "convert to a string" format, which is implicitly used for all fields. But I can imagine that we would need to implement better formatting for specific field types like IP, geo coord, etc. I'm merely trying to think ahead and avoid duplicate implementations.

[makwarth]

What is the difference between "Supporting custom structured log events" and "Custom message formatting"? To me, the former depends on the latter. How else would the "custom structured log events" be formatted into a useful string?

No, there's no dependency for https://github.com/elastic/kibana/issues/29756. Later, we could make it configurable to order custom structured fields in the "message", but we don't _need_ that functionality to ship this feature.

There currently is only the "convert to a string" format, which is implicitly used for all fields. But I can imagine that we would need to implement better formatting for specific field types like IP, geo coord, etc. I'm merely trying to think ahead and avoid duplicate implementations.

I agree we should think ahead, but I think our disagreement is about the dependency on custom message formatting for all these features. I'd like to ship the above listed features _before_ we offer any custom message formatting and therefore avoid coming to a stand still before we have a master plan figured out (that plan always changes anyway).

As a way of progressing, with FF out of the way shortly, could you write up a one-pager on the current message formatting functionality, and how you see it evolve over the next 6 months? I think that will give us more specifics to discuss / prioritize / work into other features.

[weltenwort]

After thinking this through some more I think I have pivoted my point of view enough to appreciate this feature quite a bit (except for the "pinned columns" name, so I'll call it "common fields" in my head :wink:). I now understand that we want to enhance the logs view to have more "structured logging" capabilities while I previously always looked at it from the traditional "line of text" perspective.

I still wonder what persistence we want to support for the common fields. I see some value in having persisted common fields in the source configuration, maybe even some sensible default including host.name and event.dataset.

[makwarth]

++

I see some value in having persisted common fields in the source configuration

I agree. We could have timestamp, host.name, event.dataset and message be the default columns? If the message field doesn't exists, we show the log formatted message, like today. If there's no log formatted message (non Filebeat logs), we do this https://github.com/elastic/kibana/issues/29756.

One important feature is the ability to quickly and easily alter the columns. For example, remove host.name and add log.level for ad-hoc analysis. Ad-hoc altering of the columns shouldn't require of the user to edit the source configuration, imo. It's too clumsy and the user likely doesn't want the ad-hoc changes to persist anyway.

What do you think of Option A above? I guess it would work so that any customization would override the source configuration for the duration of the browser session?

[weltenwort]

Yes, option A seems a lot more discoverable and accessible to me.

So we would add a configuration option to specify the default list of fields for a source, which can be changed via the configuration ui. In addition, the toolbar provides a quick way to configure transient overrides to this list.

[makwarth]

@weltenwort Sounds good.

Regarding the message column. It's the only field that doesn't just show the field value: It'll first try to show the value, but if it doesn't exists, it'll look for a log message rule, and if there's no rule, it'll default to something like https://github.com/elastic/kibana/issues/29756. Do you see any problems with this other than UX? We probably need some UX indicator of how the "message value" was generated.

[weltenwort]

I agree in general. As I commented on #29756 I'm still unsure whether just fetching all of source is advisable, but I can't really think of a better way.

[makwarth]

Cool. I think we have enough to create an implementation issue for this feature. I'll go ahead and do that shortly.

[makwarth]

Closed for implementation issue: https://github.com/elastic/kibana/issues/31463