Kibana: [SIEM] [Map] Support pattern matching when validating which Kibana Index Patterns to create layers for

[spong]

In effort to limit the number of layers created on the map to those specific to the SIEM App, a layer is only created for values in siem:defaultIndex that have a Kibana Index Pattern that is an exact string match.

While this ensures the map is created with only relevant layers (ensuring both performance and data relevancy), it has caused some confusion as to how and when data shows up on the map. We've improved this by adding targeting documentation, updated inline messaging when no matching index is found, and further details in the maps documentation.

I think one of the last remaining things we can do to streamline this UX is to enable pattern matching when determining which Kibana Index Patterns are relevant. This would mean that you could configure apm-*-transaction* in siem:defaultIndex, and it would match on the apm-* Kibana Index Pattern -- something we're explicitly addressing in https://github.com/elastic/kibana/issues/52297.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[P1llus]

Just adding another idea to the UX part, though don't know if it would fit your current plans.

If you want to make it more obvious to the user, which data is being added to the map, why not have a similar multi-choice dropdown like you have in Discover, in which you can choose which pattern(s) you want to use when searching/displaying?

This would also allow someone to switch between maps dynamically without much hassle, and creating their own groups of information by choosing which indices to include.

That would also make it support displaying indicies they might not want to include in the siem:defaultIndex for some reason, because while the data might not be security related, it could be beneficial to the map itself.

Ref:

[spong]

Thanks for your feedback @P1llus! 🙂

As you mention, allowing the user to select exactly what indices they want to visualize and which fields (client.geo, destination.geo, host.geo, observer.geo, server.geo, source.geo) to create layers for would provide the best UX in terms of visualizing what they'd like on the map.

Really like the idea of being able to quickly switch between these configurations as well!

In terms of progress towards that, as part of https://github.com/elastic/kibana/pull/54876 I've added support for wildcard matching of siemDefaultIndex->KibanaIndexPattern (which will improve the default scenario), and also support for creating map configurations for non-source/destination fields (e.g. client/server for APM datasets). The latter was done in such a way that creating a configuration UI for the map should now be a relatively low level of effort task once prioritized.

I'm going to close this issue as https://github.com/elastic/kibana/pull/54876 resolves what was originally reported, but this can be used as reference once we start expanding functionality around the Network Map. Thanks again for the feedback!

cc @andrew-goldstein