Kibana: [ML] DF Analytics results: object fields missing in table

[alvarezmelissa87]

Kibana latest master (7.8)

Currently the newJobCapsService (which uses the field caps api) is used in the results views to fetch the document fields that make up the result table columns. This service is specialized for the job wizards and omits fields with object type.

The api returns the fields in an object in an array of objects like:
products: [ { price: 124, … }, { price: 444, …} ]
as
products.price with a type referring to the type in the object ( so float or integer in this case)

The results table doesn’t recognize it as an array of objects so it tries to follow the path of products.price and gets undefined - this results in empty columns in the table.

The existing field-caps endpoint is tailored towards what is needed for the wizards.
The conclusion for a solution after getting thoughts from @walterra is to use a separate call to the plain ES field caps API. The call to field caps for the table can be much simpler - we just need a list of available fields and their type (along with no filter for 'allowed types' to ensure we get object types as well) as the table code itself does the rest based on that information.

This will allow object types to be shown - though we will still have the problem of blank columns when a path to an array of objects is provided as a field (e.g. products.price).

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[walterra]

FYI: Transforms now use Kibana index patterns to get the fields for the source index preview.

Moving an item here from the previous EuiDataGrid meta issue #64418:

The advantage of a raw fields caps approach would be that users wouldn't need to create Kibana index patterns and the UI would work out of the box for jobs created using APIs. However, a drawback is that to view the source or destination indices in Discover they would still need to create index patterns. At the moment we also require an index pattern for the source index to show up in the source selection dropdown. To also support saved searches in the upcoming wizard (similar to the workflow for anomaly detection and transform) an index pattern is also required.

Taking all of this into account and a possible future update of Kibana index patterns which wouldn't require their creation beforehand, we should make it a requirement to have index patterns for both source and destination indices. Also, results pages using EuiDataGrid should require a destination Kibana index pattern to allow searching for ml.* data in the query bar. If an index pattern is not present, we can help users by opening a modal and offer an option to create an index pattern on the fly for the user. See https://github.com/elastic/kibana/issues/51288#issuecomment-604448443

I suggest before we move on with any coding we should probably do another catch-up/discussion so we're all on the same page!

[walterra]

The Kibana team describes the approach using the object field and showing the full data within that in a single cell the structured approach. Showing a column for each nested field under object is called the destructured approach.

This issue is about both the structured approach missing (no column for object type fields) and the destructured approach being broken (empty cells for nested fields).

The destructured approach was fixed recently in #80302.

There is an ongoing discussion, also as part of Discover moving to EuiDataGrid, which approach is favourable as a default across Kibana. Taking that into account and since we fixed the destructured apporach I'm going to close this is issue. We'll watch the ongoing discussions and will revisit the structured approach if it turns out to be the favourable one.