Kibana: Better export for data tables in Reporting

[elasticmachine]

Original comment by @tbragin:

Right now only the preview of the data table shows up in the PDF report, because we essentially take a screenshot of the first page of the table. We should consider how to improve this.

Possibilities to discuss:
- Option to export data tables (not saved search tables) to CSV (to accompany the PDF)
- Option to export a full data table as a formatted PDF table (limited to N results, IBM Qradar does max of 500)

This is similar to the problem we have with saved search export LINK REDACTED

[elasticmachine]

Original comment by @tbragin:

cc: @kobelb

[elasticmachine]

Original comment by @tbragin:

We discussed this live and decided on the following approach in the immediate future:

• Export a full data table as a formatted PDF table, limited to N results.
• 500 seems like a reasonable default for N.
• For now, do not offer the option of making N configurable.
• Put these data tables at the end of the PDF, after the charts.

Note, this is what Splunk does as well

Reason for not going with the other option:

• Adding an option to export CSV along with PDF adds the complication of dealing with multiple files (we can zip them up, of course)

[mbarretta]

also would include saved searches in this request: a saved search added to a dashboard will be truncated, even if the number of rows within the search is a relatively low number, like 20.

[stacey-gammon]

We discussed this a bit in the Sharing team planning week. Here is some of the notes from that session:

Saved Search and Data Tables: treat them the same.

• Problems:
  
  
  
  • Panel is cut off so page 1 might not show all the rows on the first page.
  
  
  • Table might have 10 pages, but only first page. Even if panel shows all rows of the first page, users want to have all 10 pages of data.
  
  
• Possible Solutions
  
  
  
  • Show just page 1 table in the pdf, but bundle data table information as .csv
  
  
  • Show just page 1 table in the pdf but add tables at the bottom of the pdf, maybe as extra pages.
  
  
  • [NO] auto expand rows so all rows show in a screenshot.
  
  
  • We don’t want this, it could be way too much information
  
  
  • [Best option] Offer an option in report PDF:
  
  
  • [Yes/No]: Include zip of csvs, one for each table. Tablename.csv
  
  
  • Add dashboard level option, export raw data as csv. rawDataForVis1.csv
  
  

[tsullivan]

@mbarretta

also would include saved searches in this request: a saved search added to a dashboard will be truncated, even if the number of rows within the search is a relatively low number, like 20.

For saved searches, there is a CSV export option in the "Share" menu when viewing the search in Discover.

There's a gap right now on being able to export CSV from a saved search when viewing it in the Dashboard context though, and we need to fix it.

Providing more convenient access to exporting CSV, for saved search and data table, is a general theme where we could see a lot of wins.

1. Have a POST "Generation URL" to allow exporting data table (aggregated) visualization data to CSV
2. Be able to export CSV from a saved search when viewing it in the Dashboard context

[mbarretta]

@tsullivan I appreciate that the CSV export allows saved search records to be exported, but not exporting everything seen in a dashboard breaks the "promise".

I've seen many dashboards that are essentially text-only: the preference is for raw data over graphics so that they can see everything in one place and avoid the extra click required to go from a visualization to the underlying data. In those cases, the dashboard is really more like a few Discover views glued together.

So yes, users could export the various data, import them into something like Excel, and then export to PDF from there, but I think it's a fair expectation for them to assume if there are 20 rows of records on a dashboard, they will be visible when that dashboard is exported.

[Eniqmatic]

+1 is there any more progress on this or when we will likely see it? We have a dashboard that is data tables with split tables based on how many security attacks have occured. But our PDF reports only show 1 page. Is there any temporary workarounds for this?

[cmagdalena-racowireless]

+1

[tsullivan]

@Eniqmatic The App team is actively working on doing a lot of heavy lifting around how the visualizations get their data, and the fruit of that effort will be that we can get data from a saved visualization on the server-side. That will lead into options on how to implement both of the items from the description:

• Option to export data tables (not saved search tables) to CSV (to accompany the PDF)
• Option to export a full data table as a formatted PDF table

There is a workaround to export CSV from an aggregated table visualization. It's from the UI, so unfortunately it would be hard to automate: you can click the Inspector for the visualization to see the underlying data. On that screen are export controls. See: https://www.elastic.co/guide/en/kibana/6.6/vis-inspector.html

[Eniqmatic]

@tsullivan Excellent sounds like what I am looking for. Hopefully the guys make good progress with that! Thanks for your response.

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[mikeh688]

Enterprise users of Elastic need to be able to share pdf reports. The current limitation of only generating a pdf image for the first page of tabular data massively undermines the value of Elastic and KIbana in the eyes of these customers. It's fantastic that Kibana will do all the really funky new stuff (Lens etc) but, ultimately, if it cant deliver the basics, enterprise users wont adopt it because it doesnt meet their needs (mainstream adopters vs early adopters).

[fbaligand]

I fully agree with @mikeh688
Then, I have several users that ask me to have alerting with a Kibana table visualization as CSV attachment.
And today, I can't address this need, that is very useful to be able to open data in Excel, and even, simply copy the text of the data table.
Formatted data is necessary for me. Particularly to use number format locale. Microsoft Excel expects number format locale from current country. This is important for decimal numbers to us the good decimal separator. Same thing for dates.

[reighnman]

Same as above comments..

I ran into this issue with table data on a dashboard. I then tried just a pdf report at the table visualization, same issue.

I then saved the data table filters as a saved search to try a standalone csv report, but cannot do things like GROUP BY/aggregation/etc at the discovery layer that are necessary for this data.

Overall reporting still feels like a third-party addon or external cog, having to setup things like email configuration directly in the config. I would hope things are going in the same direction as alerting is going, with use of "connectors" managed internally and available to use with scheduled reports as well (in addition to fixing the issues with PDF generation on data tables).