Uwazi: Check and improve CSV Export after Library Pagination

[RafaPolit]

[simonfossom]

@fnocetti here's a new icon for CSV export and import.
Could you please replace the current one? Thanks!

icons.zip

[fnocetti]

After checking, we need no changes after the Library Pagination implementation.
Changes will be needed when the pagination allows to fetch documents above ES limit.
For now, the scope of this issue can be defined by:

• [ ] change the icon for the one @simonfossom provided
• [ ] implement proper integration testing to be prepared for when fetching above the limit happens

@RafaPolit Please let me know if you disagree and/or if this affects the priority of this issue.

[fnocetti]

I agreed con Rafa to test a little bit further and re check the scope of this issue.

[RafaPolit]

I love the spanglish version of "I agreed con Rafa"! :) Yes, please, report about what would be required to do batches of 10.000 for export of large collections.

[fnocetti]

After checking, if you try to access a set of documents with a query like the api will tell you that the results window is too large (>10000) and suggests you to use some of the ES API's meant for that purpose.
I'll create a pull request with the new icon and the new integration test when they're ready.
Maybe we can discuss in the tech call how to proceed with improving the search window?
On the while, I will continue to investigate how the search_after and scroll API's could be used within uwazi to expand that window.

[fnocetti]

After some research I'm listing my findings here:

• There are two alternatives to fetch over 10k records:
  
  
  
  • ES Scroll API: it is a stateful API that allows you to fetch contiguous pages of search results by keeping (inside of ES's own implementation) the search context in memory. The fact that it needs to keep the context alive makes it not ideal for real-time user facing navigation, but a very efficient strategy for batch non-real-time operations (like csv export). This was the strategy originally proposed by @txau and I agree that is the best option for the CSV export. We should be in a good position to add support for it in the search API.
  
  
  • ES Search After API: It's a stateless implementation of Scroll. It achieves the same results without needing to keep the context in memory but can be more complex to implement because it needs to use a tuple of order-defining values (based on the indexable properties) as a "from" parameter instead of simply an ordinal number. In the other hand, it makes it a better approach for real-time user navigation features.
  
  
• After reading some experiences from others, we should't need to implement deep pagination for the UI because users do not usually scroll more than a few pages in the search results. They'd rather look at a few pages and continue refining their search criteria. I'd love to hear @simonfossom expert position on this topic. May be we could validate this with some UX research?
• We need to do some refactoring on the CSV Export class and routes to be able to implement this strategy. Assuming the Scroll API (which seems the way to go, and it's more straightforward to implement):
  
  
  
  • The CSV Exporter class assumes a single page containing the whole set of results to be exported. We need to refactor the algorithm to use pagination.
  
  
  • While implementing pagination, the headers pre-computation strategy may stop to work correctly, so we will need to refactor it. We might need a more greedy strategy that can even be more efficient and lead to better results.
  
  
  • This could lead to a need to do a bit more manipulation of the CSV temporal file, but should not be a problem.
  
  
• Also, after discussing with Mila, this might not be urgent.

[RafaPolit]

To be revisited when there is more urgency.

For the time being, the workaround is to create filters that produce smaller batches (probably already implemented in every collection) and do partial exports.