October: Add chunking abilities to ImportExportBehavior

[chrisvidal]

Expected behavior

to export 100 000 records from a model with 2 relations

Actual behavior

Out of Memory or Timeout

Reproduce steps

setup a model with 100 000 records and 2 relations

October build

431

[munxar]

if you need that much data, use the eloquent chunk function to query your model. you can't load that much into memory.
consequently you should stream the csv or append to a local file and response that if done.
the standard ex- importer can't handle this situation. you'll have to implement the controller logic by yourself.

probably we could make the ex, importer behavior handle this?

P.S. And no, increasing php heap is only a dirty hack...

[chrisvidal]

Thanks for your answer @munxar

So basically I would have to rewrite a whole new export behavior, right?

[LukeTowers]

Chunking import and exports is something that I would like to add by default to October, there was a plugin that implemented that behaviour pretty nicely a while ago but I can't remember what it was called exactly. For now you'd have to implement it yourself @chrisvidal but it is something that would be nice to have in core.

[chrisvidal]

oh I would love to find this plugin back then hehe to avoid wasting time.
I'll try to find it. Do you remember by any chance what was doing this plugin? Was it published in the officials October plugins listing?
Chunking by default in OC import/export would be nice indeed. +1

[LukeTowers]

@chrisvidal No, I don't think it was in the marketplace, I think it was just a random Github Repository. The author may have taken it down to theoretically polish it up for a more public release but I can't remember.

[chrisvidal]

Oh I see, Hmmm, I found this one https://github.com/canthis/oc-chunk-import.

[LukeTowers]

@chrisvidal that's the one I was thinking of!

[MajorasJack]

Is there still any possibility of adding this to the core ImportExport?

I've been utilising laravel's collection chunk but still seems to be causing out of memory/timeouts when the data is being returned to the Export model.

Would be a fantastic addition 👍

[LukeTowers]

@MajorasJack @chrisvidal @canthis @thehiggster I'd be open to reviewing and merging a PR with this functionality, I like @canthis's approach in their plugin, perhaps something similar to that.

[Samuell1]

Maybe we can solved this with this https://medium.com/@barryvdh/streaming-large-csv-files-with-laravel-chunked-queries-4158e484a5a2 and Streaming can solve another issue with CSV exports #4232

[github-actions[bot]]

This issue will be closed and archived in 3 days, as there has been no activity in the last 30 days. If this issue is still relevant or you would like to see action on it, please respond and we will get the ball rolling.

[Samuell1]

This can be solved with LazyCollections that were added in Laravel 6.0
https://laravel.com/docs/master/collections#lazy-collections