Django-import-export: Large files take 5+ hours to run

[aidanlister]

We've got a whopper file we're trying to import and it runs for 5 hours before dying with lack of memory.

Can you think of a way we could build an iterative approach where we run the process in a worker, reporting back to a queue server in realtime?

e.g. upload the file somewhere, process it line by line showing import errors as we go, (e.g. make the internals use a generator).

[bmihelac]

I would like to it to be possible to work on data without loading whole set in memory. That would make it much more usable for big datasets. Unfortunately currently this option does not exists.

There is unmerged management command for importing that could be of help for you:

https://gist.github.com/bmihelac/434fceb6ba8e752f08d3

[int-ua]

Updating to Python 3.5 can improve the speed:
https://docs.python.org/3/whatsnew/3.5.html

OrderedDict is now implemented in C, which makes it 4 to 100 times faster.

[bashu]

+1

[int-ua]

Also using https://github.com/anx-ckreuzberger/django-request-cache correctly may help.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.