Django-import-export: A way to create objects in bulk?

[Rubyj]

@bmihelac @andrewgy8 @yozlet

Is there a way to create objects in bulk during an import? Maybe overriding a method?

The default way of calling save for every row is slow when you have a big dataset to import. However, it is extremely slow when that data set also has foreign keys.

There was never an answer to #588

[omertarikkoc]

Does anybody have a solution?

[honi]

This is a very quick hack I tried and seemed to work. Not sure the implications in other features like validation, database integrity errores and stuff like that. It should work as expected but test and use at your own risk. I didn't use this in the end that's why I didn't continue testing.

class BulkSaveMixin:
    bulk_instances = []
    bulk_m2m_fields = []

    def save_instance(self, instance, using_transactions=True, dry_run=False):
        self.before_save_instance(instance, using_transactions, dry_run)
        if not using_transactions and dry_run:
            # we don't have transactions and we want to do a dry_run
            pass
        else:
            self.bulk_instances.append(instance)
        self.after_save_instance(instance, using_transactions, dry_run)

    def save_m2m(self, obj, data, using_transactions, dry_run):
        if not using_transactions and dry_run:
            # we don't have transactions and we want to do a dry_run
            pass
        else:
            for field in self.get_import_fields():
                if not isinstance(field.widget, ManyToManyWidget):
                    continue
                self.bulk_m2m_fields.append((field, obj, data, True))

    def after_import(self, dataset, result, using_transactions, dry_run, **kwargs):
        self._meta.model.objects.bulk_create(self.bulk_instances)
        for m2m_field in self.bulk_m2m_fields:
            self.import_field(*m2m_field)

[andrewgy8]

AFAIK, there is not a way currently to create objects in bulk. The current implementation creates objects one by one.

A quick scan of the solution @honi posted looks like it may work, but since I have not tried it myself, I can not say for sure.

If someone tries the solution above, and it does work as expected, please let us know. And if you feel up for it, write some tests and submit a PR!

[matthewhegarty]

I have achieved some decent performance improvements for bulk creates of new rows. Some notes here

See also #1030, #730, #588