Ecto: bulk inserts

[jschoch]

please add bulk inserts

if possible i'd like to bulk load relationships as quickly as possible though i'm not sure how this is best done in posgresql

[ericmj]

This requires extending the postgresql driver with support for all steps in extended queries. I opened an issue: https://github.com/ericmj/postgrex/issues/7.

It also requires that ecto use prepared statements for queries. Issue open here: https://github.com/elixir-lang/ecto/issues/180.

[ericmj]

Closing until we have support in postgrex

[jclem]

This thread was a little hard to follow, but it seems this is in Postgrex, now. Is it possible to do bulk inserts in Ecto, yet?

[josevalim]

No, it is not possible. Unless someone tackles this from beginning to finish, we have no plans to include it in 1.0.

[loganhenson]

For those who end up here and want an update:

via @michalmuskala

There's Repo.update_all, ecto 2.0 also has Repo.insert_all

https://hexdocs.pm/ecto/Ecto.Repo.html#c:update_all/3

[edevil]

Repo.insert_all doesn't work with a changeset list. Will this be supported in the future?

[jclem]

Changesets can have associations on them, which doesn't make sense to me in the context of a bulk insert. I guess Ecto could collate the associations somehow and perform multiple bulk inserts, but seems pretty difficult.

[michalmuskala]

It's not planned to add changeset support to Repo.insert_all. Repo.insert_all issues a single database INSERT statement, which is not compatible with many features of the changesets - most notably prepare_changes and associations. Repo.insert_all is designed (like all *_all functions) as a low level operation with very predictable and clear translation to the database operations.

[edevil]

FTR, this is what I'm doing now:

    multi =
      changesets
      |> Enum.reduce(Multi.new, fn(cset, multi) -> Multi.insert(multi, Ecto.UUID.generate, cset) end)

    {:ok, _} = Repo.transaction(multi)

Of course, it's still doing multiple inserts. It would be nice to have a straightforward way of taking a bunch of changesets that only touch one table and bulk-inserting them. Maybe validating the changesets and converting them to a list of changes that can be passed to insert_all?