Ecto: Preload syntax is confusing

[iwarshak]

I feel like the documentation regarding preloads is confusing, and the docs are unclear as to which syntax to use and when. For example

App belongs_to Form
App belongs_to Provider
Form has_many Sections
Sections has_many Questions

I have found 3, maybe 4 different syntax usages, which all seem to do the same thing.

sq = from s in Section, preload: [questions: ^(from q inQuestion, order_by: q.position)], order_by: s.position
application = Repo.get!(App, id) |> Repo.preload([:provider, [form: [sections: sq]]])

(list, with a nested keyword list and a query as a value)

Repo.one from a in App, where: a.id == ^id, preload: [:provider, [form: [sections: :questions]]]

(list with a nested keyword list)

Repo.one from a in App, where: a.id == ^id,
        join: f in assoc(a, :form),
        join: s in assoc(f, :sections),
        join: q in assoc(s, :questions),
        preload: [:provider, form: {f, sections: {s, questions: q}}]

(list with a mix of keyword lists & tuples)

Repo.get!(App, id) |> Repo.preload([:provider, {:form, {:sections, :questions}}])

(list and nested tuples)

When do you use tuples?
When do you use keyword lists?
What are the preferred ways for loading nested associations?

Perhaps I am missing something, but I have found this to be pretty confusing. I am happy to work on documentation fixes, but I don't know the answer myself.

[josevalim]

That's a very excellent question. At its most basic form, preload can be two things:

• An atom - :form
• A list of atoms - [:form, :provider]

However, we want to nest preloads, so we allow a keyword list, where the key is the current preload and the value is the nesting. Those can also be nested at will:

• Keyword list - [:provider, form: :sections]

Not only that, as you know, we can allow a specific value to be given to a preload. This value can either be a query or a in-query association variable:

join: s in assoc(f, :sections),
preload: [form: s]

Or:

section = ...
Repo.preload ..., form: ^s

However, in the both cases, we may want to further set preloads that's when we allow tuples. The first element is the variable, the second are preloads (which can be atoms, lists, keyword lists... everything we have seen so far as it is recursive):

join: s in assoc(f, :sections),
preload: [form: {s, questions: q}]

Or:

section = ...
Repo.preload ..., [form: {^s, :questions}]

Those are all the supported formats. Your last query is a bug though and it has been fixed to raise on master. :)

Thank you!

[josevalim]

Btw, maybe there is a better syntax for all of this, I didn't enter such discussion above though. I just documented how it works today.

[iwarshak]

Thanks for that explanation @josevalim

What I originally set out to do was be able to preload sections and questions but have them ordered by their respective position. I am using a query like the preload documentation talks about, but I still can't seem to figure that part out:

    section_query = from s in Section, order_by: s.position
    question_query = from q in Question, order_by: q.position
    Repo.get!(App, id) |> Repo.preload([:provider, [form: {section_query, question_query}]])

^^ this fails. Is this even possible? It seems like it should be based on what you said earlier

[josevalim]

You should use the same form as you used here:

 preload: [:provider, form: {f, sections: {s, questions: q}}]

Therefore:

Repo.get!(App, id) |> Repo.preload([:provider, form: [sections: {section_query, questions: question_query}]])

[iwarshak]

It actually errors:

iex(150)>     section_query = from s in Section, order_by: s.position
#Ecto.Query<from s in Section, order_by: [asc: s.position]>
iex(151)>     question_query = from q in Question, order_by: q.position
#Ecto.Query<from q in Question, order_by: [asc: q.position]>
iex(150)> Repo.get!(App, id) |> Repo.preload([:provider, form: [sections: {section_query, questions: question_query}]])
[debug] SELECT a0."id", a0."form_id", a0."provider_id", a0."inserted_at", a0."updated_at" FROM "applications" AS a0 WHERE (a0."id" = $1) [12] OK query=1.2ms
** (ArgumentError) invalid preload `{#Ecto.Query<from s in Section, order_by: [asc: s.position]>, [questions: #Ecto.Query<from q in Question, order_by: [asc: q.position]>]}` in `[:provider, {:form, [sections: {#Ecto.Query<from s in Section, order_by: [asc: s.position]>, [questions: #Ecto.Query<from q in Question, order_by: [asc: q.position]>]}]}]`. preload expects an atom, a (nested) keyword or a (nested) list of atoms
    (ecto) lib/ecto/repo/preloader.ex:49: Ecto.Repo.Preloader.do_preload/4
    (ecto) lib/ecto/repo/preloader.ex:40: Ecto.Repo.Preloader.preload/3

Note: I am only using preloads, and not joins. I am trying to preload sections using the preload query syntax, and also preloading questions that belong to sections.

[josevalim]

Ok. So it may be a bug. Or it may not be a bug. It happens this can be solved by moving the preload to the section query:

question_query = from q in Question, order_by: q.position
section_query = from s in Section, order_by: s.position, preload: [questions: question_query]
Repo.get!(App, id) |> Repo.preload([:provider, form: [sections: section_query]])

I think though we should allow what you wrote to work because it composes better.

[josevalim]

@iwarshak I will make this syntax work soon. I would love if you could send a pull request to improve the docs! You can assume that {^query, questions: questions_query} will work by the time your PR is merged. :)

[iwarshak]

@josevalim I will work on the docs this week.

[josevalim]

Ping. :)

[josevalim]

Pushed docs to master.