Ecto: Add a function to convert to map

[jack-stasevich]

I have JSON API, and I need convert selected data with Poison. It would be better to have a function converting to map instead of do this manually.

Eg:

|> where(city: “Berlin”)
|> to_map

[michalmuskala]

I'm not really sure what such a function would do. Strip the struct and leave a bare map? If so there's Map.from_strcut/1 to do that already.
But from my experience it's rather rare to return whole database records from API, you generally at least remove sensitive fields or maybe even do some more advanced transformations.

[wojtekmach]

There was a similar issue discussed on the mailing list a while ago, see: https://groups.google.com/forum/#!searchin/elixir-ecto/wojtek/elixir-ecto/IQLJVn-1f84/9VFjMZsSIgAJ

I think the general consensus for now is that it's better to decouple getting the data from the DB and presenting it to the user - and so using something like Phoenix JSON views are recommended. See the thread for other ideas.

[jack-stasevich]

@wojtekmach yes, I need the same function as in forum topic. But as I can see it was not solved. I use socket and send response from there, no need for json views. I prefer it to be pure with as few abstraction layers as possible.

[jack-stasevich]

@michalmuskala I use select to take only needed fields, basic eg:

I want to take 2 cities and to receive:

[{
    "title": "Berlin"
},
{
    "title": "Amsterdam"
}]

I can do (from docs):

1 option

cities = City |> select([c], {c.title}) |> limit(2) |> Repo.all 

Result

[{"Berlin"}, {"Amsterdam"}]

No field, can't be used

---

2 option

cities = City |> select([c], %{"title" => c.title}) |> limit(2) |> Repo.all

Result

[%{"title" => "Berlin"}, %{"title" => "Amsterdam"}]

May look like it works, but I need to specify fields twice. Another major problem - does not work with preload

---

3 option

cities = City |> select([:title]) |> limit(2) |> Repo.all

Result

[%Project.City{__meta__: #Ecto.Schema.Metadata<:loaded>, id: nil,
  inserted_at: nil, slug: nil, title: "Berlin",
  trains: #Ecto.Association.NotLoaded<association :trains is not loaded>,
  updated_at: nil},
 %Project.City{__meta__: #Ecto.Schema.Metadata<:loaded>, id: nil,
  inserted_at: nil, slug: nil, title: "Amsterdam",
  trains: #Ecto.Association.NotLoaded<association :trains is not loaded>,
  updated_at: nil}]

Best syntax, works with preload, but shows all fields with nil and meta data.

---

I can't find any solution. Any more options I missed?

My colleagues have this function:

Rails - as_json
Laravel - toJson

It is easy to use and very convenient. I can't express how much I need it here.

Can Ecto have such function please?

[josevalim]

The linked thread has a function you could add to your code to filter out only the data you want as a map later. We recommend using it for now: https://groups.google.com/d/msg/elixir-ecto/IQLJVn-1f84/ejgWe6kUIgAJ

[jack-stasevich]

If I understand, the proposed syntax is:

select = [:title, trains: [:id, :number]]
City
|> preload(:trains)
|> select(^select)
|> Repo.all
|> to_map(select)

While it is a time walkaround, we already have a syntax for it, we just miss a function to do like this:

City
|> preload(:trains)
|> select([:title, trains: [:id, :number]])
|> Repo.all
|> to_map

@josevalim Can you please tell, if we can expect a clean function with nice syntax like above some time later (when you have time for it) ?

[josevalim]

The problem with to_map is that we no longer have the query information so we won't know what to include in the map. It could be a function in the repository but I am not convinced about it either.

[jack-stasevich]

@josevalim yes, Repo.to_map will be also great.

We are using Rails and Laravel for our projects but we like Elixir/Phoenix and want to switch to it. We use this function in every project and need it, it is very useful. I don't know how to convince you more, I hope it can be added.

Thank you.

[michalmuskala]

I'm only afraid that to_map in that context makes no sense for a function that returns a list...

[josevalim]

Yes, naming is part of the issue. :) But we can probably figure out a bare name. Maybe all_as_maps or an option to all?

[jack-stasevich]

@michalmuskala @josevalim sorry, we are only trying. I also considered single items like with Repo.first, a map / list also does not fit for a name.

An option to all, first, etc is also a great idea.

[jack-stasevich]

Sometimes there are also cases, when you need the fetched data to do something, but also to return as json.

Some kind of:

cities = City |> select([:title]) |> limit(2) |> Repo.all

# do something with cities, then convert

cities |> Repo.convert_to_map

Don't know, if it can work this way, but I will be happy with a simple solution like Repo.all_as_map, option to all or another idea.

[vladshcherbin]

@StasevichJack at least I'm not the only one who wants this function 👍

[vladshcherbin]

@josevalim can we reopen (and rename if needed) this issue and tag it as a feature to have on a roadmap, more & more people are asking for it so maybe it's not a bad idea to have it?

[josevalim]

I can reopen but I won't add to the roadmap because I still don't know how we should expose this. :)

[vladshcherbin]

Oh, man, this is the most needed feature for me as well.

Every time I need to use preload, Ecto drives me crazy. I don't know, how you guys deal with meta data and unneeded fields, but for me this is hell.

Query:

University
|> preload(:faculties)
|> where(slug: "slug")
# |> select([:id, :fullTitle, faculties: [:fullTitle]])
|> Repo.first

This gives me the result with meta data and unneeded fields. I can use select, but it won't help.

But the query is correct:

SELECT u0."id", u0."fullTitle" FROM "universities" AS u0 WHERE (u0."slug" = 'slug') ORDER BY u0."id" LIMIT 1 [] OK query=3.0ms queue=2.5ms
SELECT f0."fullTitle", f0."university_id" FROM "faculties" AS f0 WHERE (f0."university_id" IN ($1)) ORDER BY f0."university_id" [15] OK query=1.2ms queue=0.1ms

Why, why it gives me all fields 😞

And the meta data... It almost gives me what I want, the only remaining thing - remove meta data and leave only fields, I asked for.

@josevalim is there any opportunity to deal with it or a possible solution?

[zhangsoledad]

faculties_query = from f in Facultie, select: [f.fullTitle]
Repo.all(from u in University, preload: [faculties: ^faculties_query], limit: 1) |> List.first

@VladShcherbin this may what you need
see https://hexdocs.pm/ecto/2.0.0-rc.0/Ecto.Query.html#preload/3

Repo.one(from u in University, preload: [faculties: ^faculties_query]

** (Ecto.MultipleResultsError) expected at most one result but got (num) in query:

seems Repo.one has conflict with preload @josevalim , in ecto-2.0.rc.0

[vladshcherbin]

@zhangsoledad yes, select in preload query can help with returning only necessary fields. But there is another problem - we can't select fields in the main query (because preload needs all fields).

So, in your example you successfully take only needed fields from _faculties_query_, but _can't_ (or maybe I don't know how) select fields in the main query.

# error
Repo.all(from u in University, preload: [faculties: ^faculties_query], select: [u.fullTitle], limit: 1) |> List.first

[vladshcherbin]

@zhangsoledad as for the conflict, we can use first

Repo.one(first(from u in University, preload: [faculties: ^faculties_query]))

[josevalim]

I had an idea to solve this issue. Today, when you write:

select: [:id, :name, :age]

It is a shortcut for:

select: take(p, [:id, :name, :age])

What if instead of take we provide:

select: struct(p, [:id, :name, :age])
select: map(p, [:id, :name, :age])

The first returns a struct, this second returns a map. This way we provide a similar way for fetching maps or structs, solving this issue without adding to the repo API.

[michalmuskala]

@josevalim I really like this approach. Should this be struct or schema?

[vladshcherbin]

@josevalim looks like a great and elegant solution 👍 ❤️

[zhangsoledad]

@josevalim ❤️ 💚 💙 💛 💜

[jack-stasevich]

@josevalim Still celebrating this event. Thank you!!!
"Keep Calm and Carry on Coding." =)

[vladshcherbin]

@josevalim Thank you! :)

[wojtekmach]

@josevalim @michalmuskala wdyt about similar extension:

select: list(p, [:id, :name, :age])

? (it's an equivalent of: select: [p.id, p.name, p.age])

Apart from use cases similar to select: struct/map, one concrete example where this is useful is it would simplify this implementation: https://github.com/hexpm/hexpm/blob/45341b/lib/hexpm/repo.ex#L65:L71

(sorry for being lazy and not proposing this to the list :))

[josevalim]

@wojtekmach it doesn't feel to be frequent enough to warrant a new addition to the Ecto API. :)