Ecto: expose API for query creation

[mfrasca]

I have been trying to build query values from a yecc parser (Erlang). most of you in the Ecto team are surely aware of the attempt, at least @josevalim . anyhow, I first targeted the quote format, until I received comments explaining that these values are meant to be used in macros, or, as far as I understand, at compile time, not during execution.

next attempt, I described it here on the elixir forum, which did not trigger any reaction.

so I continued on that road and tried to achieve the same from my erlang parser. which again I discussed on the elixir forum, and the bottom line was that I need an API from ecto.

what I wish to have is an official way to replace the entries wheres, joins, select and from. that is, given a Ecto.Query, get a new one with one of the fields redefined.

the sequence I wish to perform, but without risking misusing ecto, is this:

1) create an empty Ecto.Query
2) based on (1), create a query with a from field,
3) based on (2), create a query with also a select field,
4) based on (3), create a query with also a joins field,
5) based on (4), create a query with also a wheres field.

w = %Ecto.Query{}
w = %{ w | from: %Ecto.Query.FromExpr{source: {"plant", nil}}}
w = %{ w | joins: [
  %Ecto.Query.JoinExpr{
    source: {"accession", nil}, 
    qual: :inner,
    on: %Ecto.Query.QueryExpr{
      expr: {:==, [], [
        {{:., [], [{:&, [], [0]}, :accession_id]}, [], []},
        {{:., [], [{:&, [], [1]}, :id]}, [], []}
      ]}}},
  %Ecto.Query.JoinExpr{
    source: {"location", nil}, 
    qual: :inner,
    on: %Ecto.Query.QueryExpr{
      expr: {:==, [], [
        {{:., [], [{:&, [], [0]}, :location_id]}, [], []},
        {{:., [], [{:&, [], [2]}, :id]}, [], []}
      ]}}}
  ]}
w = %{ w | select: %Ecto.Query.SelectExpr{expr: [{{:., [], [{:&, [], [2]}, :code]}, [], []}, {{:., [], [{:&, [], [1]}, :code]}, [], []}, {{:., [], [{:&, [], [0]}, :code]}, [], []}]}}
Botany.Repo.all(w)

or is there already a correct way to do what I wish to achieve?

[josevalim]

The API for query creation is exactly what we have today. You are thinking about the internals but it is not necessary to fiddle with them all. Our query api supports both interpolation and dynamic values. For example, imagine you have to parse "foo:43" and convert it to a query. You can do this:

query = "table" # or Schema
[field, value] = String.split("foo:43", ":")
field = String.to_existing_atom(field)
query = from q in querry, field(q, ^field) == ^value

And so on. Ben mentioned the correct way to solve this in the forum here: https://elixirforum.com/t/how-to-produce-executable-code-using-yecc/23644/12 - In a nutshell, you shouldn't build the query inside the parser because the parser goal is to return an AST. Then, in Elixir code, you traverse the AST, building the Ecto query.

Also, please note that we don't use the issues tracker for feature proposals. The mailing list is a better place for this discussion.

[josevalim]

And to be clear, yourr parser AST is not going to be elixir quoted expressions. It is simply a representation of what you parser. So going back to the previous example, if you have to parse: "foo:42", your parser should return something like:

{:where, :foo, "42"}

Which then will be translated to Ecto queries.

[mfrasca]

I'm so sorry, but I don't see how the answer closes the question. my problem is not in parsing strings, I know how to do that, and how to produce whatever reasonable representation of however complex queries. my problem is that I do not know how to gradually, stepwise, build a Ecto.Query value corresponding to a non-trivial query.

[josevalim]

In this case, it is better to continue the discussion in the forum. The issues tracker is for bugs/issues only.

[mfrasca]

you probably then want to review your labels:

[josevalim]

Review the labels in which sense? Also, the first entry in the issue template says:

• Do not use the issues tracker for help or support requests (try Elixir Forum, Stack Overflow, IRC or mailing lists, etc).

[josevalim]

Oh, I see what you mean. I said it is for bugs/issues only, but features is also on the list. It has been fixed, thanks!

[mfrasca]

YMMV/IMO, if I was member of the project, I would first make a strong point in favour, and then vote for keeping also feature requests in the issue tracker, and I would be happy with an extensive pool of open issues, to be used as backlog when planning sprints. that's how I manage the projects I manage. again, YMMV.

[josevalim]

This is how we do it at company projects but this is open source. If
someone wants a feature, then the best way is to submit a pull request,
rather than have a backlog that will never be worked on. You can use the
mailing list to propose features though, also to discuss if they would be

accepted or if someone is willing to implement it.

José Valimwww.plataformatec.com.br
http://www.plataformatec.com.br/Founder and Director of R&D

[mfrasca]

• precisely because this is open source, you should (IMO) prefer open issues,
• your way you do not promote interaction/integration (one has an idea, someone else implements it),
• you also break the single inbox policy,
• and you make feature requests less visible to people willing to dig in the software.

[josevalim]

In all projects I have tried “open issues” approach in more than 12 years maintaining open source projects, my conclusion is that it doesn’t work. The end result is an issues tracker with hundreds or thousands (yes, I have been there) of open issues, where it is impossible to do anything and a large amount of time is spent on triaging issues rather than fixing stuff. That goes on until the project declares issue bankruptcy and has a bot closing everything older than X months, regardless of how good the bug report was or of how complete a patch was.

A single inbox only makes sense if all items have the same priority and in this case they don’t. Fixing any pending issue has higher priority than any feature request.

Maybe there is an approach to making open issues work but I definitely don’t know what it is. Especially on large scale projects. Until then, on the projects I maintain, the issues tracker are about issues first and foremost. :)