Absinthe: [HELP!] Tutorial: Dataloader Chapter

[bruce]

Does anyone have the time/inclination to contribute to the official tutorial (/guides/tutorial), walking readers through going from manual resolvers to use of the dataloader-enabled features?

It would involve:

• A PR here with:
  
  
  
  • A new /guides/tutorial/*.md file
  
  
  • Adding it to the documentation configuration in mix.exs
  
  
• A PR for absinthe_tutorial implementing the changes

Maybe you're already confident in your abilities at using dataloader—or maybe this is the perfect excuse to build that confidence using an existing application to help you get there. If you've got a reasonably solid understanding of Absinthe's basics, you could help out!

Maybe you're a seasoned contributor, or maybe this is your very first contribution to the project: all are welcome, and we'd be happy to work with you to build this addition to the guide.

Thanks!

[karmajunkie]

I'm kind of interested in taking this one on—I'm going to be diving into converting my use of Absinthe.Ecto to Dataloader soon anyway, and it dovetails nicely into a blog post or series I've been cooking up in my head on how absinthe and graphql let you separate contexts within your application cleanly and stitching them together at the edge of the system.

[bruce]

Fantastic!

One thing to mention, BTW; as @benwilson512 and I discussed yesterday with @smoak, the dataloader helper that's part of Absinthe is currently pretty geared towards Ecto's use of it, batching nested values like:

{
  users {
    name
    posts {
      title
    }
  }
}

If you wanted to batch something toplevel, where everything would happen in a single batch (using Dataloader.KV or something like it to cache calls to external services), right now you need to make your own helper function.

Sorry if that sounds a bit complex & vague; here's the problem code + solution we came up with yesterday: https://gist.github.com/smoak/e24b6774d48515d9fa0fd2c94d712bd1/

I think the tutorial will probably stay focused on the Ecto side in the short-term, but it's a (temporary) issue that's worth keeping in mind as you start to dig in.

[bruce]

@karmajunkie How is this coming along? Are you still interested/need help?

[karmajunkie]

@bruce I've been a little more underwater than expected the last couple of weeks—was planning on carving out some time next week to start in on this project, but if there's someone else who's chomping at the bit to tackle it I can bow out. Definitely still interested otherwise.

[bruce]

Next week seems fine 😄. Thanks again!

[smoak]

I'm interested in doing this, but I'm still trying to understand everything. It looks like I still need to use the load helper from the gist above though.

One thing I'm a little confused about is how to make a dataloader load a list of items (i.e. when the args_map is empty). I'm thinking a query like this:

query {
  teams {
    id
    name
  }
}

There aren't any args so the arg_maps will end up being %{}. Here is some code to sort of illustrate what I am talking about:

https://gist.github.com/smoak/b7cfc956a16e08102e163ce705c70d24

I'm assuming when it gets to the fetch/2 function, the Dataloader.KV doesn't have anything in it? So at this point I need to go make an api call to fetch all the teams? Another thing that's a little unclear is will Dataloader.KV automatically put the result of the api call into the KV?

[smoak]

I got a working example here, but it just uses a module attribute with fake data:

Working repo: https://github.com/smoak/nhl_graph_api

The Dataloader code: https://github.com/smoak/nhl_graph_api/blob/master/lib/nhl_graph_api/loaders/nhl.ex

I think this is the purpose of the KV dataloader source? So if I wanted to get that info from an api call, is the intention that I create a new source (something like an ApiSource?)?

I was hoping to use the Dataloader.KV as an in memory store where it would first check if the item(s) being fetched were in the store. If they weren't found in the store, then it would go and make the api call to fetch the data and populate the store. Is that something that Dataloader will provide? Or is that something I have to implement myself?

[mgwidmann]

@smoak I think its fine that you use data embedded from within a module attribute, but to better illustrate how you'd use it in a real system I'd suggest building a simple Repo module that acted like Ecto and just keep the data in a module attribute there. You have to be careful with making it too simple or it will be too contrived to provide much value and still leave the reader with a few things to figure out themselves.

[benwilson512]

Hey folks I appreciate the interest! I actually think a KV driven example would be awesome, particularly if it doesn't use a database. I know certain people have been really looking for some non database examples involving GraphQL.

I've been working lately on some developments on the Ecto source, so exemplary material there should probably wait until those are in.

[smoak]

I see there is a guides/dataloader.md file. Is that a good file to use? Above it mentions creating files in guides/tutorial/*.md.

[benwilson512]

Ah, yea guides/dataloader.md would be the place to start. At the moment I would focus on the basics of adding dataloader as a dependency, setting one up, adding it to the GraphQL context, and setting up a source. More advanced usage should wait until https://github.com/absinthe-graphql/dataloader/pull/11 lands.

[smoak]

I created a draft with some docs: https://github.com/absinthe-graphql/absinthe/pull/501

Not quite finished, but wanted to get some early feedback

[pvj]

I just started out my first project where I intend to learn more about and
use absinthe, so if you'd like to I can try setting up the dataloader based
on this draft and see where that leads me. Let me know if you'd consider
that valuable feedback.

On Fri, Feb 16, 2018 at 6:50 AM, Scott Moak [e-mail]
wrote:

I created a draft with some docs: #501
https://github.com/absinthe-graphql/absinthe/pull/501

Not quite finished, but wanted to get some early feedback

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/absinthe-graphql/absinthe/issues/443#issuecomment-366150297,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AChPf60vk1bdPcy4Ut-SJYxYwONtXdo2ks5tVRdAgaJpZM4QeWzX
.

--
^\^

[zacck]

@smoak and everyone, going to attempt to finish #501 using this explanation from @benwilson512 that happened today on the #absinthe slack channel

what I want to show is that the N+1 problem isn't specific to GraphQL at all, but ends up being a consequence of designing your context API around this supposition:

One expects domain service functions to return back records.

consider an ordinary server rendered phoenix app, where you have some controller where you're listing posts
getting the posts themselves seems sensible enough:

def index(conn, params) do
  posts = Blog.get_posts(params)
  # rendering and what not
end

now suppose we want to grab the author for each post to include in our table
what are our options?

Option 1) get_posts always returns the author for the post. This solves this problem, but clearly doesn't scale well with complexity
Option 2) for post <- posts, do: Blog.get_author(post) Hey it's N+1 again
Option 3) Blog.get_authors(posts)
Option 4) Blog.get_posts(params, preload: [:author])
each of these suffers in some way or another
Option 1 has issues because the moment you use get_posts in more than one place you probably start needing different amounts of associated data, and so you have an over fetching / under fetching problem
Option 2 is just straight N+1
Option 4 is handy, except you break your context. It isn't that [:author] itself is an ecto query, but nonetheless when you preload you skip any and all access rules you have in your application
so for example maybe you don't show anonymous authors as some business rule
this skips right over that
Option 3 has the best performance characteristics, but if you take it to its logical extension you're going to need a _lot_ of functions like that

so let's invent something to try to make this easier. One thing we can observe straight away is that ( and this is proven by your ability to use preload ) most association lookups do not actually require dedicated business logic

some do
and it's very important that when they do we use it
but others follow the ordinary case
so let's make the get_authors function a bit more generic

Blog.get_data(posts, :author)
# in the blog context
def get_data(posts, :author) do
  posts
  |> Ecto.assoc(:author)
  |> where(anonymous: false)
  |> Repo.all
end
def get_data(entities, assoc) do
  entities
  |> Ecto.assoc(assoc)
  |> Repo.all
end

so we have a more generic function here that applies business rules to the author association, but lets other ones act normally, without a proliferation of functions
this helps, we can mitigate the function boilerplate
it still has some issues though, in that it doesn't compose particularly well. If we are ever in a situation where we don't already have all the posts we need on hand, it gets more dificult
what would be sort of cool
is if we could do this

def get_data(posts, :author) do
  posts
  |> Ecto.assoc(:author)
  |> where(anonymous: false)
end
def get_data(entities, assoc) do
  entities
  |> Ecto.assoc(assoc)
end

notice that all the Repo calls are gone (edited)
we'd be returning ecto queries, which we'd like to be able to compose together
gather everything we need, and then execute it later
however Ecto queries themselves are ill suited for this for two reasons
1) they don't compose well. If you have the query from u in User, where: u.id == 1 and from u in User, where: u.id == 2 there's no particularly clean way to merge those
2) you risk exposing data you dont' want to, since the caller can make ad hoc changes to the ecto query
enter: data loader sources. The idea here is that by rejecting the supposition "One expects domain service functions to return back records."
we're freed to have the domain service return data structures that can efficiently compose, enforce data access, and minimize context boilerplate (edited)
This is the point of a Dataloader source. The protocol provides a generic client API, and then the sources are datastructures that implement that protocol with respect to a context. The domain's job is in fact _not_ to materialize data
only the caller actually knows when they want data
because _when_ you want data depends on what you're doing with the data
and that's something only the caller knows

Notes

• Dataloader takes a default context variable

def context(ctx) do
  default_params = Map.take(ctx, [:current_user])
  source = Whatever.data(default_params)
  Dataloader.new |> Dataloader.add_source(...)
  Map.put(ctx, :loader, dataloader)
end