Active_model_serializers: Defined relationships should utilize eager loading to reduce n+1 query situations

[NullVoxPopuli]

Expected behavior vs actual behavior

Expected

when rendering a list of objects with relationships, those relationships shouldn't have to be queried for

Actual

[active_model_serializers]    (0.8ms)  SELECT COUNT(*) FROM "attendances" WHERE "attendances"."deleted_at" IS NULL AND "attendances"."level_id" = $1 AND "attendances"."attending" = $2 AND "attendances"."dance_orientation" = $3  [["level_id", 27], ["attending", "t"], ["dance_orientation", "Lead"]]
[active_model_serializers]    (0.8ms)  SELECT COUNT(*) FROM "attendances" WHERE "attendances"."deleted_at" IS NULL AND "attendances"."level_id" = $1 AND "attendances"."attending" = $2 AND "attendances"."dance_orientation" = $3  [["level_id", 27], ["attending", "t"], ["dance_orientation", "Follow"]]
[active_model_serializers]   Attendance Load (1.5ms)  SELECT "attendances".* FROM "attendances" WHERE "attendances"."deleted_at" IS NULL AND "attendances"."level_id" = $1 AND "attendances"."attending" = $2  ORDER BY attendances.created_at DESC  [["level_id", 27], ["attending", "t"]]
[active_model_serializers]   User Load (0.4ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT 1  [["id", 792]]
[active_model_serializers]   Order Load (0.8ms)  SELECT "orders".* FROM "orders" WHERE "orders"."attendance_id" = $1  [["attendance_id", 1032]]
[active_model_serializers]   Package Load (0.3ms)  SELECT  "packages".* FROM "packages" WHERE "packages"."id" = $1 LIMIT 1  [["id", 34]]
[active_model_serializers]   Level Load (0.3ms)  SELECT  "levels".* FROM "levels" WHERE "levels"."id" = $1 LIMIT 1  [["id", 27]]
[active_model_serializers]   Order Load (1.1ms)  SELECT "orders".* FROM "orders" WHERE "orders"."attendance_id" = $1  [["attendance_id", 1031]]
[active_model_serializers]   CACHE (0.0ms)  SELECT  "packages".* FROM "packages" WHERE "packages"."id" = $1 LIMIT 1  [["id", 34]]
[active_model_serializers]   CACHE (0.0ms)  SELECT  "levels".* FROM "levels" WHERE "levels"."id" = $1 LIMIT 1  [["id", 27]]
[active_model_serializers]   User Load (0.4ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT 1  [["id", 784]]
[active_model_serializers]   Order Load (0.9ms)  SELECT "orders".* FROM "orders" WHERE "orders"."attendance_id" = $1  [["attendance_id", 1008]]
[active_model_serializers]   CACHE (0.0ms)  SELECT  "packages".* FROM "packages" WHERE "packages"."id" = $1 LIMIT 1  [["id", 34]]
[active_model_serializers]   CACHE (0.0ms)  SELECT  "levels".* FROM "levels" WHERE "levels"."id" = $1 LIMIT 1  [["id", 27]]
[active_model_serializers]   User Load (0.4ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT 1  [["id", 782]]
[active_model_serializers]   Order Load (1.2ms)  SELECT "orders".* FROM "orders" WHERE "orders"."attendance_id" = $1  [["attendance_id", 1000]]
[active_model_serializers]   CACHE (0.0ms)  SELECT  "packages".* FROM "packages" WHERE "packages"."id" = $1 LIMIT 1  [["id", 34]]
[active_model_serializers]   CACHE (0.0ms)  SELECT  "levels".* FROM "levels" WHERE "levels"."id" = $1 LIMIT 1  [["id", 27]]
[active_model_serializers]   User Load (0.4ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT 1  [["id", 770]]
[active_model_serializers]   Order Load (1.2ms)  SELECT "orders".* FROM "orders" WHERE "orders"."attendance_id" = $1  [["attendance_id", 980]]
[active_model_serializers]   CACHE (0.0ms)  SELECT  "packages".* FROM "packages" WHERE "packages"."id" = $1 LIMIT 1  [["id", 34]]
[active_model_serializers]   CACHE (0.0ms)  SELECT  "levels".* FROM "levels" WHERE "levels"."id" = $1 LIMIT 1  [["id", 27]]

Steps to reproduce

_(e.g., detailed walkthrough, runnable script, example application)_

My Serializer

class AttendanceSerializer < ActiveModel::Serializer

  attributes :id,
    :attendee_name, :dance_orientation,
    :amount_owed, :amount_paid, :registered_at,
    :checked_in_at, :is_checked_in,
    :package_name, :level_name,
    :event_id, :level_id

  has_many :orders

  def amount_paid
    object.paid_amount
  end

  def registered_at
    object.created_at
  end

  def package_name
    object.try(:package).try(:name)
  end

  def level_name
    object.try(:level).try(:name)
  end

  def is_checked_in
    !!object.checked_in_at
  end

  def event_id
    object.host_id
  end

end

where package, level, orders and attendee_name trigger db hits

Environment

ActiveModelSerializers Version _(commit ref if not on tag)_:

_a1826186e556b4aa6cbe2a2588df8b2186e06252_

_(RC 5)_

Output of ruby -e "puts RUBY_DESCRIPTION":

ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-linux]
OS Type & Version:

$ lsb_release -a
LSB Version:    core-2.0-amd64:core-2.0-noarch:core-3.0-amd64:core-3.0-noarch:core-3.1-amd64:core-3.1-noarch:core-3.2-amd64:core-3.2-noarch:core-4.0-amd64:core-4.0-noarch:core-4.1-amd64:core-4.1-noarch:security-4.0-amd64:security-4.0-noarch:security-4.1-amd64:security-4.1-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 15.04
Release:    15.04
Codename:   vivid

Integrated application and version _(e.g., Rails, Grape, etc)_: Rails

---

for attributes where I'm manually calling other relationships (outside of an AMS has_many or belongs_to, I'd need a way to include on the resource, as there is no way AMS could know what relationship those attributes are going to use ahead of time.

[nullvoids]

Having a similar issue while using JsonApi Adapter. I have a many to many relationship between my Deck and Cards model.
This is how my index action looks like for DecksController

  def index
    @decks = Deck.all      # Tried '@decks = Deck.includes(:cards).all' too
    render json: @decks, include: ['cards']
  end

Serializers

class DeckSerializer < ActiveModel::Serializer
  attributes :id, :title
  has_many :cards
end

class CardSerializer < ActiveModel::Serializer
  attributes :id, :image_url, :title, :description, :stats
  has_many :decks
end

Result is a bunch of database queries.

[bf4]

Is there a reason you can't do this outside of AMS, or in a method/block?

[beauby]

:+1: for handling this outside of AMS. What AMS really does at the end of the day is take a vertex/set of vertices of a graph, and build a JSON document representing a neighborhood of the vertex/vertices. Whether you lazily load neighboring vertices (possibly multiple times, although DB caching should handle that in most cases), or you preload the whole neighborhood (using include) should be up to the user. What might make sense though would be to provide utilities to do the "right amount of preloading" based on the serialization options (various includes/fields directives).

[NullVoxPopuli]

@beauby that makes sense.

I guess I'm just lazy.

The goal is to avoid having something like this hard coded on all my controllers:

    render json: Level.includes(
      :attendances => [
        :housing_request,
        :housing_provision,
        :orders,
        :package,
        :attendee
      ]).find(params[:id]), include: params[:include]

Maybe this is turning in to more of something skinny_controllers could/should do. If I'm already passing include in the params, it feels silly to specify include again for ActiveRecord

edit: ref https://github.com/NullVoxPopuli/skinny_controllers/issues/15

[NullVoxPopuli]

I'm going to close this issue, as per @beauby's thoughts, and solve the problem here: https://github.com/NullVoxPopuli/skinny_controllers/issues/15

[beauby]

@NullVoxPopuli Indeed, I'm not saying there is no value in providing a helper for this. However, I believe there is no "one size fits all" solution to this problem, and therefore I'd be more comfortable if the suggested solution of automatically AR-preloading associations lied in an other project.

[NullVoxPopuli]

Yeah, I getchya. :-)

[vergenzt]

I'm not sure if the config was exposed when this issue was opened, but FWIW I'm removing eager loading code from my controllers by defining a custom collection serializer:

# file: app/serializers/relation_serializer.rb

# A collection serializer that allows you to modify the collection before
# iterating through it. Used to specify eager loading on relations within the
# serializer rather than in the controller.
class RelationSerializer < ActiveModel::Serializer::CollectionSerializer
  def initialize(relation, options = {})
    if options[:serializer].respond_to?(:eager_load_relation)
      relation = options[:serializer].eager_load_relation(relation)
    end

    super(relation, options)
  end
end

configuring it:

# file: config/initializers/active_model_serializers.rb

ActiveModelSerializers.config.tap do |config|
  ...
  config.collection_serializer = RelationSerializer
end

and using it from the serializer:

# file: app/serializers/post_serializer.rb

class PostSerializer < ActiveModel::Serializer
  # Return the modified relation.
  def self.eager_load_relation(relation)
    relation.includes(:comments)
  end

  has_many :comments  
  ...
end

The cleaned up controller code can then simply look like:

# file: app/controllers/posts_controller.rb

class PostsController < ApplicationController
  ...

  def index
    render json: Post.all, each_serializer: PostSerializer
  end

  ...
end

It doesn't entirely remove duplication, but IMO it strikes a good level of explicitness; it just seems like it'd be too opaque to me if eager loads were auto-inferred.

[bf4]

@vergenzt Nice. Want to make a PR? This is a good example of where user-land can do a better job extending AMS than AMS can in doing everything out of the box.

[vergenzt]

@bf4 you just mean a docs PR explaining the example? Sure! Any thoughts on where that example should go?

After reading through most of the docs (and re-reading some pages to find specific things I was looking for) I'm still a bit confused about what information lives where. :(

[bf4]

@vergenzt Yeah! Well, as a reader of the docs, you're in the best position to remember how you looked and then put things in the first place you looked :)

[dkniffin]

I really like @vergenzt's solution, but I wanted to take it one step further, so I wrote a version that would automatically do the includes, based on the associations you define in your serializer:

# A collection serializer that allows you to modify the collection before
# iterating through it. Used to automatically specify eager loading, based on the associations
# defined in the serializer
class RelationSerializer < ActiveModel::Serializer::CollectionSerializer
  def initialize(relation, options = {})
    if relation.is_a? ActiveRecord::Relation
      serializer_instance = item_serializer(relation, options)
      associations = serializer_instance.associations.map(&:name)
      relation = associations.present? ? relation.includes(*associations) : relation
    end

    super(relation, options)
  end

  private

  def item_serializer(relation, options)
    serializer_from_resource(
      relation[0],
      options.fetch(:serializer_context_class, ActiveModel::Serializer),
      options
    )
  end
end

[Fernando3643]

I have a problemn with the solution of @vergenzt,

/app/config/initializers/active_model_serializers.rb:4:in block in <top (required)>': uninitialized constant RelationSerializer (NameError)

config/initializers/active_model_serializers.rb

ActiveModelSerializers.config.tap do |config|

    config.collection_serializer = RelationSerializer
  end

app/serializers/relation_serializer.rb

class RelationSerializer < ActiveModel::Serializer::CollectionSerializer
  def initialize(relation, options = {})
    if options[:serializer].respond_to?(:eager_load_relation)
      relation = options[:serializer].eager_load_relation(relation)
    end

    super(relation, options)
  end
end

[Bajena]

Hey, for those of you that are struggling with this problem - you may also consider using a gem of mine: https://github.com/Bajena/ams_lazy_relationships/

What you'd need to do is to define your relationships like this:

class BaseSerializer < ActiveModel::Serializer
  include AmsLazyRelationships::Core
end

class AttendanceSerializer < BaseSeializer
  lazy_has_many :orders
end