Jsonapi-resources: Reducing N+1 queries by eager loading

[nathanpalmer]

I read through the documentation and couldn't find a built-in solution to this problem. When certain models are serialized we need to make sure that the query to retrieve that entity always adds an includes to avoid all the lazy loaded N+1 queries. We can do this via records as long as the request is on the resources controller. If it's not then I couldn't see a place to extend it.

Let me give an example

class ProjectResource
  has_one :company
end

class CompanyResource
  attribute :plan_start_at

  def plan_start_at
    @model.subscription.created_at
  end
end

As you'll notice each time a company is rendered it is going to retrieve a value out of it's subscription. However, the problem comes in when we request a URL like this

/api/projects?include=company

It's using the records method off of the projects resource and even when it goes to process the include on company it's not calling records on the company resource so we end up with a separate query for every companies subscription.

Now, from what I can tell maybe this is supposed to be taken care of by also requesting the subscription like this

/api/projects?include=company,company.subscription

However this doesn't completely work since I don't need the subscription object and requesting the subscription isn't a concern of the client (since we simply need the subscription in order to calculate a property on the company.)

[lgebhardt]

If requesting company.subscription does invoke the eager loading you could ensure it's in the includes call by overriding the apply_includes method on your Projects resource. You could selectively add it only when the company is requested. It's obviously a hack, but it should work.

A better solution might be to allow additional includes for eager loading on each relationship. I'm not sure how common that would be and whether it's worth the time to add it to the project.

[nathanpalmer]

Thanks, that's where I've looked. So far we've put together a quick extension that allows us to include anything additional by adding a function called always_include to the resource

    def _append_always_include(resource_klass, model_include, options)
      if resource_klass.present? && resource_klass.respond_to?(:always_include)
        resource_klass.always_include(options) + model_include
      else
        model_include
      end
    end

    def resolve_always_includes(resource_klass, model_includes, options = {})
      case model_includes
        when Array
          return model_includes.map do |value|
            resolve_always_includes(resource_klass, value, options)
          end
        when Hash
          return Hash[model_includes.map do |key, value|
            relationship = resource_klass._relationships[key]
            [key, resolve_always_includes(relationship.resource_klass, _append_always_include(relationship.resource_klass, value, options), options)]
          end]
        when Symbol
          relationship = resource_klass._relationships[model_includes]
          if relationship.present?
            always_include = _append_always_include(resource_klass, [], options)
            if always_include.is_a?(Array) && always_include.size == 1
              return { model_includes => always_include }
            end
          end

          return model_includes
      end
    end

    def apply_includes(records, options = {})
      include_directives = options[:include_directives]
      if include_directives
        model_includes = resolve_relationship_names_to_relations(self, include_directives.model_includes, options)
        always_includes = resolve_always_includes(self, model_includes, options)

        records = records.includes(always_includes)
      end

      records
    end

[lgebhardt]

Looks good. Thanks for sharing it.

[romanbsd]

Interesting. We were also looking for a declarative way to include some resources in the resource. We're going to give this a shot.

[nathanpalmer]

@romanbsd The code I posted had a small bug in it that we later fixed. When I'm around a computer I'll send an updated version.

[romanbsd]

@nathanpalmer thanks. On the other hand we're trying to do something a bit different, putting the eager loading aside for a moment; we want to declaratively define which referenced resources should be always included with a specific resource. This is made harder by the fact, that we have STI, and some models have some specific associations, while others don't. So for example, foo.bar won't work, because some foo have bar association, while others don't. At least when done on the request level. Our current approach is try to do the magic in the Resource.find...

[alexdean]

@nathanpalmer curious if you are still using this code, and how it's working out. i have a need for this same functionality (specifying includes to avoid N+1 queries, while not adding the included models to the JSON response). do you have tests for this? what was the 'small bug that we later fixed' which you mentioned?

@lgebhardt would you accept a PR adding this option to jsonapi-resources? i'd be willing to help make that happen if so.

[fractaledmind]

I implemented a patch of the Resource class to allow this functionality. It isn't anything fancy, but it gets the job done in our environment.

# frozen_string_literal: true
module JSONAPI
  class Resource
    class << self
      attr_accessor :_always_eager_load

      def always_eager_load(*relationships)
        @_always_eager_load = relationships
      end

      def apply_includes(records, options = {})
        include_directives = options[:include_directives]
        if include_directives
          model_includes = resolve_relationship_names_to_relations(self, include_directives.model_includes, options)
          records = records.includes(model_includes)
        end

        records.includes(@_always_eager_load)
      end
    end
  end
end

This allows us to call always_eager_load in any Resource. The only constraint this has is that the args passed to always_eager_load must be directly passable to includes.

Like I said, simple but effective.

[nathanpalmer]

@alexdean We're not using it right now. It was part of a mock conversion we did for JSONAPI::Resources that didn't make it to production (for various reasons.) It's still on the roadmap to hit again though and we'll have to make sure our queries are optimized so I plan on revisiting this at that time.

[danielgomezrico]

Is there any update on this? :)