Elasticsearch-rails: undefined method `total_pages' when using Kaminari

[Oshuma]

I know this has been reported before (back in 2014 with #208), but none of the mentioned workarounds fix this issue.

To cover the basics, I do have the kaminari gem before elasticsearch-model and elasticsearch-rails in my Gemfile. ElasticSearch is installed and configured properly, and the .search() method does return valid results if no extra scopes are used after the .records call.

Versions:

• rails 4.2.8
• kaminari 1.0.1
• elasticsearch-model 5.0.2
• elasticsearch-rails 5.0.2
• ElasticSearch 6.2.2, Build: 10b1edd/2018-02-16T19:01:30.685723Z, JVM: 1.8.0_151

Here is actual code from my application:

# In a controller action:
@listings = Listing.active
  .includes(:category, seller: :user_segment)
  .search(params[:q])
  .page(params[:page])
  .records

# Later on in that same action, after checking filter/sort params:
@listings = @listings.where(auction: true)
# ...
@listings = @listings.where(condition: (params[:condition_min]..params[:condition_max]))
# ...
@listings = @listings.order(params[:order_by])

In the view, we're simply using the paginate kaminari helper:

= paginate @listings

And the exact error:

Now what is interesting about this, is that .records has proper pagination methods attached to the relation unless a where() and/or order() clause is called after the fact:

[1] pry(main)> Listing.active.search('bluetooth').page(1).records.class
=> Elasticsearch::Model::Response::Records
[2] pry(main)> Listing.active.page(1).class
=> Listing::ActiveRecord_Relation
[3] pry(main)> Listing.active.search('bluetooth').page(1).records.where(auction: false).class
=> Listing::ActiveRecord_Relation
[4] pry(main)> Listing.active.page(1).total_pages
=> 127
[5] pry(main)> Listing.active.search('bluetooth').page(1).records.total_pages
=> 1
[6] pry(main)> Listing.active.search('bluetooth').page(1).records.where(auction: false).total_pages
NoMethodError: undefined method `total_pages' for #<Listing::ActiveRecord_Relation:0x000055a190f67bb0>
from /home/<cut>/.rbenv/versions/2.4.3/gemsets/<cut>/gems/activerecord-4.2.8/lib/active_record/relation/delegation.rb:136:in `method_missing'

It's almost as if the pagination methods are removed from the AR Relation after calling .records.where(), as the pagination methods (#total_pages, etc) work just fine on an AR Relation that does not go through the ES Records wrapper. There is no default_scope on that model that could be interfering.

[Oshuma]

@karmi Can I get an update on this?

[Oshuma]

For what it's worth, I just switched over to will_paginate in that same application and the exact same problem exists with that library.

Edit: Just to test the scope of the bug. We're using kaminari in production.

[karmi]

Hi @Oshuma, this sounds either like a bug in the "wrapper", as you suggest, or some gotcha in the integration between the pagination library and a) elasticsearch-model b) ActiveRecord. Just to be sure, Listing.active.page(1).total_pages does work or no?

I'm afraid I won't have time to properly look into it soon, but we've scheduled some significant effort for May/June this year to pick up the issues again.

[Oshuma]

@karmi Yes, Listing.active.page(1).total_pages does work and returns a valid value. If you could point me in the right direction, I could try taking a look at fixing this if I get some time. I briefly glanced at the code, but nothing really stuck out as being "wrong".

[yuki24]

Just checking in since it's been a while since this was first reported a few weeks ago. I'm not familiar with elasticsearch-rails, so correct me if I'm wrong.

I wonder what happens under the hood when calling Listing.active.search('bluetooth').page(1).records. If I understand this correctly, once we call the search method it'll become an elasticsearch scope. Then there's a following records call, which I think returns an ActiveRecord Relation object. From that point the ES scope is sort of discarded and now it relies solely on ActiveRecord, so I'm not sure how all these pieces work together.

I would think that this is an edge case where the user of the gem needs to call an additional page explicitly as ES scope is not directly transferable to AR scope.

[karmi]

@yuki24, I think you are somewhere near the problem indeed. The records call indeed returns an AR Relation, _but_ it does try to keep the order from Elasticsearch (see source in the "adapter")... But in some way, the relation is not augmented with Kaminari methods, with the most plausible explanation being that it's lost in the said adapter...

[estolfo]

Hi @Oshuma I don't understand what it semantically means to chain pagination-related methods after the query is "handed over" to ActiveRecord. For instance, using this example:

Listing.active.search('bluetooth').page(1).records.where(auction: false).total_pages

What do you expect 'total_pages' to refer to? The Elasticsearch query or the ActiveRecord query?

There are two queries to 2 separate data sources in the above command, the first to Elasticsearch and the second to your ActiveRecord data storage. It doesn't make sense to paginate the two queries separately. If you could do this, for example, it'll only ever return 1:

Listing.active.search('bluetooth').page(n).records.where(auction: false).total_pages

That said, what is the request in this issue?

[Oshuma]

@estolfo The issue is that pagination methods are getting "lost" somewhere along the way. As in, #total_pages doesn't exist after chaining AR methods (filtering the results from ES).

[estolfo]

Hi @Oshuma Did you see my above comment? This code:

Listing.active.search('bluetooth').page(1).records.where(auction: false).total_pages

actually makes two requests: first it queries Elasticsearch, then it fetches the records from your other datastore using the ids obtained in the Elasticsearch query. It doesn't make sense to apply the pagination to both requests. The pagination should apply to only one. If you want to get the total_pages from the pagination on the Elasticsearch query, you should do:

Listing.active.search('bluetooth').page(1).total_pages

as soon as you call #records, you are doing a separate query using ActiveRecord. If you want to paginate the results there, you should called a pagination method after #records in the method chain. For example:

Listing.active.search('bluetooth').records.where(auction: false).page(1).total_pages

[estolfo]

I'm going to close this, as there's nothing the gem can do about where the pagination is intended.
Please call a pagination method on either the Elasticsearch query chain or ActiveRecord query chain.

[vprigent]

We had this issue about two years ago when updating our implementation on search.
I think I got to the bottom of it today..

Will add my discoveries for the next developer that comes around.

We have an elasticsearch method that hides a couple of things under the rug like pagination. The only thing is that it will always return the same class.

Player.elasticsearch(params).class
> Elasticsearch::Model::Response::Response

now trying to use the .records method...

Player.elasticsearch(params).records.class
> Elasticsearch::Model::Response::Records

At this stage, pagination is kept, total_pages can be called and we can just use that as any active record collection too.

Now, the "issue" happens when we start chaining methods onto that .records method. Let say I just want to remove SQL N+1, that won't change the size of the collection, the number of searched records, or anything else... but!

Player.elasticsearch(params).records.includes(:user).class
> Player::ActiveRecord_Relation

From there, pagination is lost, we aren't relying on the ES Records class anymore but on ActiveRecord.

Hope someone else finds this useful.