Elasticpress: find_related() not limiting after 3.0 update

[columbian-chris]

After installing the 3.0 plugin update, I was getting errors using the old ep_find_related() function that was working before, so I updated my code using ElasticPress\Features::factory()->get_registered_feature('related_posts')->find_related( $post->id, 3 ) but it's no longer limiting the posts returned to 3 for some reason. Anybody else experiencing this issue?

I disabled plugins, reindexed, changed settings and reindexed again; nothing took.

Current WordPress version: 5.2
Current ElasticPress version: 3.0
Current Elasticsearch version: 6.3.0
Running on a local Ubuntu 18.04 vagrant box with Java 10.0.2

[tlovett1]

@columbian-chris how many posts are you getting? It works for me, but it's worth mentioning the widget caches output for an hour. So if you change the limit it takes awhile to reflect. We need to implement smarter cache invalidation.

[columbian-chris]

@tlovett1 It's returning 1,494 posts. The site has 2,679 published posts in it. I refreshed it just now (the next day) and it returned the same amount, so the caching hour appears to be up.

[tlovett1]

Can you install the EP Debug Bar and show us the query info from there? I'm not able to reproduce this.

[columbian-chris]

Host: http://192.168.33.10:9200
Time Taken: 93 ms
URL: http://192.168.33.10:9200/columbiantest-post-1/post/_search
Method: POST
Headers: 
array(1) {
  ["Content-Type"] => string(16) "application/json"
}
Query Args: 
array(66) {
  ["more_like"] => int(1876868)
  ["posts_per_page"] => int(3)
  ["ep_integrate"] => bool(true)
  ["error"] => string(0) ""
  ["m"] => string(0) ""
  ["p"] => int(0)
  ["post_parent"] => string(0) ""
  ["subpost"] => string(0) ""
  ["subpost_id"] => string(0) ""
  ["attachment"] => string(0) ""
  ["attachment_id"] => int(0)
  ["name"] => string(0) ""
  ["static"] => string(0) ""
  ["pagename"] => string(0) ""
  ["page_id"] => int(0)
  ["second"] => string(0) ""
  ["minute"] => string(0) ""
  ["hour"] => string(0) ""
  ["day"] => int(0)
  ["monthnum"] => int(0)
  ["year"] => int(0)
  ["w"] => int(0)
  ["category_name"] => string(0) ""
  ["tag"] => string(0) ""
  ["cat"] => string(0) ""
  ["tag_id"] => string(0) ""
  ["author"] => string(0) ""
  ["author_name"] => string(0) ""
  ["feed"] => string(0) ""
  ["tb"] => string(0) ""
  ["paged"] => int(0)
  ["meta_key"] => string(0) ""
  ["meta_value"] => string(0) ""
  ["preview"] => string(0) ""
  ["s"] => string(0) ""
  ["sentence"] => string(0) ""
  ["title"] => string(0) ""
  ["fields"] => string(0) ""
  ["menu_order"] => string(0) ""
  ["embed"] => string(0) ""
  ["category__in"] => array(0) {}
  ["category__not_in"] => array(0) {}
  ["category__and"] => array(0) {}
  ["post__in"] => array(0) {}
  ["post__not_in"] => array(0) {}
  ["post_name__in"] => array(0) {}
  ["tag__in"] => array(0) {}
  ["tag__not_in"] => array(0) {}
  ["tag__and"] => array(0) {}
  ["tag_slug__in"] => array(0) {}
  ["tag_slug__and"] => array(0) {}
  ["post_parent__in"] => array(0) {}
  ["post_parent__not_in"] => array(0) {}
  ["author__in"] => array(0) {}
  ["author__not_in"] => array(0) {}
  ["cache_results"] => bool(false)
  ["ignore_sticky_posts"] => bool(false)
  ["suppress_filters"] => bool(false)
  ["update_post_term_cache"] => bool(true)
  ["lazy_load_term_meta"] => bool(true)
  ["update_post_meta_cache"] => bool(true)
  ["post_type"] => string(0) ""
  ["nopaging"] => bool(false)
  ["comments_per_page"] => string(2) "50"
  ["no_found_rows"] => bool(false)
  ["order"] => string(4) "DESC"
}
Query Body: {
    "from": 0,
    "size": 3,
    "sort": [
        {
            "post_date": {
                "order": "desc"
            }
        }
    ],
    "query": {
        "more_like_this": {
            "like": {
                "_id": 1876868
            },
            "fields": [
                "post_title",
                "post_content",
                "terms.post_tag.name"
            ],
            "min_term_freq": 1,
            "max_query_terms": 12,
            "min_doc_freq": 1
        }
    },
    "post_filter": {
        "bool": {
            "must": [
                false,
                {
                    "term": {
                        "post_type.raw": "post"
                    }
                },
                {
                    "term": {
                        "post_status": "publish"
                    }
                }
            ]
        }
    }
}
Query Response Code: HTTP 200
Query Result: {
    "took": 42,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 0,
        "max_score": null,
        "hits": []
    }
}

[mediasecure]

I use the function in functions.php (from theme)

ElasticPress\Features::factory()->get_registered_feature( 'related_posts' )->find_related( get_the_ID(), $return = 4 );

get_the_ID() instead of $post->id

[columbian-chris]

@mediasecure In the context of where I'm calling the method, get_the_ID() is returning the same value as $post->id and they're both the expected/correct value, so that isn't it.

No success after plugin upgrade to 3.0.1, even trying the now reintroduced and now deprecated ep_find_related() function. It's still returning a massive amount of posts.

[columbian-chris]

Again tried deactivating all plugins and this time switching to twenty nineteen theme, still returning 1,494 posts. Downgraded plugin back to 2.8.2 and reindexed and it appears to be working fine again, only returning the expected 3 posts.

[columbian-chris]

Can confirm this is happening on both my local vagrant environment and on a Pagely production server. This is seriously not happening to anyone else ??

[ekandreas]

We have to adjust the count after the feature query. Sometimes it gives more posts than expected.

This is a hack:

$limit = 4;
$posts = ElasticPress\Features::factory()
    ->get_registered_feature('related_posts')
    ->find_related($post_id, $limit);

if ($posts && is_array($posts) && sizeof($posts) > $limit) {
    $posts = array_slice($related_posts, 0, $limit);
}

[columbian-chris]

@ekandreas Has the query in ElasticPress 3 changed? Why is it sometimes giving more posts than expected? Should that be considered a bug? I should emphasize that in the environments I've tested in, it's not just giving a few more posts than expected here and there; we're talking over a thousand every time.

[ekandreas]

I'm so sorry, but I just wanted to answer your question if this happens for anyone else.
In development this is not an issue but in our production environments we get more than specified posts in return.

[columbian-chris]

I have discovered what just might be the culprit.

It appears to be adding sticky posts into the results. I have 1,490 sticky posts. When I ask it to return 5 related posts, it returns 1,495. When I ask it to return 8 related posts, it returns 1,498.

Can anyone repeat this on their end?

Quick test:
If replicable, echo count( ElasticPress\Features::factory()->get_registered_feature( 'related_posts' )->find_related( $post_id, $return = 5 ) ); should return five posts plus the number of sticky posts you have.
Then try echo count( ElasticPress\Features::factory()->get_registered_feature( 'related_posts' )->find_related( $post_id, $return = 12 ) ); and that should return the number of sticky posts plus twelve.

[columbian-chris]

I added 'ignore_sticky_posts' => true to the $args in the find_related() method in /elasticpress/includes/classes/Feature/RelatedPosts/RelatedPosts.php (line 89 in 3.0.3) and now my issue is fixed. Is there any reason why adding that argument might be not be a good idea?

I'd like to add that in comparing the ep_find_related() method in 2.8.2, the code looks identical to the find_related() method in 3.0, so I'm not sure what is making the difference here.