Elasticpress: bulk indexing extremely slow

[mustafauysal]

We upgraded to version 1.8 and realized that bulk indexing extremely slow. The reason of this, calling wp_cache_flush() in wp-cli.php

Is there anybody have same problem? (We are using memcached for object caching)

Thanks.

[mustafauysal]

Thanks @cmmarslender for https://github.com/10up/ElasticPress/commit/8f5058f8973801c9029a13e01721ed86a680b988#diff-c537e2c20a88117b8e57e904f6e80417 (I didn't realize before)
Hope new version coming soon!

[tlovett1]

@mustafauysal was this never resolved?

[mustafauysal]

Nope, It isn't resolved. So we didn't update ElasticPress on production, still running 1.7.

[cmmarslender]

Just to be certain.. @mustafauysal Have you tested 1.9 to see if there were any improvements? It was only released a day or two ago.

[mustafauysal]

@cmmarslender Today I checked on same machine, at the same time.

Results

ElasticPress 1.7

Number of posts indexed on site 1: 170746
Total time elapsed: 5.972,593
ElasticPress is currently deactivated, activating...
Success: ElasticPress was activated!
Success: Done!

ElasticPress 1.9

Number of posts indexed on site 1: 170746
Total time elapsed: 15.200,681
ElasticPress is currently deactivated, activating...
Success: ElasticPress was activated!
Success: Done!

[cmmarslender]

@mustafauysal Running some tests now - Are you using an object caching drop in (object-cache.php)? If so, can you link me to the particular one you are using?

[mustafauysal]

Yes, I'm using memcached object cache (Version 2.0.2) – https://wordpress.org/plugins/memcached/

[cmmarslender]

Hi @mustafauysal

I've done a bit more looking into this, and came up with the following results:

Indexing in general is slower with object-cache.php file, unless all your posts are currently in cache. Likely, when you are indexing, most of your posts aren't in the object-cache, so things slow down a bit (at this point, I'm assuming WordPress is adding items that were missing from cache as it goes). I'm not sure if there is anything to be done about this, but it's an observation that I was able to replicate consistently.

Even with this difference though, my overall indexing average per post is quite a bit lower than yours seem to be. I have a few ideas on what might be causing this.

For me, about 15% of the indexing time is communicating with the ElasticSearch server - the rest of that is querying MySQL, running any filters for post content, etc. If you have any filters that are running on post_content, or any heavy shortcodes, you could try disabling those during indexing, to see if that helps improve speed at all (as long as you don't need the processed shortcode indexed)

Also, take a look at the "Segments and Merging" section of this page: https://www.elastic.co/blog/performance-considerations-elasticsearch-indexing
One issue that I've run into with larger sites is merges falling behind. If you see INFO level log messages saying "now throttling indexing" in the elasticsearch server log, this could be an issue for you. If you have fast enough disks, or aren't worried about saturating disk i/o during indexing on your elastic search server, you could consider setting the index.store.throttle.type to none, to prevent this throttling. In my experience, this sped things up significantly for me, on larger datasets.

[mustafauysal]

@cmmarslender Thanks for detailed info.

We fixed this problem by removing the following parameters.

'cache_results '  => false,
'update_post_meta_cache' => false,
'update_post_term_cache' => false,

If these $args effect another ElasticPress users we might think to remove it, otherwise I can use filter.

Thanks.

[cmmarslender]

Here is what the impact is, from a "time to index" point of view on 8,424 posts:

| | 1 | 2 | 3 | 4 | 5 | 6 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| No Caching | 65.736 | 63.182 | 67.756 | 67.75 | 71.885 | 63.785 | 66.68 |
| Allow Caching | 46.317 | 48.572 | 48.469 | 45.143 | 46.335 | 43.674 | 46.42 |

Pretty big change in time to index, and if we're working on a slower ES backend or with more posts, the effect will just be magnified.

I think these changes were added back when we were trying to solve issues of php running out of memory on larger datasets during indexing - the commit is here

@lukaspawlik Do you remember if this was specifically added to address memory issues?

[lukaspawlik]

@cmmarslender yes - this was specifically added to address memory issues however the final solution was added here https://github.com/10up/ElasticPress/commit/622dade9e6bd7865a33bef55f346ed28c6af9918

If we remove

'cache_results '  => false,
'update_post_meta_cache' => false,
'update_post_term_cache' => false,

we may see memory consumption issue back again however it shouldn't be as that significant as it was in the past (as https://github.com/10up/ElasticPress/commit/622dade9e6bd7865a33bef55f346ed28c6af9918 solves the biggest memory usage). Maybe we could add wp-cli switch to enable / disable using cache during indexing?

[cmmarslender]

@tlovett1 @allan23 Thoughts on your preferred way to proceed here?

[tlovett1]

I'm fine with adding a command line parameter to disable caching.

[tuanmh]

I'm fine with adding a command line parameter to disable caching.

Also would be great if we had these options on admin UI as well.

[tlovett1]

Would love to see this tested in 2.1 in the develop branch.

[cmmarslender]

Tested with and without cache_results and results were very similar to above, with indexing being significantly faster with this removed. Tested memory usage with and without, and overall, we use about 2MB more memory with it removed, but memory usage doesn't grow with each iteration, so overall, my recommendation is to remove this - Pushed f0ef475 that should help significantly with the slowness here.

Opened PR https://github.com/10up/ElasticPress/pull/532 for this