Cms: Bypass the creation of search index jobs when saving an element

[jowiz]

Description

I need to bypass the search index jobs because im importing over 8000 products into Craft.
Whenever I use the saveElement method my import gets slower and slower because of the search index job creation.

I've got one big job that fetches all products in chunks. The big job creates smaller jobs that are responsible for saving their own products it contains.

In Elements.php#1991 we can see the UpdateSearchIndex getting pushed into the queue.

Steps to reproduce

1. Create an element, could be an asset or another one.
2. Use the Craft::$app->getElements->saveElement($element); method to save an element.
3. See the job creation of Search index jobs in the queue db table or in the Craft CP sidebar.

Additional info

• Craft version: 3.2.7
• PHP version: 7.2

[brandonkelly]

How would you prefer the search indexes to be updated, then?

[naboo]

+1 for this, but maybe not "disable it". @brandonkelly how about adding an EVENT to the search index job creating so us developers could mute the creation when needed, and then afterwards create our own to update the index ourselves?

For example I have a site with +20,000 product entries in which they are batch edited a couple of times a day, sometimes they get edited 2-3 times after one another (to update specific values or so during the day). When a Content Manager does this it creates X search index jobs * the times each entry gets edited + related entries that also gets updated (if they do). This creates so many entries in the jobs table that the database grows HUGE in size.

For this case it would be nice to mute it -> then run all the entry updates -> then add it for the touched elements to update the index.

[sjelfull]

This goes under the _be smarter about mass-saving scenarios_ banner. Craft should batch the update of the search index, and only do the update once the saving is done.

[naboo]

True @sjelfull. Doesn't "fix" by scenario though when there's edits in a row (maybe even 10-15 minutes apart). Disable it and then add a cronjob to update it from the command line once a day would (for me in this scenario) be ideal.

[naboo]

@brandonkelly what happens if we set $element->propagating to false to avoid the search-index being updated. Does that have a very big negative effect on the element itself and its related functionallity?

[brandonkelly]

This goes under the _be smarter about mass-saving scenarios_ banner. Craft should batch the update of the search index, and only do the update once the saving is done.

@sjelfull Not sure I’m following; that’s basically how it works now, as the initial entry saving/propagation happens right away, and search index updating is handled later on via a queue job.

@brandonkelly what happens if we set $element->propagating to false to avoid the search-index being updated. Does that have a very big negative effect on the element itself and its related functionallity?

@naboo Yeah don’t tamper with that.

[sjelfull]

@brandonkelly Yes, the index updating is handled by a queue job. But that is one queue job per element touched.

If you are saving 8000 elements, that is 8000 queue jobs, instead of one queue job at the end that will update the index for those 8000 elements.

I think there is also other scenarios like this that I can't remember off the top of my head.

Reference

[naboo]

@brandonkelly even if the element is created and fully propegated? So basically only on resave scenarios for Sections which have entry versions disabled.

[naboo]

I think there is also other scenarios like this that I can't remember off the top of my head.

Live example from today. I logged into a site which had 40,000+ search-index queue jobs running. The Content Manager had bulk updated 5,000+ entries 3-4 times during the day. Since also related entries which are touched are also being added to the queue it starts to fill up FAST.

Here we aren't dependent on the search functionallity on the actual site, and we can live without the search-index not being up-to-date from within Craft until (for example) a nightly cronjob does the resaving of all elements.

[brandonkelly]

Alright just addressed this for Craft 3.4. saveElement() now has an $updateSearchIndex arg, which you can set to false if you want to skip it.

resaveElements() also has the same argument, and there we’re actually defaulting it to false, as it’s relatively unlikely that search indexes would need to be updated when batch-resaving elements.

resave/* commands also no longer update search indexes by default, but you can override that behavior by adding the --update-search-index argument:

./craft resave/entries --update-search-index

Search indexes also now will get updated directly in the same request, for console actions, rather than pushing it off to the queue.

@brandonkelly even if the element is created and fully propegated? So basically only on resave scenarios for Sections which have entry versions disabled.

@naboo Right, just leave that property alone. It does a lot more than determine whether to update search indexes.

[timkelty]

@brandonkelly I still have trouble when I'm not the one calling saveElement (e.g. with a FeedMe import).

What do you think about either:

• a global/general setting to toggle this
• making it a prop on craft\base\Element, so we could trigger it via event (like how propagating is), e.g.

        Event::on(
            Entry::class,
            Entry::EVENT_BEFORE_SAVE,
            function (ModelEvent $event) {
                $event->sender->updateSearchIndex = false;
            }
        );

UPDATE – actually…it seems like running 3.4, feedme isn't triggering redindexing jobs when processing…though I'm not sure 💯why

[brandonkelly]

@timkelty Are you running the queue via the CLI? If so then the search index is updated at the same time that elements are saved in 3.4.

[timkelty]

@brandonkelly yeah queue is run via daemonized queue/listen.

[Wiejeben]

This would honestly be pretty interesting. Right now I'm working on a little experiment to stress test Craft Commerce. Inserting 100k products takes me almost 2 hours, updating search indexes is being hit pretty hard in this scenario.

I do understand why the function is necessary, however for "advanced" scenarios there should be a way to trigger a bypass and execute reindexing manually (own risk).