Searchkick: Reindex creates index with wrong mapping/settings

[dvandersluis]

Intermittently (enough that this is becoming a real issue for us) a reindex job is creating a new index that has the wrong settings (for example, none of our analyzers) and wrong mapping (all nested properties aren't getting defined as nested). When this happens, it seems that not all the data gets indexed either, but no exception is raised.

This index has a very custom setup (including both settings and mappings keys). We also set merge_mappings: true, but I'm not sure if that's related. The index also has a lot of data (usually around 3.5GB).

Is there some reason this would happen and something we could do to prevent this? Perhaps a way to verify the mapping/settings before importing?

We're using ES 7.3.2, but also experienced this on 5.x and other 7.x versions.

[ankane]

Hey @dvandersluis, the only case I'm aware of that creates an index without the proper mapping is when Model.reindex is never called on the index but record.reindex is.

[dvandersluis]

It seems to be triggered by calling Model.reindex in cron (we have a nightly reindex job), so I don't believe that should be happening? I'm not sure about the internals here though.

[ankane]

If you use dynamic index names, it's possible that Model.reindex isn't being called for some indexes, but I'm not sure how that would happen with static names. If you're able to reproduce with the gist in the Contributing Guide, I can dig into it. Otherwise, you'll probably need to fork and add debugging to get to the bottom of it.

[ankane]

Cleaning up stale issues

[AlysonBasilio]

@dvandersluis Did you solve this? We were having the same problem with almost exactly the same conditions: We have a cron job that runs Model.reindex. When it was running, it changed index mappings to wrong mappings. But, if we use console to run Model.reindex, it creates index with correct mappings.

After investigating we discovered that try to reindex without last reindex have finished was causing this.

Looks like Searchkick's reindex run clean_indices before start a new reindex but it doesn't kill current reindex process, so it erases index mappings but the first reindex continues to run by creating a index with default elasticsearch mapping. @ankane, should reindex job check if index exists before insert new documents?

[ankane]

Hey @AlysonBasilio, nice debugging! I think a check before each reindex call will slow down indexing significantly, so I'd recommend one of the alternatives depending on the behavior you want:

1. To have the old job stop when a new one starts, disable automatic index creation
2. To allow for overlapping jobs, use the retain: true option and write custom cleaning logic

[ankane]

Also, fwiw, I would avoid scheduling Model.reindex to run automatically in most situations (since it can affect the mapping). You can sync records to the current index with:

Product.reindex(scoped: true)

Edit: Simpler code

[dvandersluis]

@AlysonBasilio This is really useful information, thank you. We are also running reindex in a cron.

@ankane, if we change to not call Model.reindex in a scheduled job, how do you recommend dealing with mapping changes?

[ankane]

I would run it manually when needed for most cases.

[AlysonBasilio]

@dvandersluis in our case, we implemented the follow helper:

  def run_model_reindex(model)
    index = Searchkick::Index.new(model.searchkick_index.name)

    if index.all_indices.count > 1
      log_message("Reindex is already happening for #{model}")
    else
      model.reindex
    end
  end

And we have to always use it to run reindex. This should help you with mapping changes.

@ankane I didn't find docs for retain option, but in index.rb retain: true disable clean_indices before reindex starts, right? Wouldn't it create a lot of unused indices?

[ankane]

@AlysonBasilio You'd probably want to write custom cleaning logic if you took that approach. Out of curiosity, is there a reason you're doing a full reindex vs importing records into the existing index?

[AlysonBasilio]

The reason is that we are working with an old system that we are starting to upgrade by small steps. We plan to change full reindex to importing records using async jobs soon. Full reindex started to create some problems only this week, a technical debt that we are starting to pay :disappointed:.

[ankane]

I added a safety check on master to prevent the index from being promoted without the proper mapping.