Elasticsearch-rails: Allow for separation of read and write indices/aliases

[sgringwe]

It'd be nice if there was a way to specify what index (alias) a model should write from separately from the one to read from. This would be very helpful when doing a bulk reindex so that I could have a model read from the old alias while reindexing but write to an alias that points to both the new and the old index to make sure that no writes are lost. Could this be supported?

[karmi]

@sgringwe Yeah, this looks like a nice enhancement, and it would indeed simplify these re-indexing / index swapping scenarios. I'll try to find some time for that in near future -- it will be a bit invasive, I think, since we need to make the index_name method more complex...

[sgringwe]

@karmi sounds great

[fbernier]

I might look into it eventually as I just implemented a re-indexing / index swapping rake task for an app and I had to do some dirty workarounds to get it to work.

[karmi]

@fbernier Feel free to ping me here or on IRC if you get into it, this will be invasive for the code :)

[sgringwe]

@karmi @fbernier in case it is helpful, i created a gist with a reindexer i wrote as well: https://gist.github.com/sgringwe/876d1db0880e83bec2fe.

[jadetr]

+1 for this feature

[pnull]

:+1:

[shoito]

:+1:

[alexlitvak13]

@karmi do you know if this is on the roadmap? Alternatively, can you suggest good workarounds for zero-downtime reindexing following a schema change?

For anyone else following this, this documentation describes the issue quite well: https://www.elastic.co/blog/changing-mapping-with-zero-downtime

[karmi]

@alexlitvak13 I'm starting to be more and more convinced that this actually doesn't belong to this gem at all. It's more about "index management" strategies, and the correct approach on the Elasticsearch level is to set up aliases (as Clinton Gormley explains in the article), and that is not tied to any model integration.

One solution would be to have some set of helper classes / DSL methods in the elasticsearch-extensions or elasticsearch-dsl gem, which would abstract all those acrobatics when setting up the aliases. Then the index_name method in the model would actually point to whatever has been set as the alias, and the responsibilities would be properly divided.

[karmi]

I gave it some more thinking, and now I'm 100% convinced that the right way how to approach would be:

• Write an article for the Elastic blog about this topic, with concrete, working code examples how to achieve that
• Mirror that article in the repository Wiki, so people can find it at Github as well
• Put all the accompanying code into a Gist for easy access

This not only solves the issue itself, but educates users about possibilities and patterns when using the gem. This is actually very similar to the problem posed by #288, and in the ideal scenario we end up with a nice collection of "recipes" for using the libraries, instead of a huge amount of options for this and that particular problem.

[alexlitvak13]

@karmi thanks for your response. I still think we have a need to allow separate read/write aliases. Hopefully, I'm missing something and you can help point that out.

Scenario:
Say we have index N1 and alias A. We reindex the data into N2 and repoint A. Zero downtime, but the reindexing took time. During that time, N1 was live and being updated via the Gem (by customer activity for example). When we repoint A at N2, N2 is already outdated because it never saw those updates.

In addition to alias A, we can create alias AA to point at both N1 and N2. If we then continue reading from A but write to AA, we keep both indexes up-to-date during the reindexing process.

At the moment, I don't see a way to configure the Gem to read from A but write to AA.

[karmi]

At the moment, I don't see a way to configure the Gem to read from A but write to AA.

Yeah, and that might be because you're too focused on something like Article.new(title: "test").save. And thus on something like search_index_name and index_index_name or read_index_name and write_index_name. (Notice how silly the names sound, and that's an inidication we aren't on a right track, in my eyes.)

But in reality, that code shouldn't happen in a reasonably sized application, if only because of executing the actual indexing HTTP request in background. So, in the case where we would have eg. a Sidekiq based indexer module (check out the example application templates), it could store the document into a completely unrelated index.

---

Minor note:

If we then continue reading from A but write to AA, we keep both indexes up-to-date during the reindexing process.

If I'm not missing something obvious, it's not entirely true -- you need to synchronize the indices anyhow (replaying the operations from a queue or something like that), if you want to end up with one, single index. But of course, you might point the A alias to multiple indices, and just keep on growing like that.

[shaneog]

@karmi Has any work or more thought been put into this? It's a definite issue.

[summera]

@karmi I went through this reindexing process using elasticsearch-rails recently. I wrote up a blog post about my process and how to keep things up to date during reindexing. I hope this helps others.

https://summera.github.io/infrastructure/2016/07/04/reindexing-elasticsearch.html

[alexlitvak13]

@summera Nice writeup! We do something very similar but create the new index by pulling from the master datastore (relational db). This has the added benefit of correcting any db-es sync issues that may have accumulated since the prior full reindex.

I'm still disappointed that we need to explicitly handle something that could (should?) have been accounted for at the gem layer and/or with elasticsearch aliases.

[summera]

@alexlitvak13 Thanks!

Cool. How many records do you pull from your master datastore and how long does it take?

[alexlitvak13]

@summera We maintain a couple indexes for a total of about 130M records. It can take a full day to reindex and the bottleneck is entirely in Rails/ActiveRecord object serialization. We've accelerated by running multiple chunks in parallel and can get a full reindex done in several hours.

[prikha]

Hi guys(@sgringwe, @karmi ) , surprisingly, there is so much on reindexing, and no one asks about working time-based indices? Maybe you can point me to the right direction?

it is general practice to have time-based indices like this:

docs-2017-06-04
docs-2017-06-03

and aliases:

docs_read => docs-*
docs_write => docs-2017-06-05

For now it seems like we cannot do this with the repository/model abstraction out of the box . What do you think is the right way here?

1. The requested feature could ease the straight forward usage of the library provided
2. We should have 2 repos, first with stubbed reads, second with stubbed out writes. And manage it in our code.
3. There is a right way i`m missing?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.