Circuitverse: Implement Elastic Search

[Amansingla97]

Currently, CircuitVerse is using Pg_search's full-text search.As the database grows we ultimately have to shift to elastic search.pgsearch would be too slow to process all the text data. Postgres full-text search is pretty solid for straightforward use cases, but there are a number of reasons why Lucene (and thus Solr and ElasticSearch) is superior both in terms of performance and functionality.
I was going through elastic search and noted some points which might be useful in implementation.

Step 1- ES

• Setting up ES instance
• Defining Schema for documents
• User
• Group
• Project
• Methods in Ruby to add / update / delete documents in ES Instance.

Step 2- ES - Ruby App Integration

• Adding handlers for add/update/delete operations on Users, Groups and Projects, which call the methods to do the corresponding operation on documents in ES Instance.

Step 3- Search Implementation

• Method in Ruby app to get query the ES Instance and get back the results.
• Arguments: type of documents, number of documents, tags

Step 4- UI

• Uses the search method above, to get the search results and render them on the frontend.
• Add UI to filter search results based on the type of document.

Some useful links-

Mapping
Elastic search on rails
search engine optimisation

Advantages to ElasticSearch

->Scale your indexing and search load separately from your regular database load.
->More flexible term analysis for things like accent normalizing, linguistic stemming, N-grams, markup removal… Other cool features like spellcheck, "rich content" (e.g., PDF and Word) extraction…
->Lucene can do everything on the Postgres full-text search can do.
->Much better and faster term relevancy ranking, efficiently customizable at search time.
->Probably faster search performance for common terms or complicated queries.
->Probably more efficient indexing performance than Postgres.
->Better tolerance for change in your data model by decoupling indexing from your primary data store

Are you working on this? (Yes/No)
No

[ankitkataria]

I'll start looking into this. Usually there are few shortcomings of elastic search that I encountered when I last worked on it, for eg: the resource usage is large(but can be brought down), handling indexes might be tricky with the updation and addition of records, and I noticed that it used to crash during high loads. I'll experiment with it once.

[shubhankarsharma00]

I started looking into this earlier this month and I also noticed the same shortcomings which @ankitkataria has talked about but the advantages were huge as @Amansingla97 mentioned. I was also looking into chewy which dealt with most of the issues @ankitkataria mentioned (but I have no idea whether it can solve the crash issue) but the gem currently doesn't have support for Elastic search 6+ and The EOL for Elasticsearch 5 has passed(11/03/2018).

[Amansingla97]

If you're interested in Pg_search vs. Lucene, why not both? Check out the ZomboDB extension for Pg_search, which integrates Elasticsearch as a first-class index type. Still a fairly early project but it looks really promising to me.

Zombodb

[github-actions[bot]]

Stale issue message