Elasticsearch: Document how to detect and avoid duplicate data?

[abibell]

Hi,

This is about documentation probably in tips and tricks section. Sometimes there could be duplicates in documents. Documentation about

1. How do we detect duplicates (pre and post index)
2. How to avoid/remove them?
3. Is it possible to have a document in more than one type? {"title": "Doc title", "_type": ["type1", "type2"]}

I used document content (fields) to generate a Hash and stored it with document. Used the computed hash to detect dupplicates and avoided adding them again.

[clintongormley]

Hi @abibell

Sorry it has taken a while to get to this. It sounds like you're using a good approach. At the end of the day, there is only one unique field in Elasticsearch: the _id. Content deduplication is a big and complex subject, and probably worthy of several blog posts.

[ChrisMagnuson]

I would also like to see a documented strategy recommending what to do to avoid duplicate data. Is there somewhere else we should be submitting a request to create documentation on this topic?

[GelLiNN]

For my use case I have unique IDs for my individual JSON documents in the _source field. Running a GET request on my indices I can retrieve the _id elasticsearch associates with it, and then a following PUT request on that _id to overwrite it, or create a new doc if not found.

Cheers

[alexander-marquardt]

I have written a blog post on how to detect and remove duplicate documents here: https://alexmarquardt.com/2018/07/23/deduplicating-documents-in-elasticsearch/