Elasticsearch: Deleted documents show up in completion suggester

[ankitkr]

Indexed the docs.
Used suggester
Deleted one doc
Used suggester (deleted doc still shows)
Used optimize
Used suggester (deleted doc still shows)
Used refresh
Used suggester (deleted doc still shows)

If the number of docs is less then everything works fine but when the number increases, the completion suggester shows false result. The test case can be found at
https://gist.github.com/ankitkr/e7776419b0e8ff5334af

https://gist.github.com/missinglink/01c7db8bb09f2f935574

[areek]

This is a known issue for the current completion suggester. There has been progress made to fix this, along with other enhancements/imporvements, but the support for the new completion suggester implementation in ES is still not in place. (see https://github.com/elasticsearch/elasticsearch/pull/7353, https://github.com/elasticsearch/elasticsearch/pull/7650)

[s1monw]

it's not a bug it's a feature at this point. We are working on fixes - closing.

[tvdavid]

Should the status of this bug perhaps be reflected more clearly in the completion suggester's documentation? It currently reads The suggest data structure might not reflect deletes on documents immediately. You may need to do an Optimize for that, while in reality it's very easy to get deleted documents stuck in the index with no way at all to purge them out.

[cari-cm]

How can you say it's a feature and then say you are working on fixes? Admit it's a bug. It would be really nice if there was a fix to this BUG prior to 2.0!

[gitazem]

What situation is on this bug ? How can i solve it ? I am trying many workarounds -- and nothing. this is still don't work "_optimize?only_expunge_deletes=true&max_num_segments=1&force=true&refresh=true"
any suggestions ?

[clintongormley]

@gitazem the new completion suggester coming in 2.1 solves this problem

[chape]

@clintongormley Have 2.1.1 fixed it problem yet? I don't see it in 2.1.0&2.1.1 release notes.

[clintongormley]

It was bumped to 2.2.0, but now we've just bumped it to 3.0 because it has breaking changes, unfortunately.

[chape]

@clintongormley thank you for response

[drush]

@clintongormley Can you re-open this issue as a bug, since it is in fact OPEN, and a BUG? kthxbye

[majksner]

This issue is still actual. I'm using 2.2.1

[s1monw]

@drush it's not a bug always advertised as such
@majksner it will not come until 5.0

[drush]

It is absolutely a bug @s1monw Per your docs on completion suggesters:

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters-completion.html
"NOTE: The suggest data structure might not reflect deletes on documents immediately. You may need to do an Optimize for that. You can call optimize with the only_expunge_deletes=true to only target deletions for merging."

DELETES do NOT work on suggesters even with this optimize operation. This suggestion does not in-fact solve the problem, so the bug is that it is NOT possible to delete documents without destroying the entire index.

If you are suggesting that you support a DELETE operation on suggester indexes but that the DELETE operation is never expected to work, then you need to demote this entire feature to 'Experimental' or 'Alpha' rather than a production capability.

The documentation is lacking at best and deceiving at worst. I would suggest re-evaluating this issue for proper prioritization.

[s1monw]

@drush I take these complaints serious so I added a test for your that makes sure the feature works as documented. (https://github.com/elastic/elasticsearch/commit/5fa543867f8cbd0ffde504f495d33cadc8c335ee)

The downside of the implementation (which me and @mikemccand basically wrote years ago) is that it's a very optimized data-structure and the enhancements that we are shipping in the next major version will resolve this issue.

Believe me I see a lot of people that try to be loud on issues like this just like you try to. That's fine, I can't change that. What I can change is to let you know that it doesn't help, it is counter productive. You don't like the progress over perfection approach we are practicing! This feature has already as it is provided huge improvements do large scale suggest applications, it's not perfect as documented.
We are shipping the improvement that you may need in the next major, back-porting is not an option.

Again, you may be heard in your environment when you are loud, in this environment you may just be shown wrong and we move on.

[s1monw]

@drush Just FYI I tried to clarify the documentation and added a possible caveat to the documentation.