Elasticsearch: Show force merge/optimize progress

[markwalkom]

Are we able to expose this somehow, maybe pending tasks?
At the moment it just runs until finished, but I'm hoping there may be a way to figure out some kind of progress percentage, even a rough one.

I realise this is all a little vague, but I don't really know if it's possible, just that it'd be nice to have :p

[clintongormley]

If it can be done, it'll be via the task management API. Closing in favour of https://github.com/elastic/elasticsearch/issues/15117

[imotov]

I think we dropped this issue on the floor at the end of the task management task. Now that the task management framework is in the place, should we revisit this issue?

[oleg-andreyev]

@imotov I'm not an expert in elasticsearch internal things, but here are my though as consumers:

• force merge action should have wait_for_completion flag as reindex action, if it is possible, this way I as consumer would send simple http request with wait_for_completion=false and won't wait for completion, because I could monitor force merge from _tasks endpoint
• also it would be great to know if force merge is cancelable? #17094

[elasticmachine]

Pinging @elastic/es-distributed

[bleskes]

@jpountz is there any way to track progress of a user driven forced merge?

[jpountz]

Not everything is exposed publicly today, but we could expose information about ongoing merges. Giving progress information is more complex due to the fact that Lucene doesn't merge document by document but data-structure by data-structure, eg. the inverted index of _id, then the inverted index of foo then doc values of seq_no, then doc values of foo, etc.

[redlus]

I understand this is a bit more complex than tracking the progress of other tasks in the cluster, but even showing the number of indices which already finished (in a multi-index force-merge) and the number of data-structures (or doc_values) already processed would be nice. Is there any plan to add this functionality?

Thanks

[cdenneen]

using the tasks api to see that a forcemerge might still be running doesn't really tell us anything as far as % completion to give estimation... when this is combined with something like curator to forcemerge a bunch of indexes that meet criteria it would be useful to know when these will end so they don't step on each other or cause other contention issues.

[SpencerLN]

The ability to at least understand what index is currently merging would be very useful. We have had users accidentally start of a force merge for an index actively being written to and there was no apparent way to identify which index was being operated on by a given task.

[pedrosk]

Since some large indices with many segments may take for a long time to force merge, it would be beneficial to:
1) see the task existing in a LIST (!) and being able to see its status (eg. in progress, failed, etc)

I feel it is essential to be able to know what is happening with forge merges that take extended amount of time.

[tanandy]

Hi, thats something we need too

[phalverson]

Even something as high level as whether the merge is running or complete would be huge. I don't need to have a play-by-play on what's going on in Lucene, I just need to know when I can start indexing documents again. Please address this soon.

[Ankithas93]

I am able to see merge: current by running below command:
curl -XGET 'http://els2002:9200/_stats?pretty'

  "merges" : {
    "current" : 0,

Is this the one to monitor to check the current merge in progress?
I am running 7.8.0