Elasticsearch: [CCR] Paused follower does not display in follower stats

[jen-huang]

I created a follower index, kibana_sample_data_logs.

Then I paused the follower index using:

POST /kibana_sample_data_logs/_ccr/pause_follow

Then I attempted to retrieve the stats of the follower index using both:

GET /kibana_sample_data_logs/_ccr/stats

and

GET /_ccr/stats

Neither one lists kibana_sample_data_logs as part of indices array. I resumed the follower using:

POST /kibana_sample_data_logs/_ccr/resume_follow
{}

and then the stats API showed the correct information.

For CCR UI, we need the stats API to:

1. Show paused follower indices
2. Show status for each follower index - whether it is paused or active. Could just be a boolean paused flag.

This will make it so that we can give the user the correct option in the UI (whether to resume or pause follower index based on its status).

cc @elastic/es-ui

[elasticmachine]

Pinging @elastic/es-distributed

[martijnvg]

So the reason that no stats are returned for paused follower indices, is that there are no shard follow tasks when index following is paused and the follow stats api is based on returning stats entries for shard follow tasks.

I think the best way for the UI is to figure out all the follower indices by checking whether the index setting index.xpack.ccr.following_index has been set to true via the cluster state api, then for each follower index check whether index following is active by checking the follow stats api. If there are no shard follow stats entries then index following is paused. Would that work?

[jen-huang]

@martijnvg Thanks for the insight. While using cluster state api would work, the user could potentially have a large number of indices and we will have to look through each index's xpack.ccr.following_index setting. I think the CCR api is missing a key endpoint to "get list of all follower indices and their status". This will make our future UI enhancement work easier, and round out the CCR api UX experience as a whole.

Open for more discussion on this. Let me know if a solution might be feasible for 6.7, otherwise for 6.7 we will investigate the cluster state api route and/or adjust UI scope.

[martijnvg]

@jen-huang I thought a bit more about this and I think we can change the follow stats api to also include follower indices that are paused. These indices would then we returned without any stats entries and would look like this:

{
   "indices": [
       {
            "index": "my_paused_index",
            "shards" : [] 
       },
       {
            "index": "my_active_index",
            "shards" : [
               {
                    ...
               },
               ....
            ] 
       }
   ]
}

Here the my_paused_index here has zero shard stats entries, because it is paused, but it is a follower index managed by ccr. Would this work for the UI? This change shouldn't require a lot of work on the ES side.

@jasontedor @dnhatn Do you think adding follower indices with empty shard entries is a good idea here in order to include paused follower indices in the stats api? This way the UI will then only need to check one api display stats for ccr follower indices (instead of the workaround mentioned above).

[jen-huang]

@martijnvg Yes, having zero shard stats would work for the UI 😄

[sebelga]

@martijnvg Does having 0 shards means that it is paused?

What if ES would provide then a "status" field in that case

{
   "indices": [
       {
            "index": "my_paused_index",
            "status": "paused"
       },
       {
            "index": "my_active_index",
            "status": "running",
            "shards" : [
               {
                    ...
               },
               ....
            ] 
       }
   ]
}

[martijnvg]

Does having 0 shards means that it is paused?

Yes, having 0 follow shard stats entries in the response means a follow index is paused.

What if ES would provide then a "status" field in that case

I think we could add a status field to make the the status of a follower index more clear.

[jen-huang]

Hi @martijnvg, @sebelga has started work on providing the follower index advanced settings UI at the time of creation: https://github.com/elastic/kibana/pull/28267

The advanced settings include max_read_request_operation_count, max_outstanding_read_requests, etc - everything per create follower api documentation: https://www.elastic.co/guide/en/elasticsearch//reference/master/ccr-put-follow.html#_request_body_2

If the user has specified any of these settings, we would like to display that later on in the UI for their reference. However, we don't have any way to read out their settings after creation. Would it be possible to modify the stats api to return this information as well?

[martijnvg]

If the user has specified any of these settings, we would like to display that later on in the UI for their reference. However, we don't have any way to read out their settings after creation.

This should be visible in the cluster state under persistent_tasks if index following has started. For each follow shard there is an entry that contains the value for the advanced setting. So it is kind of duplicated and another place to look inside the cluster state, but it is there.

Would it be possible to modify the stats api to return this information as well?

I think it makes sense to add the used index follow parameters as part of the follow stats api. Then there is a single place all information about for ccr index following can be read.

[martijnvg]

After thinking more about this, I think we should not include the follow index parameters / settings inside of the follow stats api. The follow stats api returns very dynamic information about the replication processes that ccr manages for each follow index shard; which is a different kind of information than the follow index parameters, what the status is of a follower index or which indices are follower indices.

What I think we should do instead, is add another api that returns that kind of information. I think we should add a follow info api. Which returns all the follower indices and for each follower index returns the status (active or paused) and index follow parameters. Then we can leave the follow stats api as is. The structure of the response of the follow info api would then look like this:

{
   "indices": {
        {
            "index": "follower_index1",
            "status": "active",
            "parameters": {
                 "remote_cluster": "eu_cluster",
                  "leader_index": "leader_index1",
                  "max_read_request_operation_count": 1024,
                  ...
             }
        },
        {
           "index": "follower_index2",
           ...
        },
        ...
    }
}

[yaronp68]

@martijnvg The separate list of follower indices API you mention will be very useful for the user IMO
In the GUI we will be able to use it for creating a list of followers with the dynamic information we need without hitting the stats API all the time

[sebelga]

Agreed @martijnvg . A separate API would be better for listing follower indices. Cheers!

[jen-huang]

@martijnvg ++ to that suggestion - a separate API would be perfect and fills a large gap.

[bmcconaghy]

++ to separate API. It would be useful for index management UI as well.