Curator: Add support for Shrink API

Created on 10 Apr 2017  路  2Comments  路  Source: elastic/curator

https://www.elastic.co/guide/en/elasticsearch/reference/5.3/indices-shrink-index.html

Depends on #656 which will bring the change_settings action (or something like that name)

The action file for a shrink will be a bit lengthy and complicated. Here's an example of what it _may_ wind up looking like, if the actions required were all broken out as separate actions:

actions:
  1:
    action: allocation
    description: >- 
      Route all shards of indices onto the named node in preparation
      for shrinkage
    options:
      key: _name
      value: target_node_name
      allocation_type: require
      wait_for_completion: True
    filters:
    - filtertype: ...
  2:
    action: change_settings
    description: >-
      Change index settings to read-only for the selected indices
    options:
      settings:
        index.block.write: True
    filters:
    - filtertype: ...
  3:
    action: shrink
    description: >-
      Shrink the selected indices
    options:
      index: NEW_INDEX_NAME
      extra_settings:
        settings:
          index.number_of_replicas: 1
          index.number_of_shards: 1
          index.codec: best_compression 
        aliases:
          my_alias:
    filters:
    - filtertype: ...
  4: 
    action: change_settings   
      Change index settings to read-write for the selected indices
    options:
      settings:
        index.block.write: False
    filters:
    - filtertype: ...
  5:
    action: allocation
    description: >- 
      Remove or change routing for shards of indices so they migrate off the shrink node
    options:
      key: _name
      value: 
      allocation_type: require
      wait_for_completion: True
    filters:
    - filtertype: ...

It makes sense for this to all be encapsulated in a single shrink action, so that shard routing multiple indices doesn't all happen at once to a single node ( 馃挘 ). In the end, the shrink action might look like this:

actions:
  1:
    action: shrink
    description: >-
      Shrink the selected indices on node FOO. 
      Re-route them (after shrinking) with the key/value/allocation_type settings.
    options:
      index: NEW_INDEX_NAME
      wait_for_completion: True
      shrink_node: FOO
      key: _name
      value: target_node_name
      allocation_type: require
      extra_settings:
        settings:
          index.number_of_replicas: 1
          index.number_of_shards: 1
          index.codec: best_compression 
        aliases:
          my_alias: 
    filters:
    - filtertype: ...

Of course, testing and the like will have to be done to hopefully see how much disk space is on FOO, and that number_of_shards is an acceptable factor for the provided indices, etc.

Most helpful comment

It would be cool to have the shrink_node picked dynamically by Curator optionally.
Then for a set of indices that need shrinking, different nodes could be used for each index to be route shards to to spread the impact of clumping shards to one node to the various nodes perhaps. Either by a round robin or a pre-check to see which node has the most available resources to do so or whatever makes sense.

All 2 comments

It would be cool to have the shrink_node picked dynamically by Curator optionally.
Then for a set of indices that need shrinking, different nodes could be used for each index to be route shards to to spread the impact of clumping shards to one node to the various nodes perhaps. Either by a round robin or a pre-check to see which node has the most available resources to do so or whatever makes sense.

Looking forward to this. Will close the loop on being able to do a complete roll-over pattern as described here: https://www.elastic.co/blog/managing-time-based-indices-efficiently with Curator only.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

exklamationmark picture exklamationmark  路  7Comments

myoung34 picture myoung34  路  7Comments

zumo64 picture zumo64  路  4Comments

ep4sh picture ep4sh  路  3Comments

jordansissel picture jordansissel  路  4Comments