Curator: Curator applies ILM filter and removes actionable items
For usage questions and help
Please create a topic at https://discuss.elastic.co/c/elasticsearch
Perhaps a topic there already has an answer for you!
To submit a bug or report an issue
Expected Behavior
Curator should be able to run snapshot actions on indices that have an ILM policy applied.
Actual Behavior
Curator applies some ILM filter and results in:
No actionable items in list: <class 'curator.exceptions.NoIndices'>
Steps to Reproduce the Problem
- Create some index and attach an ILM policy to it
- Use the following actions:
--
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
actions:
1:
action: snapshot
description: Do some snapshots
options:
repository: folkvangr
name: 'cpu-%Y%m%d%H%M%S'
ignore_unavailable: False
include_global_state: True
partial: False
wait_for_completion: True
skip_repo_fs_check: False
disable_action: False
filters:
- filtertype: pattern
kind: regex
value: '^(cpu|disk|memory).*$'
- Run Curator with the actions above
Specifications
- Version: 5.7.6
- Platform: Ubuntu 18.04 LTS
- Subsystem:
Context (Environment)
It looks to me like indexes with an ILM policy are filtered out. I hope that this is not intentional, because if it is, how else would I backup indexes?
Detailed Description
Check the following logs for more info:
2019-05-13 02:30:42,934 DEBUG curator.cli run:108 Client and logging options validated.
2019-05-13 02:30:42,934 DEBUG curator.cli run:112 default_timeout = 30
2019-05-13 02:30:42,934 DEBUG curator.cli run:116 action_file: .curator/actions.yml
2019-05-13 02:30:42,938 DEBUG curator.cli run:118 action_config: {'actions': {1: {'action': 'snapshot', 'description': "Snapshot indices older than 1 day (based on index creation_date) and younger than 30 days, with the default snapshot name pattern of 'curator-%Y%m%d%H%M%S'. Wait for the snapshot to complete. Do not skip the repository filesystem access check. Use the other options to create the snapshot.", 'options': {'repository': 'folkvangr', 'name': 'cpu-%Y%m%d%H%M%S', 'ignore_unavailable': False, 'include_global_state': True, 'partial': False, 'wait_for_completion': True, 'skip_repo_fs_check': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$'}]}}}
2019-05-13 02:30:42,938 DEBUG curator.validators.SchemaCheck __init__:26 Schema: {'actions': <class 'dict'>}
2019-05-13 02:30:42,938 DEBUG curator.validators.SchemaCheck __init__:27 "Actions File" config: {'actions': {1: {'action': 'snapshot', 'description': "Snapshot indices older than 1 day (based on index creation_date) and younger than 30 days, with the default snapshot name pattern of 'curator-%Y%m%d%H%M%S'. Wait for the snapshot to complete. Do not skip the repository filesystem access check. Use the other options to create the snapshot.", 'options': {'repository': 'folkvangr', 'name': 'cpu-%Y%m%d%H%M%S', 'ignore_unavailable': False, 'include_global_state': True, 'partial': False, 'wait_for_completion': True, 'skip_repo_fs_check': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$'}]}}}
2019-05-13 02:30:42,939 DEBUG curator.validators.SchemaCheck __init__:26 Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']")}
2019-05-13 02:30:42,939 DEBUG curator.validators.SchemaCheck __init__:27 "action type" config: {'action': 'snapshot', 'description': "Snapshot indices older than 1 day (based on index creation_date) and younger than 30 days, with the default snapshot name pattern of 'curator-%Y%m%d%H%M%S'. Wait for the snapshot to complete. Do not skip the repository filesystem access check. Use the other options to create the snapshot.", 'options': {'repository': 'folkvangr', 'name': 'cpu-%Y%m%d%H%M%S', 'ignore_unavailable': False, 'include_global_state': True, 'partial': False, 'wait_for_completion': True, 'skip_repo_fs_check': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$'}]}
2019-05-13 02:30:42,939 DEBUG curator.validators.SchemaCheck __init__:26 Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']"), 'description': Any(<class 'str'>, <class 'str'>, msg=None), 'options': <class 'dict'>, 'filters': <class 'list'>}
2019-05-13 02:30:42,939 DEBUG curator.validators.SchemaCheck __init__:27 "structure" config: {'action': 'snapshot', 'description': "Snapshot indices older than 1 day (based on index creation_date) and younger than 30 days, with the default snapshot name pattern of 'curator-%Y%m%d%H%M%S'. Wait for the snapshot to complete. Do not skip the repository filesystem access check. Use the other options to create the snapshot.", 'options': {'repository': 'folkvangr', 'name': 'cpu-%Y%m%d%H%M%S', 'ignore_unavailable': False, 'include_global_state': True, 'partial': False, 'wait_for_completion': True, 'skip_repo_fs_check': False, 'disable_action': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$'}]}
2019-05-13 02:30:42,942 DEBUG curator.validators.SchemaCheck __init__:26 Schema: {'allow_ilm_indices': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd1266f61e0>, msg=None), msg=None), 'continue_if_exception': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd1266f6378>, msg=None), msg=None), 'disable_action': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd1266f6510>, msg=None), msg=None), 'ignore_empty_list': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd1266f66a8>, msg=None), msg=None), 'timeout_override': Any(Coerce(int, msg=None), None, msg=None), 'repository': Any(<class 'str'>, msg=None), 'name': Any(<class 'str'>, msg=None), 'ignore_unavailable': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd1267542f0>, msg=None), msg=None), 'include_global_state': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd126754400>, msg=None), msg=None), 'partial': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd126754bf8>, msg=None), msg=None), 'wait_for_completion': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd126754b70>, msg=None), msg=None), 'wait_interval': Any(All(Coerce(int, msg=None), Range(min=1, max=30, min_included=True, max_included=True, msg=None), msg=None), None, msg=None), 'max_wait': Any(-1, Coerce(int, msg=None), None, msg=None), 'skip_repo_fs_check': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd126754a60>, msg=None), msg=None)}
2019-05-13 02:30:42,943 DEBUG curator.validators.SchemaCheck __init__:27 "options" config: {'repository': 'folkvangr', 'name': 'cpu-%Y%m%d%H%M%S', 'ignore_unavailable': False, 'include_global_state': True, 'partial': False, 'wait_for_completion': True, 'skip_repo_fs_check': False, 'disable_action': False}
2019-05-13 02:30:42,943 DEBUG curator.validators.SchemaCheck __init__:26 Schema: <function Filters.<locals>.f at 0x7fd12675f2f0>
2019-05-13 02:30:42,943 DEBUG curator.validators.SchemaCheck __init__:27 "filters" config: [{'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$'}]
2019-05-13 02:30:42,943 DEBUG curator.validators.SchemaCheck __init__:26 Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']"), 'kind': Any('prefix', 'suffix', 'timestring', 'regex', msg=None), 'value': Any(<class 'str'>, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7fd12675fc80>, msg=None), msg=None)}
2019-05-13 02:30:42,943 DEBUG curator.validators.SchemaCheck __init__:27 "filter" config: {'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$'}
2019-05-13 02:30:42,943 DEBUG curator.validators.filters f:48 Filter #0: {'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$', 'exclude': False}
2019-05-13 02:30:42,943 DEBUG curator.cli run:121 Full list of actions: {1: {'action': 'snapshot', 'description': "Snapshot indices older than 1 day (based on index creation_date) and younger than 30 days, with the default snapshot name pattern of 'curator-%Y%m%d%H%M%S'. Wait for the snapshot to complete. Do not skip the repository filesystem access check. Use the other options to create the snapshot.", 'options': {'repository': 'folkvangr', 'name': 'cpu-%Y%m%d%H%M%S', 'ignore_unavailable': False, 'include_global_state': True, 'partial': False, 'wait_for_completion': True, 'skip_repo_fs_check': False, 'disable_action': False, 'allow_ilm_indices': False, 'max_wait': -1, 'timeout_override': 21600, 'continue_if_exception': False, 'wait_interval': 9, 'ignore_empty_list': False}, 'filters': [{'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$', 'exclude': False}]}}
2019-05-13 02:30:42,943 DEBUG curator.cli run:126 action_disabled = False
2019-05-13 02:30:42,944 DEBUG curator.cli run:130 continue_if_exception = False
2019-05-13 02:30:42,944 DEBUG curator.cli run:132 timeout_override = 21600
2019-05-13 02:30:42,944 DEBUG curator.cli run:134 ignore_empty_list = False
2019-05-13 02:30:42,944 DEBUG curator.cli run:136 allow_ilm_indices = False
2019-05-13 02:30:42,944 INFO curator.cli run:151 Preparing Action ID: 1, "snapshot"
2019-05-13 02:30:42,944 DEBUG curator.utils get_client:807 kwargs = {'hosts': ['127.0.0.1'], 'port': 9200, 'use_ssl': False, 'ssl_no_validate': False, 'master_only': True, 'http_auth': None, 'client_cert': None, 'client_key': None, 'aws_key': None, 'aws_sign_request': False, 'url_prefix': '', 'aws_secret_key': None, 'certificate': None, 'aws_token': None, 'timeout': 21600}
2019-05-13 02:30:42,947 DEBUG curator.utils get_client:884 Not using "requests_aws4auth" python module to connect.
2019-05-13 02:30:42,952 DEBUG curator.utils check_version:693 Detected Elasticsearch version 7.0.1
2019-05-13 02:30:42,958 DEBUG curator.cli run:166 client is <class 'elasticsearch.client.Elasticsearch'>
2019-05-13 02:30:42,958 INFO curator.cli run:172 Trying Action ID: 1, "snapshot": Snapshot indices older than 1 day (based on index creation_date) and younger than 30 days, with the default snapshot name pattern of 'curator-%Y%m%d%H%M%S'. Wait for the snapshot to complete. Do not skip the repository filesystem access check. Use the other options to create the snapshot.
2019-05-13 02:30:42,959 DEBUG curator.cli process_action:44 Configuration dictionary: {'action': 'snapshot', 'description': "Snapshot indices older than 1 day (based on index creation_date) and younger than 30 days, with the default snapshot name pattern of 'curator-%Y%m%d%H%M%S'. Wait for the snapshot to complete. Do not skip the repository filesystem access check. Use the other options to create the snapshot.", 'options': {'repository': 'folkvangr', 'name': 'cpu-%Y%m%d%H%M%S', 'ignore_unavailable': False, 'include_global_state': True, 'partial': False, 'wait_for_completion': True, 'skip_repo_fs_check': False, 'max_wait': -1, 'wait_interval': 9}, 'filters': [{'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$', 'exclude': False}, {'filtertype': 'ilm'}]}
2019-05-13 02:30:42,959 DEBUG curator.cli process_action:45 kwargs: {'master_timeout': 300, 'dry_run': True}
2019-05-13 02:30:42,959 DEBUG curator.cli process_action:50 opts: {'repository': 'folkvangr', 'name': 'cpu-%Y%m%d%H%M%S', 'ignore_unavailable': False, 'include_global_state': True, 'partial': False, 'wait_for_completion': True, 'skip_repo_fs_check': False, 'max_wait': -1, 'wait_interval': 9}
2019-05-13 02:30:42,959 DEBUG curator.cli process_action:62 Action kwargs: {'repository': 'folkvangr', 'name': 'cpu-%Y%m%d%H%M%S', 'ignore_unavailable': False, 'include_global_state': True, 'partial': False, 'wait_for_completion': True, 'skip_repo_fs_check': False, 'max_wait': -1, 'wait_interval': 9}
2019-05-13 02:30:42,959 DEBUG curator.cli process_action:91 Running "SNAPSHOT"
2019-05-13 02:30:42,959 DEBUG curator.indexlist __get_indices:65 Getting all indices
2019-05-13 02:30:42,965 DEBUG curator.utils get_indices:648 Detected Elasticsearch version 7.0.1
2019-05-13 02:30:42,965 DEBUG curator.utils get_indices:650 All indices: ['encoder-000001', '.monitoring-kibana-7-2019.05.13', 'disk-000001', 'cpu-000001', 'gpu-000001', 'net-000001', 'power-scan-000001', 'sfp-000001', 'pastri-000001', '.monitoring-kibana-7-2019.05.11', 'beacon-000001', 'memory-000001', '.monitoring-kibana-7-2019.05.12', 'edfa-000001', '.monitoring-es-7-2019.05.11', '.kibana_1', '.monitoring-es-7-2019.05.12', 'ps-000001', '.monitoring-es-7-2019.05.13', '.kibana_task_manager']
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for encoder-000001
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for .monitoring-kibana-7-2019.05.13
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for disk-000001
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for cpu-000001
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for gpu-000001
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for net-000001
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for power-scan-000001
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for sfp-000001
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for pastri-000001
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for .monitoring-kibana-7-2019.05.11
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for beacon-000001
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for memory-000001
2019-05-13 02:30:42,965 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for .monitoring-kibana-7-2019.05.12
2019-05-13 02:30:42,966 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for edfa-000001
2019-05-13 02:30:42,966 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for .monitoring-es-7-2019.05.11
2019-05-13 02:30:42,966 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for .kibana_1
2019-05-13 02:30:42,966 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for .monitoring-es-7-2019.05.12
2019-05-13 02:30:42,966 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for ps-000001
2019-05-13 02:30:42,966 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for .monitoring-es-7-2019.05.13
2019-05-13 02:30:42,966 DEBUG curator.indexlist __build_index_info:80 Building preliminary index metadata for .kibana_task_manager
2019-05-13 02:30:42,966 DEBUG curator.indexlist _get_metadata:177 Getting index metadata
2019-05-13 02:30:42,966 DEBUG curator.indexlist empty_list_check:226 Checking for empty list
2019-05-13 02:30:42,995 DEBUG curator.indexlist _get_index_stats:117 Getting index stats
2019-05-13 02:30:42,995 DEBUG curator.indexlist empty_list_check:226 Checking for empty list
2019-05-13 02:30:42,995 DEBUG curator.indexlist working_list:237 Generating working list of indices
2019-05-13 02:30:42,995 DEBUG curator.indexlist working_list:237 Generating working list of indices
2019-05-13 02:30:43,002 DEBUG curator.indexlist iterate_over_stats:126 Index: gpu-000001 Size: 3.3MB Docs: 22750
2019-05-13 02:30:43,002 DEBUG curator.indexlist iterate_over_stats:126 Index: .kibana_task_manager Size: 59.2KB Docs: 4
2019-05-13 02:30:43,002 DEBUG curator.indexlist iterate_over_stats:126 Index: net-000001 Size: 15.9MB Docs: 91118
2019-05-13 02:30:43,002 DEBUG curator.indexlist iterate_over_stats:126 Index: edfa-000001 Size: 1.1KB Docs: 0
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2019.05.11 Size: 205.8MB Docs: 255676
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: encoder-000001 Size: 246.7MB Docs: 4706260
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: .kibana_1 Size: 77.1KB Docs: 14
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: sfp-000001 Size: 1.1KB Docs: 0
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: pastri-000001 Size: 1.1KB Docs: 0
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-kibana-7-2019.05.13 Size: 580.6KB Docs: 1808
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-kibana-7-2019.05.12 Size: 4.3MB Docs: 17278
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-kibana-7-2019.05.11 Size: 2.9MB Docs: 11102
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2019.05.12 Size: 298.0MB Docs: 436918
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: memory-000001 Size: 3.7MB Docs: 22920
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: .monitoring-es-7-2019.05.13 Size: 39.2MB Docs: 49068
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: ps-000001 Size: 3.3MB Docs: 22686
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: power-scan-000001 Size: 1.1KB Docs: 0
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: cpu-000001 Size: 12.5MB Docs: 23236
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: beacon-000001 Size: 1.1KB Docs: 0
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_over_stats:126 Index: disk-000001 Size: 3.3MB Docs: 23144
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_filters:1214 Iterating over a list of filters
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_filters:1220 All filters: [{'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$', 'exclude': False}, {'filtertype': 'ilm'}]
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_filters:1222 Top of the loop: ['encoder-000001', '.monitoring-kibana-7-2019.05.13', 'disk-000001', 'cpu-000001', 'gpu-000001', 'net-000001', 'power-scan-000001', 'sfp-000001', 'pastri-000001', '.monitoring-kibana-7-2019.05.11', 'beacon-000001', 'memory-000001', '.monitoring-kibana-7-2019.05.12', 'edfa-000001', '.monitoring-es-7-2019.05.11', '.kibana_1', '.monitoring-es-7-2019.05.12', 'ps-000001', '.monitoring-es-7-2019.05.13', '.kibana_task_manager']
2019-05-13 02:30:43,003 DEBUG curator.indexlist iterate_filters:1223 Un-parsed filter args: {'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$', 'exclude': False}
2019-05-13 02:30:43,005 DEBUG curator.validators.SchemaCheck __init__:26 Schema: {'aliases': Any(<class 'list'>, <class 'str'>, msg=None), 'allocation_type': Any(<class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, <class 'str'>, msg=None), 'date_from_format': Any(None, <class 'str'>, msg=None), 'date_to': Any(None, <class 'str'>, msg=None), 'date_to_format': Any(None, <class 'str'>, msg=None), 'direction': Any(<class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'field': Any(None, <class 'str'>, msg=None), 'intersect': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'key': Any(<class 'str'>, msg=None), 'kind': Any(<class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, msg=None), 'period_type': Any(<class 'str'>, msg=None), 'reverse': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(<class 'str'>, msg=None), 'source': Any(<class 'str'>, msg=None), 'state': Any(<class 'str'>, msg=None), 'stats_result': Any(None, <class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None), 'threshold_behavior': Any(<class 'str'>, msg=None), 'unit': Any(<class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'use_age': <function Boolean at 0x7fd12672dea0>, 'value': Any(<class 'int'>, <class 'float'>, <class 'bool'>, <class 'str'>, msg=None), 'week_starts_on': Any(None, <class 'str'>, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']")}
2019-05-13 02:30:43,005 DEBUG curator.validators.SchemaCheck __init__:27 "filter" config: {'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$', 'exclude': False}
2019-05-13 02:30:43,005 DEBUG curator.indexlist iterate_filters:1230 Parsed filter args: {'filtertype': 'pattern', 'kind': 'regex', 'value': '^(cpu|disk|memory).*$', 'exclude': False}
2019-05-13 02:30:43,005 DEBUG curator.indexlist iterate_filters:1239 Filter args: {'kind': 'regex', 'value': '^(cpu|disk|memory).*$', 'exclude': False}
2019-05-13 02:30:43,005 DEBUG curator.indexlist iterate_filters:1240 Pre-instance: ['encoder-000001', '.monitoring-kibana-7-2019.05.13', 'disk-000001', 'cpu-000001', 'gpu-000001', 'net-000001', 'power-scan-000001', 'sfp-000001', 'pastri-000001', '.monitoring-kibana-7-2019.05.11', 'beacon-000001', 'memory-000001', '.monitoring-kibana-7-2019.05.12', 'edfa-000001', '.monitoring-es-7-2019.05.11', '.kibana_1', '.monitoring-es-7-2019.05.12', 'ps-000001', '.monitoring-es-7-2019.05.13', '.kibana_task_manager']
2019-05-13 02:30:43,005 DEBUG curator.indexlist filter_by_regex:429 Filtering indices by regex
2019-05-13 02:30:43,005 DEBUG curator.indexlist empty_list_check:226 Checking for empty list
2019-05-13 02:30:43,005 DEBUG curator.indexlist working_list:237 Generating working list of indices
2019-05-13 02:30:43,006 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: encoder-000001
2019-05-13 02:30:43,006 DEBUG curator.indexlist __not_actionable:39 Index encoder-000001 is not actionable, removing from list.
2019-05-13 02:30:43,006 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: .monitoring-kibana-7-2019.05.13
2019-05-13 02:30:43,006 DEBUG curator.indexlist __not_actionable:39 Index .monitoring-kibana-7-2019.05.13 is not actionable, removing from list.
2019-05-13 02:30:43,006 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: disk-000001
2019-05-13 02:30:43,006 DEBUG curator.indexlist __actionable:35 Index disk-000001 is actionable and remains in the list.
2019-05-13 02:30:43,006 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: cpu-000001
2019-05-13 02:30:43,006 DEBUG curator.indexlist __actionable:35 Index cpu-000001 is actionable and remains in the list.
2019-05-13 02:30:43,006 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: gpu-000001
2019-05-13 02:30:43,006 DEBUG curator.indexlist __not_actionable:39 Index gpu-000001 is not actionable, removing from list.
2019-05-13 02:30:43,006 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: net-000001
2019-05-13 02:30:43,006 DEBUG curator.indexlist __not_actionable:39 Index net-000001 is not actionable, removing from list.
2019-05-13 02:30:43,006 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: power-scan-000001
2019-05-13 02:30:43,006 DEBUG curator.indexlist __not_actionable:39 Index power-scan-000001 is not actionable, removing from list.
2019-05-13 02:30:43,006 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: sfp-000001
2019-05-13 02:30:43,006 DEBUG curator.indexlist __not_actionable:39 Index sfp-000001 is not actionable, removing from list.
2019-05-13 02:30:43,006 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: pastri-000001
2019-05-13 02:30:43,006 DEBUG curator.indexlist __not_actionable:39 Index pastri-000001 is not actionable, removing from list.
2019-05-13 02:30:43,006 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: .monitoring-kibana-7-2019.05.11
2019-05-13 02:30:43,006 DEBUG curator.indexlist __not_actionable:39 Index .monitoring-kibana-7-2019.05.11 is not actionable, removing from list.
2019-05-13 02:30:43,006 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: beacon-000001
2019-05-13 02:30:43,007 DEBUG curator.indexlist __not_actionable:39 Index beacon-000001 is not actionable, removing from list.
2019-05-13 02:30:43,007 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: memory-000001
2019-05-13 02:30:43,007 DEBUG curator.indexlist __actionable:35 Index memory-000001 is actionable and remains in the list.
2019-05-13 02:30:43,007 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: .monitoring-kibana-7-2019.05.12
2019-05-13 02:30:43,007 DEBUG curator.indexlist __not_actionable:39 Index .monitoring-kibana-7-2019.05.12 is not actionable, removing from list.
2019-05-13 02:30:43,007 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: edfa-000001
2019-05-13 02:30:43,007 DEBUG curator.indexlist __not_actionable:39 Index edfa-000001 is not actionable, removing from list.
2019-05-13 02:30:43,007 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: .monitoring-es-7-2019.05.11
2019-05-13 02:30:43,007 DEBUG curator.indexlist __not_actionable:39 Index .monitoring-es-7-2019.05.11 is not actionable, removing from list.
2019-05-13 02:30:43,007 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: .kibana_1
2019-05-13 02:30:43,007 DEBUG curator.indexlist __not_actionable:39 Index .kibana_1 is not actionable, removing from list.
2019-05-13 02:30:43,007 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: .monitoring-es-7-2019.05.12
2019-05-13 02:30:43,007 DEBUG curator.indexlist __not_actionable:39 Index .monitoring-es-7-2019.05.12 is not actionable, removing from list.
2019-05-13 02:30:43,007 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: ps-000001
2019-05-13 02:30:43,007 DEBUG curator.indexlist __not_actionable:39 Index ps-000001 is not actionable, removing from list.
2019-05-13 02:30:43,007 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: .monitoring-es-7-2019.05.13
2019-05-13 02:30:43,007 DEBUG curator.indexlist __not_actionable:39 Index .monitoring-es-7-2019.05.13 is not actionable, removing from list.
2019-05-13 02:30:43,007 DEBUG curator.indexlist filter_by_regex:450 Filter by regex: Index: .kibana_task_manager
2019-05-13 02:30:43,007 DEBUG curator.indexlist __not_actionable:39 Index .kibana_task_manager is not actionable, removing from list.
2019-05-13 02:30:43,007 DEBUG curator.indexlist iterate_filters:1242 Post-instance: ['disk-000001', 'cpu-000001', 'memory-000001']
2019-05-13 02:30:43,007 DEBUG curator.indexlist iterate_filters:1222 Top of the loop: ['disk-000001', 'cpu-000001', 'memory-000001']
2019-05-13 02:30:43,007 DEBUG curator.indexlist iterate_filters:1223 Un-parsed filter args: {'filtertype': 'ilm'}
2019-05-13 02:30:43,009 DEBUG curator.validators.SchemaCheck __init__:26 Schema: {'aliases': Any(<class 'list'>, <class 'str'>, msg=None), 'allocation_type': Any(<class 'str'>, msg=None), 'count': Coerce(int, msg=None), 'date_from': Any(None, <class 'str'>, msg=None), 'date_from_format': Any(None, <class 'str'>, msg=None), 'date_to': Any(None, <class 'str'>, msg=None), 'date_to_format': Any(None, <class 'str'>, msg=None), 'direction': Any(<class 'str'>, msg=None), 'disk_space': <class 'float'>, 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'field': Any(None, <class 'str'>, msg=None), 'intersect': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'key': Any(<class 'str'>, msg=None), 'kind': Any(<class 'str'>, msg=None), 'max_num_segments': Coerce(int, msg=None), 'number_of_shards': Coerce(int, msg=None), 'pattern': Any(<class 'str'>, msg=None), 'period_type': Any(<class 'str'>, msg=None), 'reverse': Any(None, <class 'bool'>, <class 'int'>, <class 'str'>, msg=None), 'range_from': Coerce(int, msg=None), 'range_to': Coerce(int, msg=None), 'shard_filter_behavior': Any(<class 'str'>, msg=None), 'source': Any(<class 'str'>, msg=None), 'state': Any(<class 'str'>, msg=None), 'stats_result': Any(None, <class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None), 'threshold_behavior': Any(<class 'str'>, msg=None), 'unit': Any(<class 'str'>, msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'use_age': <function Boolean at 0x7fd126723b70>, 'value': Any(<class 'int'>, <class 'float'>, <class 'bool'>, <class 'str'>, msg=None), 'week_starts_on': Any(None, <class 'str'>, msg=None), 'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'empty', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'shards', 'space', 'state']")}
2019-05-13 02:30:43,009 DEBUG curator.validators.SchemaCheck __init__:27 "filter" config: {'filtertype': 'ilm'}
2019-05-13 02:30:43,009 DEBUG curator.indexlist iterate_filters:1230 Parsed filter args: {'filtertype': 'ilm'}
2019-05-13 02:30:43,009 DEBUG curator.indexlist filter_ilm:1176 Filtering indices with index.lifecycle.name
2019-05-13 02:30:43,012 DEBUG curator.indexlist __not_actionable:39 Index memory-000001 is not actionable, removing from list.
2019-05-13 02:30:43,012 DEBUG curator.indexlist __excludify:58 Removed from actionable list: memory-000001 has index.lifecycle.name default
2019-05-13 02:30:43,012 DEBUG curator.indexlist __not_actionable:39 Index disk-000001 is not actionable, removing from list.
2019-05-13 02:30:43,012 DEBUG curator.indexlist __excludify:58 Removed from actionable list: disk-000001 has index.lifecycle.name default
2019-05-13 02:30:43,012 DEBUG curator.indexlist __not_actionable:39 Index cpu-000001 is not actionable, removing from list.
2019-05-13 02:30:43,012 DEBUG curator.indexlist __excludify:58 Removed from actionable list: cpu-000001 has index.lifecycle.name default
2019-05-13 02:30:43,012 DEBUG curator.indexlist empty_list_check:226 Checking for empty list
2019-05-13 02:30:43,012 ERROR curator.cli run:185 Unable to complete action "snapshot". No actionable items in list: <class 'curator.exceptions.NoIndices'>
rolandjitsu
👍1
All 10 comments
Add the as-yet-undocumented option:
allow_ilm_indices: true
to permit Curator to operate on ILM managed indices.
untergeek
on 13 May 2019
👀3
👍2
I just found https://github.com/elastic/curator/issues/1207 (and the option in https://github.com/elastic/curator/pull/1218) and I'm worried if I enable this option I might end up with some other problems.
Is snapshotting and using ILM safe?
rolandjitsu
on 13 May 2019
My logic to allow snapshotting and ignore ILM is probably in need of some tweaking.
ILM should be safe to use in conjunction with external (i.e. Curator) snapshot API calls, since currently, ILM is snapshot agnostic. It should not cause issues.
A separate snapshot lifecycle management feature is coming in a future release of Elasticsearch. As with ILM, it will be available to all with at least a Basic license.
untergeek
on 13 May 2019
👍1
Yes, probably allowing snapshotting by default might be alright as it does not alter the index in anyway (I think).
I saw the issue on SLM and it looks quite nice. I'm looking forward to it and I'm happy to hear that it comes with the basic license.
rolandjitsu
on 13 May 2019
Don't quote me on the Basic license, but I _presume_ it will be so. I certainly have no insider insight into how licensing will be handled when the feature is fully released.
untergeek
on 13 May 2019
Got it. I do hope, though, that it will 馃槃
rolandjitsu
on 13 May 2019
Gonna close this as it's not as much an issue as it is a missing doc for existing option.
rolandjitsu
on 13 May 2019
Add the as-yet-undocumented option:
allow_ilm_indices: trueto permit Curator to operate on ILM managed indices.
where we should add this field ?
is it on curator configuaration or in action file configuration
vvvk-gh
on 17 Oct 2019
See here, where鈥攍ike above鈥攊t says the option must be set to true, So it goes in the options block of each action. The blocks are action, description, options, and filters.
untergeek
on 17 Oct 2019
👍1
See here, where鈥攍ike above鈥攊t says the
optionmust be set totrue, So it goes in theoptionsblock of each action. The blocks areaction,description,options, andfilters.
Thank you @untergeek .
vvvk-gh
on 17 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings
Related issues
eamanu
路
9Comments
zumo64
路
4Comments
flamaniac
路
9Comments
exklamationmark
路
7Comments
tiempososcuros
路
8Comments
Most helpful comment
Add the as-yet-undocumented option:
to permit Curator to operate on ILM managed indices.