Curator: command line options

[geek876]

Hi,
Can curator run with all options from command line only ? I have a use-case whereby I can't have config.yml or various action yml (e.g create_indices.yml) files and need to run it completely from command line. Is this possible ?

[untergeek]

Not at this time, though I'm trying to create singleton cli bits (i.e. would only delete_indices, or delete_snapshots).

[untergeek]

@geek876

I'm curious what use case prohibits you from using a configuration file of any kind. Can you elaborate?

[geek876]

Sure. So we have the ES Cluster within AWS and want to Orchestrate curator jobs from within Jenkins so having a command line would help as we could then simply do it via a 'shell script build step' via Jenkins. This will keep the jenkins job very light weight with no dependency on any repository etc to get the config files.

[untergeek]

If this is what you're trying to do, I recommend using Curator 3.x in the meanwhile. It can be installed via pip by doing pip install -U elasticsearch-curator==3.5.1 (the -U guarantees that whatever is there is replaced by the version after the ==).

Can you tell me specifically which actions are crucial to your operation? I can perhaps prioritize those.

[geek876]

I am just using a shell script now which works but if you could please prioritize snapshot and index deletion actions than that would be much appreciated. Thanks.

[FGRibreau]

This is indeed a serious regression compared to 3.x :+1:

[untergeek]

I'm going to disagree that it's a regression (I had to maintain that code). I will agree that there are use cases that the current, file-based configuration does not make easy. But for each of those use cases you can find, I can see several more that are possible with this format that would have been impossible with the 3.x code base. Inconvenience? Yes. Regression? Not from my perspective.

[untergeek]

So, what if there were a command that ran like this?

$ curator_delete_indices --dry-run --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":13},{"filtertype":"pattern","kind":"prefix","value":"logstash"}]'

2016-11-02 12:58:35,195 INFO      DRY-RUN MODE.  No changes will be made.
2016-11-02 12:58:35,195 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2016-11-02 12:58:35,195 INFO      DRY-RUN: delete_indices: logstash-2016.10.19 (CLOSED) with arguments: {}
2016-11-02 12:58:35,195 INFO      DRY-RUN: delete_indices: logstash-2016.10.20 with arguments: {}

The full range of filters for a given singleton operation becomes available, so long as you encapsulate it as JSON within single quotes.

At present, it requires that you either:

1. Use --config to point to an actual config (or use ~/.curator/curator.yml), or
2. Use --config /dev/null to use all of the defaults: logging is INFO level out to the console, Elasticsearch is at 127.0.0.1, etc.

I will investigate overriding options with config flags.

[untergeek]

Curator 4.2.1 is out, and it has the curator_cli endpoint (or actual binary, if you use that package).

It should look somewhat familiar to users of Curator 3.x:

» curator_cli
Usage: curator_cli [OPTIONS] COMMAND [ARGS]...

Options:
  --config PATH       Path to configuration file. Default:
                      ~/.curator/curator.yml
  --host TEXT         Elasticsearch host.
  --url_prefix TEXT   Elasticsearch http url prefix.
  --port TEXT         Elasticsearch port.
  --use_ssl           Connect to Elasticsearch through SSL.
  --certificate TEXT  Path to certificate to use for SSL validation.
  --client-cert TEXT  Path to file containing SSL certificate for client auth.
  --client-key TEXT   Path to file containing SSL key for client auth.
  --ssl-no-validate   Do not validate SSL certificate
  --http_auth TEXT    Use Basic Authentication ex: user:pass
  --timeout INTEGER   Connection timeout in seconds.
  --master-only       Only operate on elected master node.
  --dry-run           Do not perform any changes.
  --loglevel TEXT     Log level
  --logfile TEXT      log file
  --logformat TEXT    Log output format [default|logstash|json].
  --version           Show the version and exit.
  --help              Show this message and exit.

Commands:
  allocation        Shard Routing Allocation
  close             Close indices
  delete_indices    Delete indices
  delete_snapshots  Delete snapshots
  forcemerge        forceMerge index/shard segments
  open              Open indices
  replicas          Change replica count
  show_indices      Show indices
  show_snapshots    Show snapshots
  snapshot          Snapshot indices

Learn more about it here.

[mikej1688]

I am using curator 4.2.6. I'd like to run a command line, like
curator_cli --host "host1,host2,host3" ...
Can I do this way for multiple hosts? If not, what's the correct way? I only want to run the singleton command line for curator.

[untergeek]

You can't specify multiple hosts with a --host directive, as it only reads in a string and not a list (array).

You _can_ use a YAML configuration file with the singleton command-line, however, and that allows you the flexibility to do that.

Why do you need multiple hosts? Are you trying to round-robin the requests? Because that's all this achieves...

[mikej1688]

Thanks for the reply.
Here are the reasons why we want to use multi hosts:
1). if we run a command like, "curator_cli --host host1 ...", and if the ES node host1 is down, a connection exception would be thrown and the curator job could not be done. But in a cluster, there would be more than one ES nodes available. We would like to resolve the above issue by connecting to another active ES node if the current one was down;
2). if I use a command like "curator --config myconfig.yml ..." and inside the config file, I can specify a hosts in the client,
client:
hosts:
- "host1"
- "host2"
port: 9200
...
here 2 hosts are specified. My question is: when I run the curator command, if the host1 is down, will the host2 be automatically connected thus no failure would be occurred?

[untergeek]

You can also do: curator_cli --config myconfig.yml ..., so that should solve your issue:

$ curator_cli --help
Usage: curator_cli [OPTIONS] COMMAND [ARGS]...

Options:
  --config PATH       Path to configuration file. Default:
                      ~/.curator/curator.yml
...

When multiple hosts are specified, they are used in a round-robin manner. If Curator is unable to connect to one node, it will attempt up to three connections to a missing/failed node before removing that node from the pool. It will continue to use the other nodes specified in a round-robin manner.

[mikej1688]

sweet, thanks.
With curator_cli, I can still set the config.yml and run an action.yml file, right?
That's what I expected.