Logstash: Impossible to use variables in ilm_rollover_alias

[GuillaumeDuf]

Hello I am using logstash with ILM and rollover alias and it seems the parameter ilm_rollover_alias does not support dynamic parameters.

• Version: Logstash 6.8.1
• Operating System: Red Hat 7
  It joins one of the problems mentioned here :
  https://github.com/elastic/logstash/issues/10687
  https://github.com/elastic/logstash/issues/10596

I'm managing templates externally so I set
manage_template => false

But , in my use case , i have many input and filter sections , and i have only one output section wich was configured like this in the past with a variable index name , when I was not using ILM

    {
            hosts => ["https://d116qsdf.cheapdr.dd:9243/"]    
        index => "%{[@metadata][index]}"
        document_type => "_doc"
        document_id => "%{[@metadata][hash]}"
        action => "create"
      }

Which was working perfectly .
But now , if I want to do the same with the ilm_rollover_alias like this

output {
        elasticsearch
        {
            hosts => ["https://d116qsdf.cheapdr.dd:9243/"]    
                        ilm_enabled => true
                        manage_template => false
                        ilm_rollover_alias => "%{[@metadata][ilm_rollover_alias]}" 
                        ilm_pattern => "{now/M{YYYY.MM}}-000001"
                        document_type => "_doc"
                        document_id => "%{[@metadata][hash]}"
                        action => "create"
                        }
}

I get this error :

 An unexpected error
occurred! {:error=>java.net.URISyntaxException: Malformed escape pair at index 0: %{[@metadata][ilm_rollover_alias]}, :backtrace=>["java.net.URI$Parser.fail(java/net/URI.java:2848)", "java.net.URI$Parser.scanEscape(java/net/URI.java:2978)", "java.net.URI$Parser.scan(java/net/URI.java:3001)", "java.net.URI$Parser.checkChars(java/net/URI.java:3019)", "java.net.URI$Parser.parseHierarchical(java/net/URI.java:3105)", "java.net.URI$Parser.parse(java/net/URI.java:3063)", "java.net.URI.<init>(java/net/URI.java:588)", "java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:423)", "org.jruby.javasupport.JavaConstructor.newInstanceDirect(org/jruby/javasupport/JavaConstructor.java:279)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:894)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.manticore_adapter.format_url(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:97)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.manticore_adapter.perform_request(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:67)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.pool.perform_request_to_url(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.pool.perform_request(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.pool.with_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.pool.RUBY$method$with_connection$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java/lib/logstash/outputs/elasticsearch/http_client//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.pool.perform_request(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.pool.Pool(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.exists?(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.rollover_alias_exists?(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client.RUBY$method$rollover_alias_exists?$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java/lib/logstash/outputs/elasticsearch//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/http_client.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.ilm.maybe_create_rollover_alias(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:89)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.ilm.RUBY$method$maybe_create_rollover_alias$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java/lib/logstash/outputs/elasticsearch//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/ilm.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.ilm.setup_ilm(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:11)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.ilm.RUBY$method$setup_ilm$0$__VARARGS__(usr/share/logstash/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java/lib/logstash/outputs/elasticsearch//usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/ilm.rb)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_9_dot_4_dot_0_minus_java.lib.logstash.outputs.elasticsearch.common.setup_after_successful_connection(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-9.4.0-java/lib/logstash/outputs/elasticsearch/common.rb:51)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}

I don't know if there's a real reason why this feature is not supported.
Is there a plan to propose this feature in the future?

Thank you

[jsvd]

If you already have setup ILM, then you can continue using the index parameter with interpolation.

the ILM fields are used at startup to create the write alias, install template, etc. All these actions are not related to events, since at that point there may be no events yet, so they can't use any event related information.

[lucabelluccini]

Hello, as @jsvd suggests, the problem is the fact ILM requires Logstash to:
1) Install an Index Template containing the index.lifecycle settings
2) It bootstraps the alias for the given index if not present

Unfortunately, at the moment this is not applicable for dynamic index names as otherwise Logstash would have to lookup into Elasticsearch before each bulk request writing the data.

The possible workarounds:
1) Use the index parameter and disable ilm on Logstash. You'll have to bootstrap the alias manually.
2) If you know in advance all the different values, partition them with if/else if/else conditions so that the ilm_rollover_alias is a static value for every output.

[GuillaumeDuf]

If you already have setup ILM, then you can continue using the index parameter with interpolation.

the ILM fields are used at startup to create the write alias, install template, etc. All these actions are not related to events, since at that point there may be no events yet, so they can't use any event related information.

Thank you for your answer.
Bootstraping the alias manually is not an option for me.
Why logstash is doing this at start-up and not when the first event comes up ?

To make it works it force you to use that kind of syntax

output {
        if [@metadata][ilm_rollover_alias] == "idx_perf_apache"
    {
             elasticsearch
                {
                hosts => ["https://d116qsdf.cheapdr.dd:9243/"]    
                        ilm_enabled => true
                        manage_template => false
                        ilm_rollover_alias => "idx_perf_apache" 
                        ilm_pattern => "{now/M{YYYY.MM}}-000001"
                        document_type => "_doc"
                        document_id => "%{[@metadata][hash]}"
                        action => "create"
                         }
        }
}

this is not really what you might call a "dynamic parameter" But it works ..
Logstash does not execute the code sequentially.
In fact, it is not possible to start creating the alias when the first event arrives, for example

output {
        if [@metadata][ilm_rollover_alias] == [@metadata][ilm_rollover_alias]
    {
             elasticsearch
                {
                hosts => ["https://d116qsdf.cheapdr.dd:9243/"]    
                        ilm_enabled => true
                        manage_template => false
                        ilm_rollover_alias => "%{[@metadata][ilm_rollover_alias]}" 
                        ilm_pattern => "{now/M{YYYY.MM}}-000001"
                        document_type => "_doc"
                        document_id => "%{[@metadata][hash]}"
                        action => "create"
                         }
        }
}

[GuillaumeDuf]

Hello, as @jsvd suggests, the problem is the fact ILM requires Logstash to:

1. Install an Index Template containing the index.lifecycle settings
2. It bootstraps the alias for the given index if not present

Unfortunately, at the moment this is not applicable for dynamic index names as otherwise Logstash would have to lookup into Elasticsearch before each bulk request writing the data.

The possible workarounds:

1. Use the index parameter and disable ilm on Logstash. You'll have to bootstrap the alias manually.
2. If you know in advance all the different values, partition them with if/else if/else conditions so that the ilm_rollover_alias is a static value for every output.

Thank you for your answer . I got the problem
I think that partitioning the output with if/else conditions is the only workaround on my side.

[HeWhoWas]

This is an issue that I have run into as well, I was going to report it today. Creating the index write_alias manually is acceptable, but it isn't ideal - beyond the issue of manually have to bootstrap every alias/index, if the write alias is deleted, data will automatically be pushed into a concrete index with the same name - making re-creation of the alias impossible until the entire pipeline is shutdown and data is no longer being received for the index.

I'm going to look at templating the elasticsearch output file and generating it with static fields, but am curious whether there is any view towards changing this implementation?

[lucabelluccini]

I feel like we should move this issue to the Elasticsearch Output repo (e.g. https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/858)

[JathinSanghvi]

The possible workarounds:

1. Use the index parameter and disable ilm on Logstash. You'll have to bootstrap the alias manually.

Can you please enumerate what actions needs to be performed during bootstrapping a index with lifecycle.

we are using daily index with YYYY.MM.DD in index name, does that mean bootstrapping has to be performed every day?

[lucabelluccini]

Hello @JathinSanghvi - Bootstrapping means creating the first index and its associated write alias for the first time. It is documented at this page.

E.g. in your case might be:

# PUT <yourindex-{now/d}-000001>
PUT /%3Cyourindex-%7Bnow%2Fd%7D-000001%3E
{
  "aliases": {
    "yourindexalias": {
      "is_write_index": true
    }
  }
}

The command above will create the index yourindex-2019.08.20-000001 with an alias yourindexalias pointing to it with the attribute is_write_index to true.

[ppf2]

To help consolidate feedback (and votes) from the field on this request:

If you end up in this issue, please vote in the parent issue in the logstash-output-elasticsearch repository: https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/858

[pickypg]

As a drive by comment, if you setup your own ILM policy, without using Logstash and attach it via your template, then you can index using the alias _as your index_:

elasticsearch {
  hosts => ["https://d116qsdf.cheapdr.dd:9243/"]    
  index => "%{[@metadata][ilm_rollover_alias]}"
  document_type => "_doc"
  document_id => "%{[@metadata][hash]}"
  action => "create"
}

Then, as far as Logstash is concerned, you should just be indexing into a normal index. As long as you have ILM configured, it should be acting appropriately and automatically without involving Logstash.

This does assume that you have a fixed number of indices coming through Logstash, but if you don't then you probably need to reconsider your workflow regardless of this issue.

[lucabelluccini]

To apply the solution suggested on the previous comment:

• on Logstash 7.x if the destination cluster is on 7.x, ilm_enabled must be set to false (doc)
• it is necessary to bootstrap the rollover_alias manually (doc) - as it is a variable, we need to bootstrap all the possible destination aliases