Logstash: Encountered a retryable error. Will Retry with exponential backoff

[liza-mae]

Version: master

Could be related to #11221, but creating a new issue in case it is not. The latest available master does have the fix from that issue, so this could be some other root cause.

Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /tmp/logstash-8.0.0-SNAPSHOT/logs which is now configured via log4j2.properties
[2019-10-23T20:23:22,478][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-10-23T20:23:22,583][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.0.0"}
[2019-10-23T20:23:24,519][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2019-10-23T20:23:24,522][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2019-10-23T20:23:25,570][INFO ][org.reflections.Reflections] Reflections took 31 ms to scan 1 urls, producing 20 keys and 40 values
[2019-10-23T20:23:26,503][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:[email protected]:9200/]}}
[2019-10-23T20:23:26,603][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://elastic:[email protected]:9200/"}
[2019-10-23T20:23:26,617][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>8}
[2019-10-23T20:23:26,620][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2019-10-23T20:23:26,664][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.0.2.15:9200"]}
[2019-10-23T20:23:26,732][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2019-10-23T20:23:26,798][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2019-10-23T20:23:26,803][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, "pipeline.sources"=>["/tmp/logstash-8.0.0-SNAPSHOT/config/logstash.conf"], :thread=>"#"}
[2019-10-23T20:23:26,856][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>80001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2019-10-23T20:23:27,195][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/tmp/logstash-8.0.0-SNAPSHOT/data/plugins/inputs/file/.sincedb_f5fdf6ea0ea92860c6a6b2b354bfcbbc", :path=>["/var/log/syslog"]}
[2019-10-23T20:23:27,257][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2019-10-23T20:23:27,369][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-10-23T20:23:27,414][INFO ][filewatch.observingtail ][main] START, creating Discoverer, Watch with file and sincedb collections
[2019-10-23T20:23:28,476][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=>, hosts=>[https://10.0.2.15:9200], cacert=>"/tmp/logstash-8.0.0-SNAPSHOT/ca/ca.crt", sniffing=>false, manage_template=>false, id=>"d45100a788cbe2c3fe412cc8f68ac0e3505f79ea96fc58bfc20a69ca84dc9525", user=>"logstash_system", ssl=>true, document_type=>"%{[@metadata][document_type]}", enable_metric=>true, codec=>, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2019-10-23T20:23:28,603][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] * WARNING * Detected UNSAFE options in elasticsearch output configuration!
* WARNING * You have enabled encryption but DISABLED certificate verification.
* WARNING * To make sure your data is secure change :ssl_certificate_verification to true
[2019-10-23T20:23:28,710][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_system:[email protected]:9200/]}}
[2019-10-23T20:23:28,742][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Restored connection to ES instance {:url=>"https://logstash_system:[email protected]:9200/"}
[2019-10-23T20:23:28,747][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] ES Output version determined {:es_version=>8}
[2019-10-23T20:23:28,749][WARN ][logstash.outputs.elasticsearch][.monitoring-logstash] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2019-10-23T20:23:28,791][INFO ][logstash.outputs.elasticsearch][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://10.0.2.15:9200"]}
[2019-10-23T20:23:28,798][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#"}
[2019-10-23T20:23:28,891][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2019-10-23T20:23:28,933][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
[2019-10-23T20:23:29,466][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://10.0.2.15:9200/_bulk"}
[2019-10-23T20:23:29,471][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://10.0.2.15:9200/_bulk"}
[2019-10-23T20:23:29,504][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-10-23T20:23:31,531][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://10.0.2.15:9200/_bulk"}
[2019-10-23T20:23:31,528][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://10.0.2.15:9200/_bulk"}
[2019-10-23T20:23:35,549][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://10.0.2.15:9200/_bulk"}
[2019-10-23T20:23:35,559][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://10.0.2.15:9200/_bulk"}

[liza-mae]

@andsel I am getting the above errors on master, not sure why.

[liza-mae]

1. more logstash.conf
   input {
   file {
   path => "/var/log/syslog"
   start_position => "beginning"
   }
   }
   output {
   elasticsearch {
   hosts => [":9200"]
   }
   stdout { codec => rubydebug }
   }
2. Error happens on startup

So this is using the latest master snapshot with java 1.8 -- I am using scripts to run this, but I will work on getting some simple manual steps to reproduce.

[liza-mae]

Steps to reproduce:

I am seeing the error on both default and oss distributions using the tar file.
Running on Ubuntu 16.04.3 LTS

1. Download and install elasticsearch and logstash from latest master snapshot build for oss.
2. Edit elasticsearch.yml to have:

network.host: <IP>
discovery.type: single-node

1. Start elasticsearch
2. Edit logstash.conf

input {
  file {
    path => "/var/log/syslog"
    start_position => "beginning"
  }
}
output {
  elasticsearch {
    hosts => ["<IP>:9200"]
    }
  stdout { codec => rubydebug }
}

1. Start logstash and observe error
   [2019-10-24T15:54:13,384][ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"http://:9200/_bulk"}

*Note: I also have Kibana in my setup and I don't see the logstash index pattern available to create.

[andsel]

@liza-mae this is related to the removal of types in ES master and so the Logstash ES output plugin in on fixing it (https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/892)

[robbavey]

This issue should now be fixed with the release of new Elasticsearch input, filter and output plugins and updating the elasticsearch gem requirements in Logstash.

[DasonBryant]

Hello. The same problem occured recently. Would u mind sharing how to solve the problem eventually? thanks a lot