Logstash: Warning in logs: A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids

[ycombinator]

Observed on master / 8.0.0

Steps to reproduce

1. Run Logstash with any pipeline.

   ./bin/logstash -e 'input { stdin {} }'
   

2. Check the Logstash log for warnings.

Observed

The following warning is emitted in the logs.

[2019-06-24T08:45:07,882][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.

Expected

The above warning is not emitted.

[ycombinator]

/cc @cachedout

[cachedout]

I know what causes this but I would like to merge the pending PRs before pushing a fix for this so that it can be properly tested before review.

[inqueue]

Observed with 7.2.0.

[2019-06-27T14:00:07,092][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2019-06-27T14:00:07,122][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2019-06-27T14:00:07,754][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2019-06-27T14:00:07,805][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"5df3376e-d892-4c9d-85cc-0986c2b199d1", :path=>"/var/lib/logstash/uuid"}
[2019-06-27T14:00:15,847][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2019-06-27T14:00:15,855][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x7444122a run>"}
[2019-06-27T14:00:16,208][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"main"}
[2019-06-27T14:00:16,245][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:5044", :ssl_enable=>"false"}
[2019-06-27T14:00:16,344][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-06-27T14:00:16,846][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

One pipeline configuration in /etc/logstash/conf.d/run.conf.

cat /etc/logstash/conf.d/run.conf 
input {
  tcp {
    port => 5044
    codec => json
  }
}

filter {
}

output {
  stdout {
    codec => rubydebug
  }
}

[alexPolisevschi]

I am having the same issue in version 7.0 (I upgradet the DEV environment today).

{"log":"[2019-07-16T12:32:46,533][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.\n","stream":"stdout","time":"2019-07-16T12:32:46.533830481Z"}

[cachedout]

This should be fixed by #10970 which is awaiting a review from @elastic/logstash

EDIT: This ended up not getting merged.

[amitavmohanty01]

[2019-11-06T02:37:07,694][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.

I am observing this in 7.4.2 packages.

[gdcrocx]

@cachedout Please advise ETA on the merge and that the merge will fix this issue. Thank you.

[Skip75]

FYI
Still observing this in 7.5.2 packages.

[samxiaoxiao]

Wrong post. I'll delete. Sorry. :P

[ljufer]

I'm observing the same issue on 7.5.1 and 7.5.2.

LS on Ubuntu 18.04.3 LTS with :

• logstash/stable,stable,now 1:7.5.1-1 all
• openjdk-11-jre-headless/now 11.0.5+10-0ubuntu1.1 18.04 amd64

LS on centos-release-7-7.1908.0.el7.centos.x86_64 with:

• logstash-7.5.2-1.noarch
• java-1.8.0-openjdk-headless-1.8.0.232.b09-0.el7_7.x86_64

The WARN messages are (approx.) the same as mentioned in the initial post above.

[2020-02-12T11:36:16,839][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][filebeat_op] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2020-02-12T11:36:19,394][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][infoblox_grid] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2020-02-12T11:36:19,569][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][trendmicro_iwsva] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
...

The message is repeated for every pipeline and every time logstash is started/restarted.
However, no negative impact observed on any function so far.

[bdwoodruff]

Wondering the status of this seeing the issue on Mac version 7.2.0

[2020-02-13T10:33:19,870][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-02-13T10:33:19,883][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2020-02-13T10:33:24,009][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-02-13T10:33:24,149][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-02-13T10:33:24,186][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2020-02-13T10:33:24,188][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2020-02-13T10:33:24,207][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2020-02-13T10:33:24,259][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2020-02-13T10:33:24,306][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-02-13T10:33:24,406][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-02-13T10:33:24,410][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, :thread=>"#"}
[2020-02-13T10:33:24,704][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2020-02-13T10:33:24,760][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-02-13T10:33:24,765][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2020-02-13T10:33:25,030][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[DevendranR]

am totally new to logstash and even i am facing this in 7.5.2 package with windows. I am not able to read any log files.
Below is the logs

D:Appslogstash-7.5.2.tarlogstash-7.5.2bin>logstash -f D:Appslogstash-7.5.2.tarlogstash-7.5.2configlogstash-sample.conf
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.runtime.encoding.EncodingService (file:/D:/Apps/logstash-7.5.2.tar/logstash-7.5.2/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.Console.cs
WARNING: Please consider reporting this to the maintainers of org.jruby.runtime.encoding.EncodingService
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to D:/Apps/logstash-7.5.2.tar/logstash-7.5.2/logs which is now configured via log4j2.properties
[2020-02-18T14:27:15,043][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-02-18T14:27:15,197][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.5.2"}
[2020-02-18T14:27:16,877][INFO ][org.reflections.Reflections] Reflections took 39 ms to scan 1 urls, producing 20 keys and 40 values
[2020-02-18T14:27:18,708][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-02-18T14:27:18,899][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-02-18T14:27:18,946][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-02-18T14:27:18,953][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2020-02-18T14:27:19,007][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2020-02-18T14:27:19,019][INFO ][logstash.filters.geoip ][main] Using geoip database {:path=>"D:/Apps/logstash-7.5.2.tar/logstash-7.5.2/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.3-java/vendor/GeoLite2-City.mmdb"}
[2020-02-18T14:27:19,066][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-02-18T14:27:19,136][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-02-18T14:27:19,187][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-02-18T14:27:19,192][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["D:/Apps/logstash-7.5.2.tar/logstash-7.5.2/config/logstash-sample.conf"], :thread=>"#"}
[2020-02-18T14:27:19,641][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"D:/Apps/logstash-7.5.2.tar/logstash-7.5.2/data/plugins/inputs/file/.sincedb_98bbbdaa3b2ef5cfe264cb687a57b46c", :path=>["D:\SampleLog.log"]}
[2020-02-18T14:27:19,669][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-02-18T14:27:19,709][INFO ][filewatch.observingtail ][main] START, creating Discoverer, Watch with file and sincedb collections
[2020-02-18T14:27:19,730][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-02-18T14:27:19,996][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

[nrodriguezffdc]

I'm getting that issue as well using version 7.6.0 in the whole ELK stack.

[jai890216]

I am using windows 10 and exactly same issue and cannot read log files
log and config file attached below. I tried to load CSV with logstash, but failed every time.
Any solution you have, please share

input{

file{
path=>"C:Usersajung153611DesktopELKtest.csv"
start_position=>"beginning"
sincedb_path => "NUL"
}
}
filter{
csv{
separator=>","
columns=>["Action Conversation Log","Time in Esc-Days","Escalation Description","Escalation Reason","Escalation Trigger","SERVICE CASE","Resolution Text","Service Order Descrption",
"Solution Method","Technology","WAFER_SIZE_GROUP","Symptom","Fixassembly","Fixsubassembly","Action"]
}

}

output{
elasticsearch{
hosts=>["localhost:9200"]
index=>"semiserach"
}
stdout{codec=>rubydebug}
}

warnings

[2020-02-28T00:21:03,055][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.

[2020-02-28T00:20:58,043][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

[2020-02-28T00:21:02,545][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}

[2020-02-28T00:21:02,673][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}

[Guido11]

I get the exact same warning on a very simple pipeline i use for validation on 7.6.1 logstash going to 7.6.0 elastic server.
[WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.

[duylong]

Same issue on the version 7.6.2

[nerdvegas]

Same on 7.4.1

[zen]

It should not be closed, still observing it in 7.7.0

[robcowart]

It is happening in 7.9.2 as well.