Logstash: Failed to create monitoring event during logstash startup

[vbohata]

During LS startup I can see following message multiple times:

[logstash.inputs.metrics ] Failed to create monitoring event {:message=>"undefined method `ephemeral_id' for nil:NilClass", :error=>"NoMethodError"}

After some time it is OK, monitoring data is in ES. I am using PKI authentication for Logstash to Elasticsearch, so maybe it is related to this.

[marioluan]

Can you provide information ref. your setup (LS, ES, OS, etc)?

[vbohata]

All 6.x LS affected. Currently I am using LS 6.2.1.
Logstash in CentOS 7 docker containers, custom images using standard LS + some additional plugins + docker healthcheck using trying to read "http://127.0.0.1:9600/_node/stats/events" each 30 seconds.
ES 6.2.1

xpack monitoring conf in logstash.yml:

xpack:
    monitoring:
        elasticsearch:
            password: ''
            ssl:
                ca: ${LOGSTASH_CONFIG_PATH}/certs/cacert.crt
                keystore:
                    password: mypass
                    path: ${LOGSTASH_CONFIG_PATH}/certs/mycert.jks
            url:
            - es1.mydomain:9200
            - es2.mydomain:9200
            - es3.mydomain:9200
            username: ''
        enabled: true

[SuperFunks]

I have the same issue on Logstash 6.2.2 - Im providing the username and password in the logstash.yml
After a while - the xpack data does flow to ES.

[rugenl]

Me too, Logstash 6.2.2, every 10 seconds. At least some monitoring data is being indexed. Has anyone found a fix? I don't see any other errors in the logstash logl.

[Tapppi]

Also seeing this once on startup. Weirdly we do still get at least some monitoring data, so I doubt this is actually misconfigured credentials?? Have not yet been able to verify whether we are missing something..

[rugenl]

Mine mysteriously went away. This is new cluster and some changes were made, but since there are no other errors, who knows. We are overloaded running reindex from a prior cluster, maybe it's performance?

[Tapppi]

I have confirmed that this is another case of false positive/wrong startup order for us. We get normal monitoring in kibana, but always get this message on startup.

[leandrojmp]

I'm having a similar issue.

I'm using multiple pipelines in the pipelines.yml and when I disable a pipeline I start to get the following message in the logstash log file.

[2018-03-27T12:27:28,587][ERROR][logstash.inputs.metrics  ] Failed to create monitoring event {:message=>"undefined method `system?' for nil:NilClass", :error=>"NoMethodError"}

If I restart the logstash service, the message stop, but if I add a new pipeline and then remove it, I will receive the message again.

I'm using Logstash 6.2.1 with X-Pack.

[Dave-c-Ross]

Samething here ...

[ld57]

Same here on 6.2.1 ...
it seems appearing when logstash is a bit overloaded.

[AddoSolutions]

Same issue here. Using UDP and TCP listeners

[andrewvc]

This is a weird one. Seems like a probable race. I assume everyone here is using x-pack, since the top stack trace occurs within x-pack. We'll track this internally. Once x-pack is opened it'll be easier to have visibility here since the code will be in this repo.

[ghost]

same here...
[logstash.inputs.metrics ] Failed to create monitoring event {:message=>"undefined method `ephemeral_id' for nil:NilClass", :error=>"NoMethodError"}
6.2.2 x-pack

[ghost]

[2018-04-27T12:35:05,338][INFO ][logstash.inputs.metrics ] Monitoring License OK
[2018-04-27T12:35:07,923][ERROR][logstash.inputs.metrics ] Failed to create monitoring event {:message=>"undefined method `ephemeral_id' for nil:NilClass", :error=>"NoMethodError"}

[spawluk]

In my case error was caused by missing/wrong credentials in logstash config.yml file

[markandrewj]

We had an issue where we got similar errors. There were two things in our case causing the issue. We have automated our upgrade process, and in our pipeline.yml file we were loading both the old 'conf.d' files and the new pipelines. After resolving this issue we were left with a couple pipelines that were giving us errors, we commented them out leaving us with out any issues. We likely have a syntax issue in the couple pipelines that were complaining to resolve. I don't know if this will be helpful, but I came across this issue just now trying to resolve ours, and it seemed similar in nature. We just upgraded to 6.2.4.

[migibert]

Same issue here with Logstash 6.2.4

[TheVastyDeep]

I was getting this error every 10 seconds. Turned out to be a syntax error in the configuration of one of the pipelines, so logstash was not starting that pipeline.

Sadly now that I have fixed the syntax error logstash crashes and exits as soon as it starts all the pipelines. But hey, at least I don't have this error message any more.

[Cyb3rWard0g]

Same issue here with 6.2.4, but I still get events and everything is working fine. the error showed up after restarting Logstash Docker Container with an updated Logstash config:

[2018-06-11T06:27:38,983][ERROR][logstash.inputs.metrics  ] Failed to create monitoring event {:message=>"undefined method `ephemeral_id' for nil:NilClass", :error=>"NoMethodError"}

Running Official Elastic Logstash Docker Image (centos)
OS: Linux Ubuntu Server

[TheVastyDeep]

Definitely feels like a race condition. Running 6.3.1 with this configuration

    input { generator { count => 1 message => '0' } }
    output { stdout { codec => rubydebug } }

The good result is

[2018-07-19T13:57:23,400][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2018-07-19T13:57:23,428][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:main, :".monitoring-logstash"], :non_running_pipelines=>[]}
[2018-07-19T13:57:23,439][INFO ][logstash.inputs.metrics  ] Monitoring License OK
[2018-07-19T13:57:23,640][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9601}

However, sometime "Monitoring License OK' is delayed. If the generated event has already been output by the time it arrives then I will get this every 10 seconds

Failed to create monitoring event {:message=>"undefined method `ephemeral_id' for nil:NilClass", :error=>"NoMethodError"}

[remd]

We're running into this as well. Running on 6.4.2, with X-Pack.

[2018-10-23T15:16:09,600][ERROR][logstash.inputs.metrics  ] Failed to create monitoring event {:message=>"For path: http_address. Map keys: [:stats, :jvm]", :error=>"LogStash::Instrument::MetricStore::MetricNotFound"}
[2018-10-23T15:16:11,257][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

[atompi]

Same issue here with Logstash 6.2.4, running official Logstash docker image.

X-Pack monitoring conf in logstash.yml:

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: ["http://192.168.1.127:9200","http://192.168.1.127:9201","http://192.168.1.127:9202"]
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: 123456

[maggieghamry]

Encountered the same error in 7.4 (X-Pack monitoring enabled):

[2019-10-23T18:04:04,471][ERROR][logstash.inputs.metrics ] Failed to create monitoring event {:message=>"undefined method `[]' for nil:NilClass", :error=>"NoMethodError"}

[YAClydeM]

We're also seeing the same error in 7.4

[ankitsynx]

Getting the same error on 7.4 when enables centralized pipeline management. All pipelines work fine otherwise. Error shows up only when xpack management is enabled in logstash.yml.

Noticed this error in logs during startup:
[2019-11-22T19:44:54,919][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
Below error keeps coming after this:
[2019-11-22T19:45:07,527][ERROR][logstash.inputs.metrics ] Failed to create monitoring event {:message=>"For path: events. Map keys: [:reloads, :pipelines]", :error=>"LogStash::Instrument::MetricStore::MetricNotFound"}

[blackswan1]

Just tested with logstash 7.5 and xpack.monitoring.enabled: true
logstash throws constantly the following error:

Dec 03 09:39:35 host.xyz.com logstash[21178]: [2019-12-03T09:39:35,810][ERROR][logstash.inputs.metrics ] Failed to create monitoring event {:message=>"undefined method `[]' for nil:NilClas s", :error=>"NoMethodError"}

[cantipop]

Just encounter this issue. Logstash 6.8.5. I modified configuration to one of the pipeline and this modification resulted in this error.
Fixed by removing /var/lib/logstash/uuid file.
I did not investigated why or what can cause this, maybe because i modified some of the "id" field in the pipeline.