Logstash: Receiving "Empty configuration for pipeline_id" error when try send data from logstash to Elastic Cloud.

[zabaala]

Additional info:

• Logstash version: 6.3.1
• OS: macOS 10.13.4
• Elastic Search: 6.2.24 on Elastic Cloud
• Kibana: 6.2.24 on Elastic Cloud

Problem:

Hi,

I'm trying send data from logstash to Elastic Cloud, but i'm receiving the follow error when logstash run:

"Empty configuration for pipeline_id: artist_profile_views"

But if i try run logstash without define xpack configurations in logststah.yml, this work fine and the stdout {} send collected data to output.

Please, see my configurations file:

# logststash.yml

cloud.id: "[...]"
cloud.auth: "[user]:[password]"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: https://elasticCloudUrl
xpack.monitoring.elasticsearch.username: [user]
xpack.monitoring.elasticsearch.password: [password]
xpack.management.enabled: true
xpack.management.pipeline.id: ["artist_profile_views", "searched"]
xpack.management.elasticsearch.username: [user]
xpack.management.elasticsearch.password: [password]
xpack.management.elasticsearch.url: ["https://elasticCloudUrl"]

# pipelines.yml

- pipeline.id: artist_profile_views
  path.config: "pipelines/artist_profile_views.conf"

- pipeline.id: searched
  path.config: "pipelines/searched.conf"

# artist_profile_views

input { 
    file {
        path => "/Users/zabaala/Sites/cna/stats/artist_profile_views/artist_profile_views_*.log"
        codec => json
        start_position => "beginning"
    }
}

filter {
    geoip {
        source => "[ip]"
    }   
    useragent {
        source => "[headers][user_agent]"
        target => "[headers][request]"
    }
    mutate {
        remove_field => ["[headers][user_agent]"]
    }
}


output {
    # elasticsearch { 
        # hosts => ["https://elasticCloudUrl"] 
        # index => "stats"
    #}
    stdout { 
        codec => rubydebug
    }
}

Sample data:

{"artist_profile_views":{"id":"510","type":"ARTIST","area":"PHOTOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:47-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"PHOTOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:48-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"EVENTS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:50-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"AREA_AUDIOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:52-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"VIDEOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:20:55-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"HOME"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:31:32-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}
{"artist_profile_views":{"id":"510","type":"ARTIST","area":"AREA_AUDIOS"},"env":"local","ip":"172.18.0.1","index":"stats","doc":"artist_profile_views","when":{"date":"2018-07-06T17:31:43-0300"},"viewer":{"id":null,"context":"GUEST","by_himself":true},"headers":{"user_agent":"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/67.0.3396.99 Safari\/537.36"}}

What's wrong?

Thanks.

[danhermann]

This is a usage question rather than a bug and should be asked in the Logstash forums at https://discuss.elastic.co/c/logstash. From a quick read of your post, it sounds like you may be inadvertently enabling Centralized Pipeline Management in your config with the xpack.management.pipeline.id setting. If that's what you're intending to do, please post a follow-up question in the Logstash forum.

[zabaala]

Thanks, @danhermann.

Centralized Pipelines work fine for me.

[gydtc]

Hi , I have the same problem. Could u tell me how to solve it.
Thanks!!!
@zabaala

[zabaala]

@gydtc the link passed by @danhermann talk about centralized pipelines. In my case, I was trying to send the data to the Elastic Cloud, informing the pipeline, but it did not exist there in Kibana > Management > Pipeline.

What is not clear in the link (or at least I did not realize) is that you do not need to have the pipelines configuration files in the location where the logstash is. The role of the centralized pipeline is precisely not to use them, since they are in the elasticsearch.

With this, it is enough, in the configurations, you define the ID of the pipeline that that logstash needs to execute and ready, problem solved.

[gydtc]

@zabaala Thanks!!!
As I know, I can define the pipeline and id named just like "test123" in Kibana > Management > Pipeline.
and then I cat set the logstash.yml > xpack.management.pipeline.id : [ "test123" ]。
Then the logstash will read the pipeline from elasticsearch，is it？

[zabaala]

Yes, @gydtc.

And if you have a xpack license installed in you stack, you can see each event triggered by logstash in Kibana > Monitoring > Pipelines.