Logstash: Multiple pipelines - logstash-6.0.0-alpha2

[twinome]

I’m keen to try the multiple pipelines functionality in logstash-6.0.0-alpha2. So far I’ve been unable to get it to work. I’ve edited the “pipelines.yml” file in the config folder. I’ve added the following (fyi - the hyphens are coming out as bullets on this forum post):

• pipeline.id: pipeline1
  path.config: "c:\aconfigfile1.config"
• pipeline.id: pipeline2
  path.config: "c:\aconfigfile2.config"

I receive the following error when launching Logstash:

[2017-08-02T11:01:29,433][ERROR][logstash.agent ] An exception happene d when converging configuration {:exception=>RuntimeError, :message=>“Could not fetch the configuration, message: undefined method pipeline_id' for nil:NilClas s", :backtrace=>["C:/logstash-6.0.0-alpha2/logstash-core/lib/logstash/agent .rb:153:inconverge_state_and_update’”, “C: /logstash-6.0.0-alpha2/logstash -core/lib/logstash/agent.rb:88:in execute'", "C: /logstash-6.0.0-alpha2/log stash-core/lib/logstash/runner.rb:302:inexecute’”, “org/jruby/RubyProc.java:28 1:in call'", "C:/ logstash-6.0.0-alpha2/vendor/bundle/jruby/1.9/gems/stud-0 .0.22/lib/stud/task.rb:24:ininitialize’”]}

It's possible I've used the wrong format for the config paths. If not, perhaps a bug. Any guidance appreciated - thanks.

[jsvd]

@twinome it is a bug that will be fixed in beta1. that said, it's a poor handling of failure to create the pipeline, are there any other logs about no finding the files?

[jsvd]

For example:

% bin/logstash
[2017-08-03T09:30:25,002][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/tmp/ogstash/*.config"}
[2017-08-03T09:30:25,006][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>NoMethodError, :message=>"undefined method `pipeline_id' for nil:NilClass", :backtrace=>["/tmp/logstash-6.0.0-alpha2/logstash-core/lib/logstash/config/source_loader.rb:120:in `find_duplicate_ids'", "org/jruby/RubyArray.java:1613:in `each'", "org/jruby/RubyEnumerable.java:1709:in `group_by'", "/tmp/logstash-6.0.0-alpha2/logstash-core/lib/logstash/config/source_loader.rb:120:in `find_duplicate_ids'", "/tmp/logstash-6.0.0-alpha2/logstash-core/lib/logstash/config/source_loader.rb:62:in `fetch'", "/tmp/logstash-6.0.0-alpha2/logstash-core/lib/logstash/agent.rb:146:in `converge_state_and_update'", "/tmp/logstash-6.0.0-alpha2/logstash-core/lib/logstash/agent.rb:88:in `execute'", "/tmp/logstash-6.0.0-alpha2/logstash-core/lib/logstash/runner.rb:302:in `execute'", "org/jruby/RubyProc.java:281:in `call'", "/tmp/logstash-6.0.0-alpha2/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:in `initialize'"]}

[twinome]

@jsvd good to know, thanks. Do you know when beta1 is due for release?

[jsvd]

@twinome should be soon, sometime next week unless we find other issues that require delaying it a bit more.

[twinome]

@jsvd just to confirm, are you saying that multiple pipelines don't work in logstash-6.0.0-alpha2 or just the handling for missing config files or bad path?

[jsvd]

The loading of multiple pipelines will work correctly if there are no errors. if there is an error you'll see that exception. example with correct path:

- pipeline.id: test
  config.string: "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }"
- pipeline.id: another_test
  path.config: "/tmp/logstash/*.config"

/tmp/logstash-6.0.0-alpha2 % bin/logstash -r 
[2017-08-03T10:15:09,358][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2017-08-03T10:15:09,689][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"another_test", :thread=>"#<Thread:0x3404a46a run>", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-08-03T10:15:09,701][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:5555"}
[2017-08-03T10:15:09,702][INFO ][logstash.pipeline        ] Pipeline started {"pipeline.id"=>"another_test"}
[2017-08-03T10:15:09,891][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"test", :thread=>"#<Thread:0x1e1cc02e run>", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-08-03T10:15:09,900][INFO ][logstash.pipeline        ] Pipeline started {"pipeline.id"=>"test"}
[2017-08-03T10:15:09,914][INFO ][logstash.agent           ] Pipelines running {:count=>2, :pipelines=>["another_test", "test"]}

[twinome]

@jsvd thanks for the info. I've had a look at the verbose logs and I think I'm not loading the configs property. I'll have a play. Cheers. Good to know the better handling for bad path or no file will be in beta1.

[jakelandis]

This has been fixed and was only applicable to pre-released versions. Closing.