Logstash: Logstash v6 pipelines.yml ignored

[mori-ol]

Hello,

I am running Logstash v.6 on Centos 7.3.. I have tried to run it without specifying any modules or command line options but it ignores my pipeline.yml.

When I run logstash, without any option, I get the following message:
sudo /usr/share/logstash/bin/logstash
[WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

No modules or command line options is specified.

[jakelandis]

@mori-ol - Thanks for reporting this !

We have a check in our code that does not allow for multiple pipelines AND path.config set. However, it appears that in our packaging (.rpm, .deb) we set the path.config by default in logstash.yml.

For example for CentOs: https://github.com/elastic/logstash/blob/master/pkg/centos/after-install.sh#L5

This means that by default we prefer a single pipeline configuration over a multi-pipeline configuration. I _think_ this is intentional behavior, but the error message, documentation, and possibly the default behavior can certainly be improved.

The _fix_ for this it open your logstash.yml and comment out the path.config line, and your pipelines.yml should get picked up.

[jakelandis]

I think the best way forward here may be promote multi-pipeline as the default with the following strategy:

• Stop setting the path.config by default
• Ship default configurations (generator -> elasticsearch + stdin -> stdout ?)
• Ship a default pipelines.yml that points to the default configurations.
• Update the documentation.

cc: @jsvd @andrewvc @jordansissel

[jordansissel]

@jakelandis I agree with your proposal. If we can do this without breaking compatibility (users dropping files in /etc/logstash/conf.d), I am +1

[mori-ol]

Hi Guys,

Thanks for your input! I have commented out the path.config in my installation and now logstash is trying to execute pipelines.yml.
I am getting the following error now though:

[datamgr@desktopoli logstash]$ sudo /usr/share/logstash/bin/logstash --verbose
2017-10-04 10:45:33,387 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2017-10-04 10:45:34,707 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help

my pipelines.yml is pretty basic at the moment:

• pipeline.id: infl-remote
  path.config: "/etc/logstash/conf.d/logstash-influx-data.conf"

pipeline.workers: 4

pipeline.batch.size: 1000

• pipeline.id: elas-remote
  path.config: "/etc/logstash/conf.d/logstash-es-data.conf"

Any ideas?

[mori-ol]

Ignore the appender errors below, got rid of them and now just getting:
ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml

[jsvd]

I think the best way forward here may be promote multi-pipeline as the default with the following strategy

@jakelandis @jordansissel Just for context, we had this discussion in the past and ended up deciding to leave the packages use -f by default. IIRC the main argument being that this new feature should be opt in for the first release that ships with it, and reducing the surprise for existing users.

[endersonmaia]

I'm having the same problem using the Docker Container Image docker.elastic.co/logstash/logstash:6.0.0.

▶ docker-compose up logstash Recreating gstsync_logstash_1 ... Recreating gstsync_logstash_1 ... done Attaching to gstsync_logstash_1 logstash_1 | 2017/11/22 13:37:13 Setting 'queue.max_bytes' from environment. logstash_1 | 2017/11/22 13:37:13 Setting 'http.host' from environment. logstash_1 | 2017/11/22 13:37:13 Setting 'queue.max_events' from environment. logstash_1 | 2017/11/22 13:37:13 Setting 'queue.checkpoint.writes' from environment. logstash_1 | 2017/11/22 13:37:13 Setting 'queue.page_capacity' from environment. logstash_1 | 2017/11/22 13:37:13 Setting 'xpack.monitoring.enabled' from environment. logstash_1 | 2017/11/22 13:37:13 Setting 'queue.checkpoint.acks' from environment. logstash_1 | 2017/11/22 13:37:13 Setting 'queue.type' from environment. logstash_1 | 2017/11/22 13:37:13 Setting 'log.level' from environment. logstash_1 | 2017/11/22 13:37:13 Setting 'config.reload.automatic' from environment. logstash_1 | 2017/11/22 13:37:13 Setting 'config.debug' from environment. logstash_1 | 2017/11/22 13:37:13 Setting 'queue.checkpoint.interval' from environment. logstash_1 | Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties logstash_1 | [2017-11-22T13:37:34,124][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"} logstash_1 | [2017-11-22T13:37:34,142][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"} logstash_1 | [2017-11-22T13:37:35,059][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.0.0-java/modules/arcsight/configuration"} logstash_1 | [2017-11-22T13:37:35,377][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified logstash_1 | [2017-11-22T13:37:35,638][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} logstash_1 | [2017-11-22T13:37:36,253][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 78, colu mn 1 (byte 2288) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_ast'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:54 :in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logsta sh-core/lib/logstash/pipeline.rb:107:in `compile_lir'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:215:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipel ine_action/create.rb:35:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:3 32:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/usr/sha re/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/usr/share/logstash/logstash -core/lib/logstash/runner.rb:362:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

For what I did understand, it's enabling 3 modules by default, and that's the reason my pipelines.yml is being ignored.

I couldn't find in the documentation, a way I can disable the modules.

[jsvd]

The initializing modules stage only loads the content of the modules so that logstash can use them, they aren't actually being started.
The Expected one of #, input, filter, output at line 78, column 1 (byte 2288) after error in the message you provided points to a syntax error in the line 78 of one of the pipelines in the pipelines.yml path.config you set.

[endersonmaia]

@jsvd that's weird, 'cause none of the files I could identify have that (78) many lines, but I'm going to double check everything, simplify the setup to isolate the problem

and the message seems very clear Ignoring the 'pipelines.yml' file because modules or command line options are specified, so my pipelines.yml wasn't even read, according to the message

I assume, unless I did another error, that, at least the message is misleading, right ?

[jsvd]

oh, apologies, I misread the issue, this is another instance of https://github.com/elastic/logstash-docker/issues/64, the docker image forcefully uses the equivalent of -f in it's definition.

[Camillevau]

The fix for this it open your logstash.yml and comment out the path.config line, and your pipelines.yml should get picked up.

I tried comment it out: same "error".
putting /etc/logstash/pipelines.yml: same "error".

how to override the value properly ?

I am in a docker environment,I just pushed my own logstash.yml in docker-compose config. After deployment and double check, value is still the one I set, not the one replaced by script example pointed by @jakelandis.

[DiogoAndre]

Hi all,

If pipelines.yml is being ignored because a module is loaded, can anyone confirm it is a current limitation?

The only thing close to a confirmation that I could find was on this comment from an old issue (when modules were implemented in Logstash)

Based on our discussion, we are only supporting a single module to be run at a time on Logstash. So in other terms, 1 module is equivalent to 1 pipeline, which is equivalent to running bin/logstash -f cef.conf.
@suyograo

https://github.com/elastic/logstash/issues/6851#issuecomment-298733217

I'd appreciate if this could be confirmed, and would be happy to send a PR to better reflect this limitation in the documentation.

Thanks!

[ajpahl1008]

I've run into this same problem where I had perfectly great running pipelines until I turned on Netflow module in logstash.yml. Ubuntu 17.10.

[2018-01-04T12:50:05,126][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help

[guyboertje]

I can confirm that the modules feature in Logstash is a stand alone pipeline. Using -e, -f or --modules will amount to pipelines.yaml being ignored.
The reason for this is Modules are a "getting started" quickly feature so that new Elastic Stack users can quickly build up an appreciation of the Stack without having to build Kibana objects and learn the LS config language. To do this we did not want a "toy" implementation that looks nice but has no real value.

The netflow module code is open source, this means that you can find and extract the LS config after you have ran the --setup to get the ES and Kibana artefacts installed. The config is an ERB template so there are sections that are overwritten, in between <%=, %>, with real values - replace these and you will have a netflow config.

[jakelandis]

A multi-pipeline setup will be the default in 6.2.0 as of https://github.com/elastic/logstash/pull/8894.

A default pipelines.yml will ship configured with 1 pipeline that pulls it's configuration from /etc/logstash/conf.d/*.conf

Closing this issue.

[tfalcao]

If you still use logstash 6.0.1 as container use this command for Dockerfile:

RUN sed -i -e 's/path.config/#path.config/g' /usr/share/logstash/config/logstash.yml

It will comment path.config inside /usr/share/logstash/config/logstash.yml

[3rdstage]

To launch multiple pipelines with Logstash, it is necessary NOT to use -f or -e command-line options and path.config or config.string settings (which are equivalent with -f or -e command-line options) in logstash.yml file.

• Logstash command-line
• logstash.yml
• Introducing Multiple Pipelines in Logstash

[Rakeshsambari]

I am getting below error even after comment out path.config in logstash.yml I am working on windows7
please help

logstash.config.source.multilocal] Ignoring the
'pipelines.yml' file because modules or command line options are specified

[lwx2615]

Ignoring the 'pipelines.yml' file because modules or command line options are specified,when the modules set in the logstash.yml file,the error comes out!I am use the version 6.3.2

[axerontios]

i have been trying this for a week without a success, im running on the latest docker image
logstash:6.4.2

/usr/share/logstash/bin/logstash --path.settings /etc/logstash --debug
logstash:6.4.2
[DEBUG] 2018-10-11 19:23:07.932 [LogStash::Runner] multilocal - Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
ERROR: Failed to read pipelines yaml file. Location: /etc/logstash/pipelines.yml
usage:

its a da£$ shame this is full of potential but its such a apain to make it work

[psychogun]

Just wanted to chime in;
Running ELK Stack 7.2 on an Ubuntu 18.04 and I have configured it for syslogs from my pfSense. I installed softlow on pfSense to use the Netflow module also for my pfSense.

I got netflow configured and looking good in Kibana by using the --setup parameter on the first time run of logstash, then I put the -modules netflow configuration in logstash.yml. But I noticed that the syslogs had stopped; in the logstash logs it said 'Ignoring the 'pipelines.yml' file because modules or command line options are specified'. Kibana stopped showing firewall syslog events.

So I googled and came here.

And basically, what I did, was to do what @guyboertje said;

replace the sections with <%=, %>, with real values.

Copy the netflow.conf.erb file to your path.config directory ("/etc/logstash/conf.d/*.conf"):
cp /usr/share/logstash/modules/netflow/configuration/logstash/netflow.conf.erb /etc/logstash/conf.d/netflow.conf
and then
sudo nano /etc/logstash/conf.d/netflow.conf to edit and remove the ERB formatting.

E.g.:
port => <%= setting("var.input.udp.port", 2055) %> to port => 2055

and

dictionary_path => "<%= ::File.join(LogStash::Environment::LOGSTASH_HOME, "/modules/netflow/configuration/logstash/dictionaries/iana_service_names_tcp.yml") %>" to dictionary_path => "/usr/share/logstash/modules/netflow/configuration/logstash/dictionaries/iana_service_names_tcp.yml"

and
output { <%= elasticsearch_output_config() %> } to
output { elasticsearch { hosts => ["http://localhost:9200"] index => "netflow-%{+YYYY.MM.dd}" } }

etc. ..

Remember to remove any -modules config you have in logstash.yml.

Then this .conf file will be picked up as a pipeline, too. Now I have a working Kibana with both syslogs and netflow with great dashboards :)

PS: I do not know if that netflow.conf.erb file existed before i ran logstash with -modules netflow --setup parameter, and this might not be how others would do it, but atleast it worked for me- running with --setup first, and then run it as a pipeline.

PS:PS: sudo tail -f /var/log/logstash/logstash-plain.log helped a lot.