Logstash: Multiple Pipelines doesn't seem to work with Windows OS

[cinhtau]

• We have created two logstash configurations to run in separate pipelines in logstash.
• If we run logstash with -f config file separately it works.

We have defined a pipelines.yml

---

- pipeline.id: "astrex-logs"
  path.config: "C:/Daten/QS_Progs/logstash-6.2.1/config/astrex-logs.conf"
  pipeline.workers: 1
  queue.type: memory

- pipeline.id: "astrex-html"
  path.config: "C:/Daten/QS_Progs/logstash-6.2.1/config/astrex.conf"
  pipeline.workers: 1
  queue.type: memory

• Version: Logstash v.6.2.1
• Operating System: Windows Server 2012
• Config File: pipelines.yml
• Steps to Reproduce:
  
  
  
  • Create two simple test configurations
  
  
  • Define pipelines.yml in config directory
  
  
  • Start bin\logstash.bat
  
  

If we run bin\logstash.bat --debug -r he falls back to the pipelines.yml and give us

[2018-02-14T17:00:50,201][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.

More context:

[2018-02-14T17:00:16,561][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2018-02-14T17:00:16,576][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"C:/Daten/QS_Progs
/logstash-6.2.1/config/pipelines.yml"}
[2018-02-14T17:00:16,936][DEBUG][logstash.api.service     ] [api-service] start
[2018-02-14T17:00:17,061][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2018-02-14T17:00:17,092][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
[2018-02-14T17:00:17,217][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
...
[2018-02-14T17:00:50,201][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"C:/Daten/QS_Progs
/logstash-6.2.1/config/pipelines.yml"}
[2018-02-14T17:00:50,201][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2018-02-14T17:00:50,201][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
[2018-02-14T17:00:51,591][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-02-14T17:00:51,591][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-02-14T17:00:51,841][WARN ][logstash.runner          ] SIGINT received. Shutting down.
[2018-02-14T17:00:51,857][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2018-02-14T17:00:51,873][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2018-02-14T17:00:51,873][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2018-02-14T17:00:51,873][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2018-02-14T17:00:51,919][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>0}
[2018-02-14T17:00:51,935][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
Terminate batch job (Y/N)?

Our current workaround is to start logstash with one config only by omitting -f astrex.conf.
Can you confirm this is bug under windows?

Using pipelines under Linux wasn't a problem so far.

[abunet]

+1 Same problem on Windows Server 2012 and Windows 10.
logstash version: 6.2.1

D:\strumenti\logstash-6.2.1>.\bin\logstash.bat
Sending Logstash's logs to D:/strumenti/logstash-6.2.1/logs which is now configured via log4j2.properties
[2018-02-14T21:39:11,618][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"D:/strumenti/logstash-6.2.1/modules/fb_apache/configuration"}
[2018-02-14T21:39:11,650][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"D:/strumenti/logstash-6.2.1/modules/netflow/configuration"}
[2018-02-14T21:39:13,776][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.1"}
[2018-02-14T21:39:15,068][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2018-02-14T21:39:15,573][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

and after the last message the process stops.

same output with specifying _path.settings_
D:\strumenti\logstash-6.2.1>.\bin\logstash.bat --path.settings D:/strumenti/logstash-6.2.1/config

config/pipelines.yml

- pipeline.id: pipeline-1
  path.config: "D:/strumenti/logstash-6.2.1/config/pipeline-1.yml"
- pipeline.id: pipeline-2
  path.config: "D:/strumenti/logstash-6.2.1/config/pipeline-2.yml"

running single configurations work
D:\strumenti\logstash-6.2.1>.\bin\logstash.bat -f config\pipeline-1.yml
or
D:\strumenti\logstash-6.2.1>.\bin\logstash.bat -f config\pipeline-2.yml

[cinhtau]

@abunet pipeline-1.yml is a logstash config or a yaml file? That doesn't seem right to me.

It seems to work if you use relative paths and remove the drive letters. Just investigating and collecting all details.

[abunet]

@cinhtau the content of pipeline-1.yml (for testing purpose) is:
input { stdin { } } output { stdout {} }

tried also changing the extension in pipeline-1.conf and pipeline-2.conf
remember that on linux the pipeline starts the two instances fine.

[cinhtau]

@abunet I think there is a little misconception from your side. The content is a logstash configuration, which is ok. You really shouldn't name it *.yml since the content is not YAML. I would recommend to name it *.conf or *.config, which you did. It's not about the file extension.

My team changed it to this and got it running. Assuming that you are in the Logstash home directory.

- pipeline.id: pipeline-1
  path.config: "./config/config-1.conf"
- pipeline.id: pipeline-2
  path.config: "./config/config-2.conf"

Still I would prefer that Elastic either mention that in their documentation or just fix absolute path.

[abunet]

Thanks @cinhtau,
sorry but the extension was incorrectly written .yml building the test files, but the difference is clear to me.

Thanks again

[cinhtau]

It must be documented appropriately. That's all.

[cheffe]

I faced the same issue. You may fix this, through adding a / before the drive letter. I got the configuration working with these paths:

- pipeline.id: pipeline-1
  path.config: "/D:/strumenti/logstash-6.2.1/config/pipeline-1.yml"
- pipeline.id: pipeline-2
  path.config: "/D:/strumenti/logstash-6.2.1/config/pipeline-2.yml"

[bencody]

Thanks, @cheffe, that solved it for me!

It's a shame that Logstash doesn't simply output a _could not find 'config-1.conf'_ error message that indicates that there was something wrong with the path.config property. It took me over an hour to resolve this trivial issue. >:(

[fuzolan]

windows example would be nice