Logstash: Improve warning messages about logstash.yml location and log4j2 config file

Created on 24 Jan 2017 · 21Comments · Source: elastic/logstash

Version: 5.1.2, deb installer
Operating System: Ubuntu 16.04
Steps to Reproduce:

When starting logstash from /usr/share/logstash dir with following command:

sudo bin/logstash -f ~/test.conf

It prints out:

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console

1) There is a logstash.yml in my /etc/logstash dir. The warning message above implies that if I had logstash.yml in /etc/logstash, there wouldn't be a problem. The docs suggest that logstash.yml should be in my $LS_HOME/config dir, so since I never set LS_HOME, it would make sense to get a warning message. However, if it's not okay to just have the logstash.yml in the /etc/logstash dir, the warning message should be changed.
2) The message about not being able to find log4j2 is odd in that it is looking for it in /usr/share/logstash/config. The default deb installation puts the log4j2 config file in /etc/logstash so Logstash should look in there as the default location.
3) The final sentences in each warning message should end in a period. :)

Source

peterskim12

👍7

Most helpful comment

+1 this is confusing.

bensebborn on 27 Apr 2017

👍3 😄2

All 21 comments

@peterskim12 You tried to start Logstash manually after installing via deb/rpm or other package managers. The right thing is to use sudo service logstash start with the config file stored in /etc/logstash/conf.d/. This way all your config is loaded from the right location.

Point taken about improving the message. ++

suyograo on 24 Jan 2017

Ok, yeah I figured if you're installing the deb or rpm version, it assumes certain things about how you're going to run it (e.g. running it as a service). It might not be clear to a new user that things may not act as expected when running the logstash executable directly. Thanks!

peterskim12 on 24 Jan 2017

👍1

Actually, it's one of:

systemctl start logstash for systemd
initctl start logstash for upstart
/etc/init.d/logstash start for SysV

It auto-detects and uses the appropriate method for the system.

untergeek on 24 Jan 2017

👍2

I have _logstash.yml_ in _/etc/logstash_. When I run from the command line (as sudo) I get the error "_Could not find logstash.yml ... You can specify the path using --path.settings._"

I then add _--path.settings=/etc/logstash/logstash.yml_ and get the same error.

Turns out I need to add the flag _--path.setting=/etc/logstash_, then it works. I hope this helps someone.

hatdropper1977 on 13 Apr 2017

👍5

I tried the following 2 commands:
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash/logstash.yml -f /etc/logstash/conf.d/status.conf and
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings -f /etc/logstash/conf.d/status.conf

Both the commands did not help to solve this warning: WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /etc/logstash/logstash.yml/log4j2.properties. Using default config which logs to console.

ajaycj on 21 Apr 2017

Ajay, change your command from --path.settings=/etc/logstash/logstash.yml to --path.settings=/etc/logstash (Leave off the logstash.yml) and it will work.

hatdropper1977 on 21 Apr 2017

👍2

+1 this is confusing.

bensebborn on 27 Apr 2017

👍3 😄2

Hello Elastic team, Quick question, how I can see the verbose of Logstash when is started as a service?
Ubuntu server 16.04 Logstash 5.4.1
Thank you so much

Nebadon1 on 3 Jun 2017

@untergeek @suyograo so this is just a WARN message but the file seems to be picked up. Do we need to enhance the message or the logic?

gmoskovicz on 12 Jun 2017

"just restart with the service file" is not a suitable answer for testing config changes before you restart the service file.

Had to include the path.settings below in order for it to work. While path.settings is mentioned in the current error message, it also says it couldn't find a file at the default location when there is one there.
/usr/share/logstash/bin/logstash --path.settings=/etc/logstash -t

@gmoskovicz for testing rather than starting the service, it fails to work without path.settings.

root@mayfly:/etc/logstash# ls $PWD/logstash.yml
/etc/logstash/logstash.yml

root@mayfly:/etc/logstash# time /usr/share/logstash/bin/logstash -t
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
ERROR: No configuration file was specified. Perhaps you forgot to provide the '-f yourlogstash.conf' flag?
usage:
-- snip --

(logstash 5.3)

vacri on 13 Jul 2017

@hatdropper1977
I tried this command and my logstash service is active.
sudo -u logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -f /etc/logstash/conf.d/logstash.conf

But it gave me this error. Could you help me with this.
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties

rasleens on 19 Jul 2017

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties

This error can be ignored, and will no longer be seen in the next version of Logstash.

Note - this is a different error and root cause then the OP's issue.

jakelandis on 19 Jul 2017

👍3

Hi, I am using logstash 5.5.1 and I am still getting this message.

Leluke on 11 Aug 2017

@Leluke - I mis-spoke about the 'next version'. This will part of 5.6+. Sorry for the mis-information.

jakelandis on 11 Aug 2017

@jakelandis I am using 5.6.3 and still get the message.

rustigano on 5 Nov 2017

@tobiasbeuving - can you provide a bit more about your environment (specifically installation method) and an exact copy of the message ?

For me, fresh unzip of tar.gz:
Sending Logstash's logs to /Users/jake/workspace/releases/logstash-5.6.3/logs which is now configured via log4j2.properties
(this is the expected output)

jakelandis on 6 Nov 2017

@jakelandis For the record: The message is not bothering me, everything is running fine, having said that:

About my environment:

OS: Ubuntu 16.04 64bit

I installed it with APT as described here:
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html#package-repositories

Java:

tobias@tobias-ubuntu-virtualBox:~$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

The message is:

tobias@tobias-ubuntu-virtualBox:~$ sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-mysql-import.conf 

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console

Cheers!

rustigano on 6 Nov 2017

It's working in Logstash 6.0 when i used @hatdropper1977 approach. Thanks.

$> sudo -u logstash /usr/share/logstash/bin/logstash --path.settings=/etc/logstash -f bin/LogstashPipelineCSV.conf

muthhus on 3 Dec 2017

@jakelandis I am using 6.1.1-1 and still get the message.

Installed via .deb (apt)
ubuntu 16.04 64bit

followed same as @tobiasbeuving did.
https://www.elastic.co/guide/en/logstash/current/installing-logstash.html#package-repositories

(and same error message as tobias had)

fyi JAVA version

openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

DanielTimLee on 1 Jan 2018

For logstash 6.7.1, and installed via .deb package:

cd /usr/share/logstash/bin && sudo -u logstash ./logstash -f '/etc/logstash/conf.d/beats-input-glassfish.conf' --path.settings='/etc/logstash/'
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-06-06T22:13:25,279][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-06T22:13:25,318][FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[2019-06-06T22:13:25,334][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

martinmurciego on 7 Jun 2019

For anyone using logstash-keystore or similar commands, you must put options before the commands, and you MUST NOT use the equals sign with the option: logstash-keystore --path.settings /etc/logstash list

I'm not sure if there should be a separate error for what I was doing, or if this is part of the same issue with the "could not find logstash.yml" not being clear.