Logstash: Logstash won't start with systemd / error with logging?

[mrjameshamilton]

Since today, I cannot start Logstash. As far as I know nothing changed.

systemctl start logstash results in starting Logstash but when it gets to initialise the logger it fails:

mai 09 11:09:19 cern.ch logstash[15115]: Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$R...olling.Rolli mai 09 11:09:19 cern.ch logstash[15115]: at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:75) mai 09 11:09:19 cern.ch logstash[15115]: at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:81) mai 09 11:09:19 cern.ch logstash[15115]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:103) mai 09 11:09:19 cern.ch logstash[15115]: at org.apache.logging.log4j.core.appender.RollingFileAppender.createAppender(RollingFileAppender.java:191) mai 09 11:09:19 cern.ch logstash[15115]: ... 105 more mai 09 11:09:19 cern.ch logstash[15115]: 2017-05-09 11:09:19,291 main ERROR Null object returned for RollingFile in Appenders. mai 09 11:09:19 cern.ch logstash[15115]: 2017-05-09 11:09:19,291 main ERROR Null object returned for RollingFile in Appenders. mai 09 11:09:19 cern.ch logstash[15115]: 2017-05-09 11:09:19,291 main ERROR Unable to locate appender "plain_rolling" for logger config "root"

And there's nothing in the log file.

it works if I run it like

/usr/share/logstash/bin/logstash --path.settings=/etc/logstash/

[untergeek]

What OS? What version of Java? What does the rest of your configuration (logstash.yml, etc.) look like?

[mrjameshamilton]

In the end, I uninstalled and re-installed Logstash. Using the same config files, there is now no problem. I don't understand why it broke though.

[tobiasfielitz]

Same problem over here and no solution :-(

[rugenl]

Ok, same problem. I suspect that this happens after logstash runs once as root, I think it's changing ownership of a file, then logstash fails due to permission. I just can't find the file :-(

[rugenl]

DOH, it's logstash-plain.log

It was owned by root, remove it, systemctl start works.

[Sphinxs]

Same here

[Asey80]

Same here, but - removing of logstash-plain.log (owner root) and granting access to 'queue' and 'dead_letter_queue' are solve the issue in my case, like:
chmod 775 /var/lib/logstash/queue
chmod 775 /var/lib/logstash/dead_letter_queue

good luck!

[gnujeremie]

@rugenl thanks a lot, it solved my problem.
I ran logstash on debian directly to test my config files, and then running using systemctl or service command failed. Removing the only logstash log file owned by root fixed it. 👍

[krishanranditha]

Double check whether you have killed existing logstash.
In my case PID was there even if I killed before. After kill that running PID, logstash started successfully.

$ ps -ef | grep logstash

dev 16820 11257 0 05:16 pts/0 00:00:00 grep --color=auto logstash
dev 50216 1 2 May10 ? 03:46:27 /bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/google-java-format-1.1.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/guava-19.0.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/jackson-annotations-2.9.1.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/jackson-core-2.9.1.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/jackson-databind-2.9.1.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.1.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/janino-3.0.8.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/log4j-api-2.9.1.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/log4j-core-2.9.1.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/logstash-core.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/home/dev/elk/logstash-6.2.4/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash -f kafkaLogstas

[jjdelc]

Had the same error, the problem was that the path log in elasticsearch.yml had a typo.