Beats: When journalbeat logs to journal, journalbeat can process own logs, can lead to loops

[mattpr]

Summary:

when journalbeat logs to journal, journalbeat can consume its own logs which can lead to a loop.

For instance journalbeat logs journalbeat successfully published 1 events to journal which is then read by journalbeat.

Version:

6.6.1 (tarball)

Operating System:

Ubuntu 16.04

Discuss Forum URL:

https://discuss.elastic.co/t/journalbeat-input-filtering-journalbeat-processing-its-own-logs-and-then-logging-that-it-processed-them/171513

Steps to Reproduce:

• Run journalbeat via systemd unit

#...
ExecStart=/opt/journalbeat/current/journalbeat run -e
StandardOutput=journal
StandardError=journal
#...

• journalbeat.yml config

#...
journalbeat.inputs:
- paths: []
#...
# drop log lines from journalbeat using processor
processors:
  - drop_event:
      when:
        equals:
          process.name: "journalbeat"

• despite events/log-lines from journalbeat being dropped, they are still logged as "processed" to the journal like so:

Mar 08 14:15:56 somehostname journalbeat[10995]: 2019-03-08T14:15:56.833Z INFO [input] input/input.go:133 journalbeat successfully published 1 events {"id": "2d121d9b-6458-40ef-a9b5-f642c0218916"}

• note that each of these "published 1 events" log lines itself is then processed by journalbeat leading to another one.
• the drop_event processor prevents these from being sent to logstash or elastic search but still really fills up the logs.

Work-arounds

1. Disable journal logging for journalbeat by changing the systemd unit to remove the -e flag and journal logging for stderr/out.

#...
ExecStart=/opt/journalbeat/current/journalbeat run
#StandardOutput=journal
#StandardError=journal
#...

1. Use journalbeat.inputs > include_matches to filter input. Presumably log lines filtered at input stage won't be logged about (compare to log events dropped by processor).
2. Use journalbeat.inputs > paths combined with journal config changes to filter which journal files/directories are used as input (idea being you could configure journalbeat to use journal but log to a different journal/file which could be excluded from input).

Thoughts

• Excluding all journalbeat logs from journalbeat might be too drastic. After all if journalbeat is generating errors, we might want to send those to logstash/es if we can.
• This problem seems to come up with "status" type log messages (ie "processed 3 log lines").
  Trying to avoid that developers ever add this type of logging doesn't seem sustainable and the issue would probably come up over and over...but could be somewhat covered with test cases.
• Better input filtering might be one approach...if we could for instance say exclude_matches that match unit journalbeat and are below a certain severity (ie exclude everything from journalbeat that isn't WARN or ERROR). However if journalbeat is erroring trying to send logs to logstash and then logs the error...that error presumably is "severe" and so journalbeat would try to send that error resulting is another error/loop condition.

In my case I stopped journalbeat from logging to journal to avoid this completely. However that decision means that if journalbeat is erroring...that needs to be detected though other monitoring as journalbeat logs aren't collected. If error condition is detected (for instance absence of log data from given host for >X time) then someone has to go look at the journalbeat logs on that box.

[kvch]

This issue is due to the logging configuration of Beats on systemd. See more on setting logging: https://www.elastic.co/guide/en/beats/journalbeat/master/running-with-systemd.html#_customize_systemd_unit_for_journalbeat

[opoplawski]

Any progress here?

[chrisnew]

This just comes with insane defaults out of the box imho. I also ran into the same issue that journalbeat is forwarding its own log messages.

My workaround is basically the following:

logging.level: warning
processors:
  - drop_event:
      when:
        equals:
          systemd.unit: "journalbeat.service"

[Dreamerset]

I would love to see exclude_matches option as suggested by Matt Probert.