Beats: Journalbeat

[kvch]

Journalbeat is going to be introduced, so Elastic stack users can forward journal entries to outputs.

Features

• Seeking to head, tail or to cursor
• Exact match for fields of journal entries using include_matches option
• ...

Configuration

journalbeat.inputs:
  # Paths that should be crawled and fetched.
  # When empty starts to read from local journal.
- paths: []

  # The number of seconds to wait before trying to read again from journals.
  #backoff: 1s
  # The maximum number of seconds to wait before attempting to read again from journals.
  #max_backoff: 20s

  # Position to start reading from journal. Possible values: head, tail, cursor
  seek: cursor

  # Exact matching for field values of events.
  # Matching for nginx entries: "systemd.unit=nginx"
  #include_matches: []

#========================= Journalbeat global options ============================
#journalbeat:
  # Name of the registry file. If a relative path is used, it is considered relative to the
  # data path.
  #registry_file: registry

TODO

6.5

• [x] README
• [x] add fields.yml && index template #8277
• [x] ability to add filters to inputs #8324
• [x] simple E2E tests #8277
• [x] more comlex E2E tests with registry #8277
• [x] investigate errors during reading (operation not permitted)
• [x] adding journald specific metrics (e.g. journal size) #8324
• [x] adding dashboard
• [x] add Jenkins job
• [x] add Dockerfile to build #8324
• [x] support processors #8324
• [x] add Fields support #8324
• [x] convert numbers in events #8618
• [x] tie version to vendored lib #8618

6.6 (needs backward-compatibility)

• [x] prefix fields with journald.
• [x] add Dockerimage https://github.com/elastic/beats-docker/pull/58
• [x] add cursor_seek_fallback #9234

later

• [ ] support filtering of entries like community Journalbeat: unit, kernel, identifier: https://github.com/elastic/beats/pull/10985
• [ ] support for multiline and other readers of Filebeat
• [ ] support autodiscovery
• [ ] support input file globbing: https://github.com/elastic/beats/issues/21128
• [ ] support more boolean expressions in case of include_matches
• [ ] support wildcards in include_matches

[kvch]

Example event

{
  "@timestamp": "2018-09-05T16:51:46.798Z",
  "@metadata": {
    "beat": "journalbeat",
    "type": "doc",
    "version": "7.0.0-alpha1"
  },
  "read_timestamp": "2018-10-01T06:57:16.424Z",
  "beat": {
    "name": "sleipnir",
    "hostname": "sleipnir",
    "version": "7.0.0-alpha1"
  },
  "process": {
    "uid": "0",
    "name": "dhclient",
    "executable": "/sbin/dhclient",
    "cmd": "dhclient -i wlp4s0",
    "capabilites": "3fffffffff",
    "audit": {
      "login_uid": "1000",
      "session": "1"
    },
    "pid": "10145"
  },
  "syslog": {
    "identifier": "dhclient",
    "pid": "10145",
    "priority": "6",
    "facility": "3"
  },
  "systemd": {
    "slice": "user-1000.slice",
    "owner_uid": "1000",
    "user_slice": "-.slice",
    "cgroup": "/user.slice/user-1000.slice/session-1.scope",
    "unit": "session-1.scope",
    "invocation_id": "95e83dc146c446c5880c56270101993c",
    "session": "1",
    "transport": "syslog"
  },
  "host": {
    "boot_id": "a5b9a1fe16874501b7552bbcc5efc911",
    "name": "sleipnir",
    "id": "f53e8c67094c443c88c2f5e1f130ceb6"
  },
  "message": "bound to 192.168.0.80 -- renewal in 1620 seconds.",
}

[kvch]

Example configurations

Monitoring multiple journals under the same directory

Journals under /path/to/journal/directory are merged into a single journal and read. By setting cursor to seek, Journalbeat reads from the beginning of the journal. Then after reload/restart it continues from where it left off.

journalbeat.inputs:
- paths: ["/path/to/journal/directory"]
  seek: cursor

Get redis events from a Docker container tagged redis

Using the translated fields by Journald.

journalbeat.inputs:
- paths: []
  include_matches:
    - "container.image.tag=redis"
    - "process.name=redis"

Using the field names of systemd journal.

journalbeat.inputs:
- paths: []
  include_matches:
    - "CONTAINER_TAG=redis"
    - "_COMM=redis"

[tardyp]

Hi, how is this going on? looks like the docker images are not available anymore in the docker repository.

[kvch]

There was a minor hiccup when publishing the Journalbeat image. It is resolved now.

[jalaziz]

Are there any plans to support complete unit matching as was done with the community beat or should I file a separate issue? (see https://github.com/mheese/journalbeat/commit/2989e18a4e60c7457340e0bf57c5562a241d354c)

It seems that matching on systemd.unit doesn't necessarily capture all the logs that journalctl -u does.

[kvch]

Yes, it is planned. But first, a minimal matching is coming then the support for the full filtering functionality of the community Beat.
There is one PR in progress with the next step towards the goal: https://github.com/elastic/beats/pull/10985

[urso]

@kvch is this meta ticket up to date?

[kvch]

Now yes. I have added the last Journalbeat PR: https://github.com/elastic/beats/pull/10985

[sepich]

Is there any date for multiline: support?
Right now (7.4.0) it just silently does not work, should there be at least some not implemented warning in the logs?
Thanks

[kvch]

@sepich What do you mean by silently not working? Is this option documented somewhere?

[jbguerraz]

@kvch we're ready to basically port (understand, mostly copy/paste :D) https://github.com/elastic/beats/pull/570/files to journalbeat
Is that the way you had (or, you would have) in mind ? if so, we can tackle it, otherwhise, could you provide some guidance ? :)

[kvch]

Our plan in the long term is to merge Journalbeat into Filebeat as an input. When merging we would like to provide the same options for the new input as log. This mostly depends on the PR https://github.com/elastic/beats/pull/12908.
@jbguerraz I will get back to you in a few days about short term plans. Thank you for your patience.

[jbguerraz]

Hello @kvch :)
Is there a way to help you move ahead ?
Have a great day!

[kvch]

@jbguerraz I am working on a proof of concept for what I have in mind for addressing multiline. However, it involves a somewhat bigger refactoring, so it might take a few days. I will ping you in the PR.

[jbguerraz]

Hello @kvch :)
What's up ? any way to provide some help on the matter ?
Thank you!

[kvch]

I have started a POC but I ended up taking a step back because it Filebeat handles events differently than other Beats. I have opened a proposal for refactoring the pipelines: https://github.com/elastic/beats/issues/16137

But there are a few open questions.

[q2dg]

Hello.
I see you keep "support more boolean expressions in case of include_matches" task as pending.
Currently, there's an implicit OR between several elements of include_matches array, right?
What do you mean is adding some sort of AND? Anything more specific?
Thanks!!

[kvch]

ATM you can only link conditional expressions with OR when reading from the journal. My plan is to add support for AND.

In the meantime, you can use the processors of Journalbeat to drop events if it does not fit your requirements just like in case of other Beats. For example, you can drop events from the dbus-daemon with the following processor configuration:

processors:
- drop_event:
    when:
       equals:
           journald.process.name: dbus-daemon