Fluent-bit: Not getting the journactl logs for kubelet.service in kuberenets cluster

Created on 30 Nov 2018  路  7Comments  路  Source: fluent/fluent-bit

Bug Report

I am trying to configure the "journalctl -u kubelet" logs to push to the file using fluentBit. But I am not getting data on to the file.

# Source: mls-fluentbit/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: "mls-fluentbit-logging-mls-fluentbit-kafka-tls-secret"
  namespace: mls-logging
  labels:
    app: mls-fluentbit-logging-mls-fluentbit
type: Opaque
---
# Source: mls-fluentbit/templates/config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: mls-fluentbit-logging-mls-fluentbit-config
  namespace: mls-logging
  labels:
    app: mls-fluentbit-logging-mls-fluentbit
data:
  fluent-bit.conf: |-
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf
    [INPUT]
        Name            systemd
        Tag             host.**
        Path         /var/run/systemd/journal/
        Systemd_Filter  _SYSTEM_UNIT=kubelet.service
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        use_journal         On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
    [FILTER]
        Name      record_modifier
        Match     kube.*
        Record  sourcetype kubernetes
        Record index stg_sde
    [FILTER]
        Name      modify
        Match     kube.*
        Rename    log    message

    [OUTPUT]
        Name  file
        Match *
        Path /tmp/fluent-bit.log

```apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: mls-logging
labels:
app: mls-fluentbit-logging-mls-fluentbit
spec:
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: mls-fluentbit-logging-mls-fluentbit
annotations:
checksum/config: 26fdeb4c1dad3292ef704b8f6e423af89004b05d2aae81361b38b95116598aae
spec:
serviceAccountName: mls-logging-admin-sa
containers:
- name: fluent-bit
image: "hub.docker.prod.walmart.com/fluent/fluent-bit:latest"
imagePullPolicy: "Always"
env:
[]

    resources:
      limits:
        memory: 200Mi
      requests:
        cpu: 100m
        memory: 100Mi

    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: varlibdockercontainers
      mountPath: /var/lib/docker/containers
      readOnly: true
    - name: config
      mountPath: /fluent-bit/etc/fluent-bit.conf
      subPath: fluent-bit.conf
    - name: journald-socket1
      mountPath: /run/systemd/journal/
      readOnly: true
    - name: journald-socket2
      mountPath: /var/run/systemd/journal
      readOnly: true

  terminationGracePeriodSeconds: 10
  volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
  - name: journald-socket1
    hostPath:
      path:  /run/systemd/journal/
  - name: journald-socket2
    hostPath:
      path:  /var/run/systemd/journal
  - name: config
    configMap:
      name: mls-fluentbit-logging-mls-fluentbit-config```

Files are present at the location:-

root@fluent-bit-74q68:/# ls -ltr /var//run/systemd/journal/ total 4 srw-rw-rw- 1 root root 0 Oct 2 08:20 dev-log srw-rw-rw- 1 root root 0 Oct 2 08:20 syslog srw-rw-rw- 1 root root 0 Oct 2 08:20 stdout srw-rw-rw- 1 root root 0 Oct 2 08:20 socket -rw-r--r-- 1 root root 8 Oct 2 08:20 kernel-seqnum -rw-r--r-- 1 root root 0 Oct 2 08:20 flushed drwxr-xr-x 2 root root 360 Nov 30 06:49 streams root@fluent-bit-74q68:/#

fixed question
Source
picture AkshayDubey29

Most helpful comment

When I run fluent-bit with config file, systemd input does nothing, no error, no output

picture tritruong on 28 Jan 2019
👍2

All 7 comments

I'm having the same issues. Not systems logs at all, no errors, nothing. Is someone looking into this?

ghost picture ghost on 12 Jan 2019

I'm having the same problem with systemd input (fluent-bit 1.0.3, Debian GNU/Linux 9 (stretch), kernel 4.9.0-7-amd64)

Systemd input only works if I run fluent-bit with command line options

kubectl exec -ti fluent-bit-2qp5r -n logging -- /fluent-bit/bin/fluent-bit -i systemd -p path=/var/log/journal/ -p systemd_filter=_SYSTEMD_UNIT=kubelet.service -p systemd_filter=_SYSTEMD_UNIT=docker.service -p tag='host.*' -p read_from_tail=true -o es -p Host=logging-elasticsearch-client -p Port=9200 -p logstash_format=on -p retry_limit=false -p time_key=@timestamp -p replace_dots=on -p logstash_prefix=kubernetes_cluster_log -f 1 -vv

Fluent Bit v1.0.3
Copyright (C) Treasure Data

[2019/01/28 06:43:25] [debug] [storage] [cio stream] new stream registered: systemd.0
[2019/01/28 06:43:25] [ info] [storage] initializing...
[2019/01/28 06:43:25] [ info] [storage] in-memory
[2019/01/28 06:43:25] [ info] [storage] normal synchronization mode, checksum disabled
[2019/01/28 06:43:25] [ info] [engine] started (pid=48)
[2019/01/28 06:43:25] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2019/01/28 06:43:25] [debug] [in_systemd] add filter: _SYSTEMD_UNIT=kubelet.service (or)
[2019/01/28 06:43:25] [debug] [in_systemd] add filter: _SYSTEMD_UNIT=docker.service (or)
[2019/01/28 06:43:25] [debug] [out_es] host=logging-elasticsearch-client port=9200 index=fluent-bit type=flb_type
[2019/01/28 06:43:25] [debug] [router] default match rule systemd.0:es.0
[2019/01/28 06:43:30] [debug] [task] created task=0x7f7bc58340c0 id=0 OK
[2019/01/28 06:43:30] [debug] [task] created task=0x7f7bc5834180 id=1 OK
[2019/01/28 06:43:30] [debug] [task] created task=0x7f7bc5834240 id=2 OK
[2019/01/28 06:43:30] [debug] [task] created task=0x7f7bc5834300 id=3 OK
[2019/01/28 06:43:30] [debug] [task] created task=0x7f7bc58343c0 id=4 OK

But when I run fluent-bit using config file, it doesn't work. Here is my config files

fluent-bit-filter.conf:
[FILTER]
    Name kubernetes
    Match kube.*
    Kube_URL https://kubernetes.default.svc:443
    Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log On
    K8S-Logging.Parser On
    K8S-Logging.Exclude On

fluent-bit-input.conf:
[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Parser docker
    Tag kube.*
    Refresh_Interval 5
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On
[INPUT]
    Name systemd
    Tag host.*
    Systemd_Filter _SYSTEMD_UNIT="docker.service"
    Systemd_Filter _SYSTEMD_UNIT="kubelet.service"
    Max_Entries 1000
    Read_From_Tail true
    Path /var/log/journal/

fluent-bit-output.conf:
[OUTPUT]
    Name es
    Match *
    Host logging-elasticsearch-client
    Port 9200
    Logstash_Format On
    Retry_Limit False
    Type flb_type
    Time_Key @timestamp
    Replace_Dots On
    Logstash_Prefix kubernetes_cluster_log

fluent-bit-service.conf:
[SERVICE]
    Flush 1
    Daemon Off
    Log_Level info
    Parsers_File parsers.conf

fluent-bit.conf:
@INCLUDE fluent-bit-service.conf
@INCLUDE fluent-bit-input.conf
@INCLUDE fluent-bit-filter.conf
@INCLUDE fluent-bit-output.conf

tritruong picture tritruong on 28 Jan 2019

When I run fluent-bit with config file, systemd input does nothing, no error, no output

tritruong picture tritruong on 28 Jan 2019
👍2

I had to specify /var/log/journal for the Path to get it to work, not sure but I think SD_JOURNAL_LOCAL_ONLY is not the right constant for CentOS/RHEL

chiefy picture chiefy on 1 Feb 2019

The issue from @AkshayDubey29 has to do with the path -- instead of /var/run/systemd/journal which does not contain journal logs and is also not mounted by fluent-bit, you likely want /var/log/journal.

The issue for @tritruong is different -- _SYSTEMD_UNIT values cannot be quoted so you should instead have:

    Systemd_Filter _SYSTEMD_UNIT=docker.service
    Systemd_Filter _SYSTEMD_UNIT=kubelet.service

If you are using the helm chart, I filed an issue and a PR to address it: https://github.com/helm/charts/issues/12161

flands picture flands on 13 Mar 2019

Faced a similar issue:

I ended up having to mkdir /var/log/journal on each k8s node to get the journald logs to persist. Setup of journald might vary depending on your Linux distro (using CentOS 7.2 in my case).

Ended up with the following config and it jumped to life, logging both the containers logs as well as the kubelet.service. 

input-kubernetes.conf: |
    [INPUT]
        Name              systemd
        Tag               kube.**
        Path              /var/log/journal
        Systemd_Filter    _SYSTEMD_UNIT=docker.service
        Systemd_Filter    _SYSTEMD_UNIT=kubelet.service
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

Hope this helps.

recipedude picture recipedude on 14 Jul 2019

main issue addressed, closing as fixed.

edsiper picture edsiper on 23 Jul 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

UladzimirSemiankou picture UladzimirSemiankou  路  3Comments
JavaCS3 picture JavaCS3  路  3Comments
jcdauchy-moodys picture jcdauchy-moodys  路  3Comments
mbelchin picture mbelchin  路  3Comments
huanggze picture huanggze  路  3Comments