Fluent-bit: How to Configure fluentbit with journald input, kubernetes filter

[jenningsloy318]

Hi All,

I have created a kubernetes cluster, wanna use flunetbit+influxdb+grafana for log processing. right now there seems some issue with configure the journald with the input, and kubernetes as the filter and docker as the parser, finally save the logs into influxdb.

journald: enable this log driver in docker service

fluentbit: manually buid the docker image with 0.12 branch, and modify the Dockerfile, install libsystemd-dev, then it can build journald support.

fluent-bit.conf

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File fluentbit-parser.conf

[INPUT]
    Name           systemd
    Tag             *
    Systemd_Filter  _SYSTEMD_UNIT=docker.service
    Parser        docker
[FILTER]
    Name          kubernetes
    Kube_URL      https://kubernetes.default.svc:443
    Kube_CA_File  /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
[OUTPUT]
    Name          influxdb
    Match         *
    Host          monitoring-influxdb
    Port          8086
    Database      k8slog
    Sequence_Tag  _seq

fluentbit-parser.conf

[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep   On

when start fluentbit, got error

 NO match rule for kubernetes.0 filter instance

Can anybody help me to figure out how to configure the correct kubernetes filter?
also the default parser is json, but here I use journald, how to implement it as journald format?

[nokute78]

'Match' is needed at [FILTER] section.
Could you add 'Match *' at [FILTER] section?

[jenningsloy318]

@nokute78

I added 'Match *' at [FILTER] into [FILTER] section, this error is gone. but another issue appers,

[2017/07/24 02:23:07] [ info] [engine] started
[2017/07/24 02:23:07] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443
[2017/07/24 02:23:07] [ info] [filter_kube] local POD info OK
[2017/07/24 02:23:07] [ info] [filter_kube] testing connectivity with API server...
[2017/07/24 02:23:07] [error] [filter_kube] could not get meta for POD fluent-bit-v2rq0

which shows that it can't get the metadata from apiserver, my k8s cluster is RBAC-enabled and fluentbit is running in monitoring namespace not kube-system, is there any configure that makes it possible to get metadata from all namespace?

[edsiper]

Assuming you are using the image fluent/fluent-bit:0.12-dev, you have to append the following configuration property to the Kubernetes filter:

[FILTER]
    name         kubernetes
    match        *
    use_journal  On
    ...

[jenningsloy318]

still got same errors:

log:

# kubectl logs fluent-bit-ngxjs -n monitoring
[2017/07/25 01:48:47] [ info] [engine] started
[2017/07/25 01:48:47] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443
[2017/07/25 01:48:47] [ info] [filter_kube] local POD info OK
[2017/07/25 01:48:47] [ info] [filter_kube] testing connectivity with API server...
[2017/07/25 01:48:47] [error] [filter_kube] could not get meta for POD fluent-bit-ngxjs

POD conf

# hostname
fluent-bit-ngxjs
# cat fluent-bit.conf
[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File fluentbit-parser.conf
[INPUT]
    Name           systemd
    Tag             *
    Systemd_Filter  _SYSTEMD_UNIT=docker.service
    Parser        docker
[FILTER]
    Name          kubernetes
    Kube_URL      https://kubernetes.default.svc:443
    Kube_CA_File  /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
    Match       *
    use_journal  On
[OUTPUT]
    Name          influxdb
    Match         *
    Host          monitoring-influxdb
    Port          8086
    Database      k8slog
    Sequence_Tag  _seq

[edsiper]

I have not tested a cluster with RBAC enabled but looking at Tectonic Logging docs, they deploy Fluentd with the following roles (can be used as a reference):

• https://coreos.com/tectonic/docs/latest/files/logging/fluentd-service-account.yaml
• https://coreos.com/tectonic/docs/latest/files/logging/fluentd-role.yaml
• https://coreos.com/tectonic/docs/latest/files/logging/fluentd-role-binding.yaml

refs from: https://coreos.com/tectonic/docs/latest/admin/logging.html

[jenningsloy318]

My configuration is same as this one, but no working at this moment. can we have more detauls debug info from fluentbit? more details configuration about the kubernetes scenario ?

[edsiper]

sure, I've just pushed a commit that add a debug message from the API server on failure:

• set log_level to debug
• pull image from fluent/fluent-bit:0.12-dev (it's still building now, give it 20 mins)

[jenningsloy318]

for kubernetes filter issue, it is caused by the insufficient permission when use earlier clusterrole. now I changed it. not it can connect to apiserver.

I can move the next step, for the parser and output configuration.

I still don't find any data in influxdb.
just wonder if the parser matters, since there isn't Time_Key found in journal raw log or get from fluentbit input plugin.

[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep   On

for a standard journal log

• raw log from journal, use journalctl CONTAINER_ID=xxx

Jul 25 22:04:29 cnpvgl56588417 dockerd[23332]: I0725 14:04:29.002171       7 pathrecorder.go:247] kube-aggregator: "/api/v1/namespaces/kube-system/endpoints/kube-dns" satisfied by prefix /api/

• log from fluent-bit, with stdout as the output plugin

[1224] docker: [1500965638.000000771, {"PRIORITY"=>"3", "_TRANSPORT"=>"journal", "_PID"=>"23332", "_UID"=>"0", "_GID"=>"0", "_COMM"=>"dockerd", "_EXE"=>"/usr/bin/dockerd", "_CMDLINE"=>"/usr/bin/dockerd", "_CAP_EFFECTIVE"=>"3fffffffff", "_SYSTEMD_CGROUP"=>"/container.slice/docker.service", "_SYSTEMD_UNIT"=>"docker.service", "_SYSTEMD_SLICE"=>"container.slice", "_BOOT_ID"=>"77099da9b6ed46c7ae9e0b3ddc89aeb9", "_MACHINE_ID"=>"94b52f20059d43be9b0055ffbd24b4e6", "_HOSTNAME"=>"cnpvgl56588417", "CONTAINER_ID"=>"8d0afcd9275c", "CONTAINER_ID_FULL"=>"8d0afcd9275c9b7c1eb73c40b0b8f14fe2b13c5b1be5035a1f69b72b908f5470", "CONTAINER_NAME"=>"k8s_kube-apiserver_kube-apiserver-cnpvgl56588417_default_2673c496a3acd9ba32e6f33c6b4e17ce_1", "CONTAINER_TAG"=>"docker_k8s_kube-apiserver_kube-apiserver-cnpvgl56588417_default_2673c496a3acd9ba32e6f33c6b4e17ce_1_8d0afcd9275c", "MESSAGE"=>"I0725 06:53:58.768086       7 pathrecorder.go:247] kube-aggregator: "/api/v1/namespaces/kube-system/endpoints/kube-dns" satisfied by prefix /api/", "_SOURCE_REALTIME_TIMESTAMP"=>"1500965638768152"}]

[edsiper]

actually the Systemd input plugin do not use a Parser because the logs are already structured by Journald.

Some suggestions:

• set a database file (DB parameter): http://fluentbit.io/documentation/0.12/input/systemd.html
• verify that Fluent Bit container can reach monitoring-influxdb:8086

[jenningsloy318]

when I execute journalctl inside the pod, no ouput. seems it don't find the journald socket
but use journalctl -D /var/log/journal inside the pod, I can get the logs, I already share the sockets and log directory between pod and host.

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: monitoring
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccountName: fluentbit
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:0.12-dev
        command: ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/etc/fluent-bit.conf","--verbose"]
        env:
        resources:
          limits:
            memory: 100Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: journald-socket1
          mountPath: /run/systemd/journal/
          readOnly: true
        - name: journald-socket2
          mountPath: /var/run/systemd/journal
          readOnly: true
        - name: fluentbit-conf
          mountPath: /fluent-bit/etc/
          readOnly: true
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: journald-socket1
        hostPath:
          path:  /run/systemd/journal/
      - name: journald-socket2
        hostPath:
          path:  /var/run/systemd/journal
      - name: fluentbit-conf
        configMap:
          name: fluentbit-conf

Another thing: I think there is a need to display more details when fluentbit is checking the journald socket or journal logs

[edsiper]

@jenningsloy318

I've added to GIT master (fluent/fluent-bit:0.12-dev) the 'path' parameter so you can specify where your journal logs exists, please try that and let me know if it works.

[edsiper]

I've updated the plugin with some improvements:

• speed up consumption of old logs
• handle of mem_buf_limit case

Documentation has been updated: http://fluentbit.io/documentation/0.12/input/systemd.html

[jenningsloy318]

@edsiper

Thank you so much. now I can see logs in influxdb.

As you said, add Path resolved this issue. but I wonder in code, where does fluentbit find the journal logs, which socket it connects

I another thing, in repo fluent-bit-docker-image, the Dockerfile of 0.12-dev seems don't have libsystemd-dev package installed, thus compiled fluentbit don't have journald/systemd support.

I managed to use my docker images to resove this issue by modifying the Dockerfile.

[edsiper]

@jenningsloy318

thanks for the hints for the Docker image.

fluent/fluent-bit:0.12-dev has been updated, it should be OK now.

[edsiper]

Fixed.