Fluent-bit: in_tail: how to collect docker container logs?

[alanchchen]

With docker image fluent/fluent-bit:v0.11-dev and here is the command I used

fluent-bit -i tail \
  -p path=/var/lib/docker/containers/*/*.log  \
  -o es://$ELASTICSEARCH_HOST:$ELASTICSEARCH_PORT 

I tried
path=/var/lib/docker/containers/*/*.log
path=/var/lib/docker/containers/*.log
path=/var/lib/docker/containers/**/*.log

None of them work.
Any ideas? Thanks

[edsiper]

Note: 0.11 is under ACTIVE development.

Note that when Fluent Bit receives a property (plugin configuration key), if you are going to use a wildcard try to use single quotes, e.g:

fluent-bit -i tail \
  -p 'path=/var/lib/docker/containers/*/*.log' 
  -o es://$ELASTICSEARCH_HOST:$ELASTICSEARCH_PORT 

If you don't use single/double quotes, the shell is going to expand the wildcard.

Let me know how that goes.

[alanchchen]

@edsiper , I'm running fluent-bit on Kubernetes
I tried what you've said.

fluent-bit -v -i tail  -p 'path=/var/lib/docker/containers/*/*.log' -o es://$ELASTICSEARCH_HOST:$ELASTICSEARCH_PORT  -o stdout

$ kubectl logs -f fluentd-2gbs3
Fluent-Bit v0.10.0
[2017/02/11 02:04:22] [ info] [engine] started
[2017/02/11 02:04:22] [debug] [in_tail] inotify watch fd=17
[2017/02/11 02:04:22] [debug] [in_tail] scanning path /var/lib/docker/containers/*/*.log
[2017/02/11 02:04:22] [debug] [es] host=elasticsearch.development port=9200 index=fluentbit type=test
[2017/02/11 02:04:22] [ warn] [router] NO match for es.0 output instance
[2017/02/11 02:04:22] [ warn] [router] NO match for stdout.0 output instance

I'm not sure about this error but I think the container is running as root and I did mount the volume.

$ kubectl exec fluentd-2gbs3 -- sh -c 'ls /var/lib/docker/containers'
ls: can't open '/var/lib/docker/containers': Permission denied

Will fluent-bit print such errors if failed to access?

[edsiper]

The warning is here:

[2017/02/11 02:04:22] [ warn] [router] NO match for es.0 output instance

Each input source have a _tag_, if not defined a default one is assigned. If you assigned two destinations (stdout and elasticsearch) you __must__ define a __match__ rule. For testing you can do:

fluent-bit -v -i tail  -p 'path=/var/lib/docker/containers/*/*.log' \
        -o es://$ELASTICSEARCH_HOST:$ELASTICSEARCH_PORT  -m '*' \
        -o stdout -m '*'

[alanchchen]

@edsiper

$ kubectl logs -f fluentd-vpccx    
Fluent-Bit v0.10.0
[2017/02/11 02:29:59] [ info] [engine] started
[2017/02/11 02:29:59] [debug] [in_tail] inotify watch fd=17
[2017/02/11 02:29:59] [debug] [in_tail] scanning path /var/lib/docker/containers/*/*.log
[2017/02/11 02:29:59] [debug] [es] host=elasticsearch.development port=9200 index=fluentbit type=test
[2017/02/11 02:29:59] [debug] [router] match rule tail.0:es.0
[2017/02/11 02:29:59] [debug] [router] match rule tail.0:stdout.0

Now the match rules are applied.
I think should see some logs at least from stdout.
There must be something I missed.

[edsiper]

I've got the point, there is no error if the file cannot be opened (permission issues).

I logged #186 to fix that early next week, sorry for that. Please try to run fluent bit as root or a user that can read the files for now.

[alanchchen]

Actually I'm still not sure if it's about permission issues.
I'll do more tests. Thank you.

[alanchchen]

@edsiper
I solved my issues by applying chown -R 1000:1000 /var/lib/docker/containers before running fluent-bit.

Sorry for bothering and thank you :)

[edsiper]

thanks for the update.

I pushed a fix (#186) to gather some errors when some permission problem happens.

[happyyangyuan]

if the file does not exist, there is no warning or error with "Fluent Bit v0.13.0" either.