Fluent-bit: filter_kubernetes: [warn] could not get meta for POD fluent-bit

[valentin2105]

Bug Report

Describe the bug
Fluent-bit doesn't launch well on my IPv6-only Kubernetes cluster

To Reproduce
Create k8s objects from your Kubernetes repo following the doc (serviceAccount,role,binding,config,ds...) and then inspect the fluent-bit pod's logs.

Expected behavior
Get my logs shipped to ES.

Your Environment

• Version used: fluent/fluent-bit:0.14.3
• Configuration: using different k8s Endpoint that 'svc.default.cluster.local..' (that work with curl)
• Environment name and version: Kubernetes 1.10.6, IPv6-only , Calico
• Server type and version: Debian 9.x up-to-date

Logs :

[2018/10/15 03:48:57] [ info] [engine] started (pid=1)
[2018/10/15 03:48:57] [ info] [filter_kube] https=1 host=k8s-api.domain.tld port=443
[2018/10/15 03:48:57] [ info] [filter_kube] local POD info OK
[2018/10/15 03:48:57] [ info] [filter_kube] testing connectivity with API server...
[2018/10/15 03:48:57] [error] [filter_kube] upstream connection error
[2018/10/15 03:48:57] [ warn] [filter_kube] could not get meta for POD fluent-bit-t2l45
[2018/10/15 03:48:57] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020

<nothing after that>

[edsiper]

For some reason, Fluent Bit cannot reach Kubernetes API server. If you ssh into Fluent Bit pod are you able to connect to https://k8s-api.domain.tld ?

[valentin2105]

@edsiper, Yes for sure, the connectivity work great without issue. My API endpoint is IPv6-only, maybe it's the issue and fluent-bit can't talk to it for any reason ?

[hhoover]

same issue here. Fluent-bit can't handle the IPv6 address for the API server, even if I provide a custom IP address in brackets for Kube_URL since it's parsing on colons:

[2018/10/25 14:59:25] [ info] [engine] started (pid=1)
[2018/10/25 14:59:25] [ info] [filter_kube] https=1 host=[fd00 port=1234
[2018/10/25 14:59:25] [ info] [filter_kube] local POD info OK
[2018/10/25 14:59:25] [ info] [filter_kube] testing connectivity with API server...
[2018/10/25 14:59:25] [ warn] net_tcp_fd_connect: getaddrinfo(host='[fd00'): Temporary failure in name resolution
[2018/10/25 14:59:25] [error] [filter_kube] upstream connection error
[2018/10/25 14:59:25] [ warn] [filter_kube] could not get meta for POD fluent-bit-bjnhq
[2018/10/25 14:59:25] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2018/10/25 14:59:25] [ warn] net_tcp_fd_connect: getaddrinfo(host='[fd00'): Temporary failure in name resolution
[2018/10/25 14:59:25] [error] [filter_kube] upstream connection error

[lth2015]

Fluent-Bit v0.14.9
Copyright (C) Treasure Data

[2018/12/27 03:10:32] [ info] [engine] started (pid=1)
[2018/12/27 03:10:32] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443
[2018/12/27 03:10:32] [ info] [filter_kube] local POD info OK
[2018/12/27 03:10:32] [ info] [filter_kube] testing connectivity with API server...
[2018/12/27 03:10:32] [ warn] [filter_kube] could not get meta for POD fluent-bit-6jpx2

I have the same problem.

When I use the v0.14.9 in my kubernetes cluster(version 1.9.3)，it works well. But, I create the same resource (the same yaml) in my kubernetes cluster(version 1.11.3), it's cannot works.

---

It's need rabc cofing, when I create role and rolebinding, it's works~

rolebinding
role

[MartinForReal]

If you enable hostNetwork, It is also important to bind the HOSTNAME env to pod.meta.name as follows:

          - name: HOSTNAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name

[patrickshan]

@MartinForReal any reason why we need to set up HOSTNAME environment variable using pod name when using hostNetwork ?

[MartinForReal]

please refer to
https://github.com/fluent/fluent-bit/blob/d97d51371c8da25cb102bdf2f619441a17f4bd26/src/flb_env.c#L50

[edsiper]

On IPv6 only environment the option ipv6 on must be set in the configuration.

A fix just landed on GIT master and this won't be necessary anymore, the fix will be part of 1.7 release on ~Feb 2021.

ref: c40c1e26eae100084d0747854c1c2a632d07a6cc