Falco: Falco stopped annotate K8s containers

Created on 30 Sep 2020 · 5Comments · Source: falcosecurity/falco

Describe the bug

Falco running 5d on cluster stopped annotate k8s workload with k8s metadata. Possibly K8s was restarted or certificates on api server re-newed.

The bug on falco side is that it does not throw any error if K8s connectivity is lost. Thus events start to fire as they do not match rule.

See

image=argoproj/argoexec:v2.8.2) k8s.ns=<NA> k8s.pod=<NA> container=02e889f6b2dd k8s.ns=<NA> k8s.pod=<NA>

^^ I only run argocd in k8s (not on the host). The %container.info was used to annotate output.

Happened to all falco nodes, when DaemonSet restarted, K8s workload was again properly annotated ie: namespace/pod ...


Full log:
{"output":"10:43:47.900648203: Warning Docker or kubernetes client executed in container (user=root k8s.ns=<NA> k8s.pod=<NA> container=02e889f6b2dd parent=argoexec cmdline=docker inspect 2ef419197776dc547c3613b6f5a4fe5ff3a667968d8e98bd4e530e7cf92f5611 --format='{{.State.ExitCode}}' image=argoproj/argoexec:v2.8.2) k8s.ns=<NA> k8s.pod=<NA> container=02e889f6b2dd k8s.ns=<NA> k8s.pod=<NA> container=02e889f6b2dd k8s.ns=<NA> k8s.pod=<NA> container=02e889f6b2dd k8s.ns=<NA> k8s.pod=<NA> container=02e889f6b2dd k8s.ns=<NA> k8s.pod=<NA> container=02e889f6b2dd k8s.ns=<NA> k8s.pod=<NA> container=02e889f6b2dd k8s.ns=<NA> k8s.pod=<NA> container=02e889f6b2dd","priority":"Warning","rule":"The docker client is executed in a container","time":"2020-09-30T10:43:47.900648203Z", "output_fields": {"container.id":"02e889f6b2dd","container.image.repository":"argoproj/argoexec","container.image.tag":"v2.8.2","evt.time":1601462627900648203,"k8s.ns.name":null,"k8s.pod.name":null,"proc.cmdline":"docker inspect 2ef419197776dc547c3613b6f5a4fe5ff3a667968d8e98bd4e530e7cf92f5611 --format='{{.State.ExitCode}}'","proc.pname":"argoexec","user.name":"root"}}

How to reproduce it

Not sure what happen in my case. Not a single cluster owner.
Possibly, (not tested)

K8s api inaccessible
K8s not responding properly (ie falco can reach host, but is not giving http 200)

Expected behaviour

metric for failed k8s connection (so livenes probe can track + can be highlighted in monitoring)
retry connection
or even falco to exit (ie container is refreshed)

Find a way not to fire Rules that are expected for container, but seems as on host due this behaviour.

Note, Falco can from docker metadata recognize K8s containers, note:

84ee8c9ca0dc        k8s.gcr.io/pause-amd64:3.1                       "/pause"                 2 months ago        Up 2 months                             k8s_POD_azure-cni-networkmonitor-gq2bb_kube-system_9e6a99f1-214f-43f7-bdff-a68bcf45aaab_2

Additional context

as rules are not matched, simply any container activity (write under root, /etc) becomes an event and through process.cmdline can leak secrets to log subsystem, etc.. that would be in other cases would never hit an Rule.

Falco version:

Falco version: 0.25.0
Driver version: ae104eb20ff0198a5dcb0c91cc36c86e7c3f25c7

make -j8 KERNELRELEASE=4.15.0-1092-azure -C /lib/modules/4.15.0-1092-azure/build M=/var/lib/dkms/falco/ae104eb20ff0198a5dcb0c91cc36c86e7c3f25c7/build.....

/bug

kinbug lifecyclstale

Source

epcim

All 5 comments

saidly, seems like not just an k8s issue as container.image.repository is not populated either :(

epcim on 1 Oct 2020

+1 to this issue. Running on GKE most of the alerts are missing all of the relevant pod meta-data.

Running on GKE 1.18 with COS_CONTAINERD. I checked this issue which was the first I came across. The fixes that are in the merge are all in my deployment, so I'm guessing this is new behaviour