falco failed to run in kind

[oke-py]

What happened:
falco failed to run in kind.

$ kubectl logs falco-1575678340-6fxr4
* Setting up /usr/src links from host
* Unloading falco-probe, if present
* Running dkms install for falco
Error! echo
Your kernel headers for kernel 4.9.184-linuxkit cannot be found at
/lib/modules/4.9.184-linuxkit/build or /lib/modules/4.9.184-linuxkit/source.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/0.17.1/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 4.9.184-linuxkit
Found kernel config at /proc/config.gz
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-0.17.1-x86_64-4.9.184-linuxkit-aa7fd392e72f1a4dcbe68e1d89467b54.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the sysdig community
Sat Dec  7 00:26:13 2019: Falco initialized with configuration file /etc/falco/falco.yaml
Sat Dec  7 00:26:13 2019: Loading rules from file /etc/falco/falco_rules.yaml:
Sat Dec  7 00:26:14 2019: Loading rules from file /etc/falco/falco_rules.local.yaml:
Sat Dec  7 00:26:14 2019: Unable to load the driver. Exiting.
Sat Dec  7 00:26:14 2019: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

What you expected to happen:
falco DaemonSet is successfully running.

How to reproduce it (as minimally and precisely as possible):

$ helm install stable/falco --generate-name
# or
$ kubectl create deploy falco --image=falcosecurity/falco:0.17.1

Anything else we need to know?:

Environment:

• Falco version (use falco --version): 0.17.1
• System info
• Cloud provider or hardware configuration: kind v0.6.1
• OS (e.g: cat /etc/os-release): macOS Catalina 10.15.1
• Kernel (e.g. uname -a):
• Install tools (e.g. in kubernetes, rpm, deb, from source): kubernetes
• Others:

[markyjackson-taulia]

/assign @markyjackson-taulia

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[leogr]

Same issue.

$ kubectl logs -f falco-2t8dh                                                                                                                               * Setting up /usr/src links from host
ln: failed to create symbolic link '/usr/src//host/usr/src/*': No such file or directory* Unloading falco-probe, if present
* Running dkms install for falco
Kernel preparation unnecessary for this kernel.  Skipping...
Building module:
cleaning build area.....make -j8 KERNELRELEASE=5.5.7-arch1-1 -C /lib/modules/5.5.7-arch1-1/build M=/var/lib/dkms/falco/0.20.0+d77080a/build......(bad exit status: 2)
Error! Bad return status for module build on kernel: 5.5.7-arch1-1 (x86_64)Consult /var/lib/dkms/falco/0.20.0+d77080a/build/make.log for more information.
* Running dkms build failed, dumping /var/lib/dkms/falco/0.20.0+d77080a/build/make.log
DKMS make.log for falco-0.20.0+d77080a for kernel 5.5.7-arch1-1 (x86_64)
Thu Mar  5 14:06:25 UTC 2020
make: Entering directory '/host/lib/modules/5.5.7-arch1-1/build'  AR      /var/lib/dkms/falco/0.20.0+d77080a/build/built-in.a
  CC [M]  /var/lib/dkms/falco/0.20.0+d77080a/build/main.o  CC [M]  /var/lib/dkms/falco/0.20.0+d77080a/build/dynamic_params_table.o
  CC [M]  /var/lib/dkms/falco/0.20.0+d77080a/build/fillers_table.o
cc1: error: cannot load plugin ./scripts/gcc-plugins/structleak_plugin.so
./scripts/gcc-plugins/structleak_plugin.so: undefined symbol: _ZN8opt_pass14set_pass_paramEjb  CC [M]  /var/lib/dkms/falco/0.20.0+d77080a/build/flags_table.o
make[1]: *** [scripts/Makefile.build:265: /var/lib/dkms/falco/0.20.0+d77080a/build/main.o] Error 1make[1]: *** Waiting for unfinished jobs....
cc1: error: cannot load plugin ./scripts/gcc-plugins/structleak_plugin.so
./scripts/gcc-plugins/structleak_plugin.so: undefined symbol: _ZN8opt_pass14set_pass_paramEjb
cc1: error: cannot load plugin ./scripts/gcc-plugins/structleak_plugin.so
./scripts/gcc-plugins/structleak_plugin.so: undefined symbol: _ZN8opt_pass14set_pass_paramEjb
make[1]: *** [scripts/Makefile.build:266: /var/lib/dkms/falco/0.20.0+d77080a/build/fillers_table.o] Error 1
make[1]: *** [scripts/Makefile.build:266: /var/lib/dkms/falco/0.20.0+d77080a/build/dynamic_params_table.o] Error 1
  CC [M]  /var/lib/dkms/falco/0.20.0+d77080a/build/ppm_events.o
cc1: error: cannot load plugin ./scripts/gcc-plugins/structleak_plugin.so
./scripts/gcc-plugins/structleak_plugin.so: undefined symbol: _ZN8opt_pass14set_pass_paramEjb
make[1]: *** [scripts/Makefile.build:266: /var/lib/dkms/falco/0.20.0+d77080a/build/ppm_events.o] Error 1
cc1: error: cannot load plugin ./scripts/gcc-plugins/structleak_plugin.so
./scripts/gcc-plugins/structleak_plugin.so: undefined symbol: _ZN8opt_pass14set_pass_paramEjb
make[1]: *** [scripts/Makefile.build:266: /var/lib/dkms/falco/0.20.0+d77080a/build/flags_table.o] Error 1
make: *** [Makefile:1693: /var/lib/dkms/falco/0.20.0+d77080a/build] Error 2
make: Leaving directory '/host/lib/modules/5.5.7-arch1-1/build'
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 5.5.7-arch1-1
Found kernel config at /proc/config.gz
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-0.20.0%2Bd77080a-x86_64-5.5.7-arch1-1-ac921da40a489206e6993186c8c954c5.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community
Thu Mar  5 14:06:35 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Thu Mar  5 14:06:35 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Thu Mar  5 14:06:38 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Thu Mar  5 14:06:40 2020: Unable to load the driver. Exiting.
Thu Mar  5 14:06:40 2020: Runtime error: error opening device /dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

[leogr]

Quick update.

I tried to install the driver directly in the host machine, Falco found it but still does not work:

* Trying to load a system falco-probe, if present
falco-probe found and loaded with modprobe
Fri Mar  6 10:52:29 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Fri Mar  6 10:52:29 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Fri Mar  6 10:52:31 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Fri Mar  6 10:52:33 2020: Runtime error: error opening device /dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

Then I tried a prebuilt eBPF probe. I had to use SYSDIG_BPF_PROBE: /lib/modules/falco-probe.o (/lib/modules is already shared with kind, but I know it's a bit hacky).
It worked, but I don't believe this is a good solution.

[leogr]

I have finally found the problem, below the steps to make it work.

1. Install the driver on the host machine

Clone the Falco project and checkout the tag corresponding to the same Falco version used within the helm chart (0.20.0 in my case), then:

mkdir build
cd build
cmake ..
sudo make install_driver

2. Mount /dev on kind-control-plane
To make the driver work, the host's /dev must be accessible from within the container. To do that, kind requires the following:

kind-config.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /dev
    containerPath: /dev

Then create the cluster so:

kind create cluster --config=./kind-config.yaml

3. DONE!
You can now deploy Falco to kind's cluster as usual. You should notice that the driver will automatically be loaded with modprobe:

* Trying to load a system falco-probe, if present
falco-probe found and loaded with modprobe
Fri Mar  6 14:54:49 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Fri Mar  6 14:54:49 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Fri Mar  6 14:54:51 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Fri Mar  6 14:54:54 2020: Starting internal webserver, listening on port 8765

@oke-py @markyjackson-taulia can you please take a look and let me know if that works for you too?

[oke-py]

@leogr Great news! However, I'm not familiar with cmake. I can't find Makefile and the error below occurs.

sudo make install_driver
Password:
make: *** No rule to make target `install_driver'.  Stop.

[leogr]

@leogr Great news! However, I'm not familiar with cmake. I can't find Makefile and the error below occurs.

Once cmake finishes, the makefile should be inside the build dir. You can find more details in the documentation.

PS
Which Falco version are you using?

[oke-py]

This documentaion is helpful.
I haven't installed the required dependencies, and so cmake failed (I didn't notice that).

Now I'm running sudo make install_driver. It will probably take time.
I'm using falco 0.20.0.

[oke-py]

Oh, I made a mistake. I was running sudo make install.
sudo make install_driver failed as below.

$ sudo make install_driver
Built target driver
At main.c:160:
- SSL error:02001002:system library:fopen:No such file or directory: ../crypto/bio/bss_file.c:72
- SSL error:2006D080:BIO routines:BIO_new_file:no such file: ../crypto/bio/bss_file.c:79
sign-file: certs/signing_key.pem: No such file or directory
Warning: modules_install: missing 'System.map' file. Skipping depmod.
Built target install_driver

I'm using Ubuntu 18.04 on GCE.

$ uname -a
Linux falco 5.0.0-1031-gcp #32-Ubuntu SMP Tue Feb 11 03:55:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

$ git rev-parse HEAD
d77080a8c2db0c290b5d7552bd5b49e1b23fb5ad

$ git rev-parse tags/0.20.0
d77080a8c2db0c290b5d7552bd5b49e1b23fb5ad

[fntlnz]

Related to this, I think we want to get the manifests for newer kubernetes versions in the Falco repo.

Also @oke-py thanks for getting this done on the helm charts! :point_down:
https://github.com/helm/charts/pull/17339

[leogr]

$ sudo make install_driver
Built target driver
At main.c:160:
- SSL error:02001002:system library:fopen:No such file or directory: ../crypto/bio/bss_file.c:72
- SSL error:2006D080:BIO routines:BIO_new_file:no such file: ../crypto/bio/bss_file.c:79
sign-file: certs/signing_key.pem: No such file or directory
Warning: modules_install: missing 'System.map' file. Skipping depmod.
Built target install_driver

@oke-py
Unfortunately, I was not able to reproduce this. But it should not related to kind.
Could you try with eBPF?
https://github.com/helm/charts/blob/master/stable/falco/values.yaml#L73

[oke-py]

@leogr falco failed to run with eBPF.

kubectl logs falco-1583916426-9d98p
ln: failed to create symbolic link '/usr/src//host/usr/src/*': No such file or directory
* Setting up /usr/src links from host
* Unloading falco-probe, if present
* Running dkms install for falco
Error! echo
Your kernel headers for kernel 5.0.0-1031-gcp cannot be found at
/lib/modules/5.0.0-1031-gcp/build or /lib/modules/5.0.0-1031-gcp/source.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/0.20.0+d77080a/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 5.0.0-1031-gcp
Cannot find kernel config
Wed Mar 11 08:47:33 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Wed Mar 11 08:47:33 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Wed Mar 11 08:47:34 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Wed Mar 11 08:47:35 2020: Unable to load the driver. Exiting.
Wed Mar 11 08:47:35 2020: Runtime error: can't open BPF probe '/root/.sysdig/falco-probe-bpf.o': Errno 2. Exiting.

I agree that the error should not related to kind.
I can't install the driver on the host machine.
I'll try it again on another OS.

[leogr]

@oke-py If you want to try, there's an alternative/new approach to build and install the kernel module: driverkit (I tried on Arch, and it worked though it is still under development).
Basically, you have to:

• build the kernel module with driverkit
• copy it to /lib/modules/`uname -r`/falco.ko
• run depmod
  And finally, deploy Falco on Kind.

[leogr]

@oke-py here you find further details on how to install the kernel module using driverkit. Let me know!

[oke-py]

At last, it worked fine! Thank you so much.
I used ubuntu-18.04 on EC2.

install go

$ sudo add-apt-repository ppa:longsleep/golang-backports
$ sudo apt update
$ sudo apt install golang-go
$ go version
go version go1.14 linux/amd64

install docker

$ sudo apt-get update
$ sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) \
  stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce docker-ce-cli containerd.io
$ sudo usermod -aG docker `whoami`
$ exit

re-login

set env-var

$ export GOPATH=~/go
$ export PATH=$PATH:$GOPATH/bin
$ export GO111MODULE="on"

install kernel module

$ docker pull falcosecurity/driverkit-builder
$ go get github.com/falcosecurity/driverkit
$ uname -v
#59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019
$ uname -r
4.15.0-1057-aws
$ driverkit docker --output-module /tmp/falco.ko --kernelversion=59 --kernelrelease=4.15.0-1057-aws --driverversion=dev --target=ubuntu-aws
$ sudo cp /tmp/falco.ko /lib/modules/`uname -r`/falco.ko
$ sudo depmod
$ sudo modprobe falco
$ lsmod | grep -i falco

install kubectl & helm

$ curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
$ chmod +x ./kubectl
$ sudo mv ./kubectl /usr/local/bin/kubectl
$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | sudo bash
$ helm repo add stable https://kubernetes-charts.storage.googleapis.com/

install kind & create cluster

$ go get sigs.k8s.io/[email protected]
$ cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /dev
    containerPath: /dev
EOF 

install & run falco

$ helm install stable/falco --generate-name
$ kubectl get po -w
NAME                     READY   STATUS    RESTARTS   AGE
falco-1584781190-f5wdw   1/1     Running             0          30s

[leodido]

Thanks @leogr for stepping up and helping with this.

I'll call it solved :)

/close

[poiana]

@leodido: Closing this issue.

In response to this:

Thanks @leogr for stepping up and helping with this.

I'll call it solved :)

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yashbhutwala]

How do I get it running for Docker Desktop for Mac? Can we please document this?

[leogr]

Hey @yashbhutwala
as far as I know, Falco on Docker for Mac has a different kind of issue that's related to linuxkit and @leodido is working on that.
Feel free to open another issue in the falco repo and add more details.