Falco: Helm install for falco not working on EKS - error opening device /dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded

[ahilmathew]

Describe the bug

• Setting up /usr/src links from host
  ln: failed to create symbolic link '/usr/src//host/usr/src/debug': No such file or directory
  ln: failed to create symbolic link '/usr/src//host/usr/src/kernels': No such file or directory
• Unloading falco-probe, if present
• Running dkms install for falco
  Error! echo
  Your kernel headers for kernel 4.14.154-128.181.amzn2.x86_64 cannot be found at
  /lib/modules/4.14.154-128.181.amzn2.x86_64/build or /lib/modules/4.14.154-128.181.amzn2.x86_64/source.
• Running dkms build failed, couldn't find /var/lib/dkms/falco/0.20.0+d77080a/build/make.log
• Trying to load a system falco-probe, if present
• Trying to find precompiled falco-probe for 4.14.154-128.181.amzn2.x86_64
  Found kernel config at /host/boot/config-4.14.154-128.181.amzn2.x86_64
• Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-0.20.0%2Bd77080a-x86_64-4.14.154-128.181.amzn2.x86_64-2a1b4e38248e08669f1ed37604b3ad74.ko
  curl: (22) The requested URL returned error: 404 Not Found
  Sun Mar 8 22:49:33 2020: Falco initialized with configuration file /etc/falco/falco.yaml
  Sun Mar 8 22:49:33 2020: Loading rules from file /etc/falco/falco_rules.yaml:
  Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community
  Sun Mar 8 22:49:34 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
  Sun Mar 8 22:49:35 2020: Unable to load the driver. Exiting.
  Sun Mar 8 22:49:35 2020: Runtime error: can't open BPF probe '/root/.sysdig/falco-probe-bpf.o': Errno 2. Exiting.

How to reproduce it

helm install falcohelm stable/falco --set ebpf.enabled=true

Expected behaviour

Falco pods should have been running without errors.

Environment
EKS cluster

• OS:
  Amazon Linux 2

• Installation method:
  Helm Install

[nibalizer]

I'm on IKS (IBM Kubernetes service) not EKS but I was able to change version to 0.19.0 in the values.yaml of the helm chart and successfully bring up falco.

[fntlnz]

@ahilmathew we have multiple fixes for the problems described here:

• https://github.com/falcosecurity/falco/pull/981
• https://github.com/falcosecurity/falco/pull/1050

In the meanwhile can you please try to use the falcosecurity/falco:master image? It contains those fixes until we are not ready to release 0.21.0.

[rabidsloth]

I'm also on EKS and getting this same error. I've tried to use master and 0.19.0 as the tag and neither of these work.

log using the master image tag:

Loading...* Setting up /usr/src links from host
* Unloading falco-probe, if present
* Running dkms install for falco
Error! echo
Your kernel headers for kernel 4.14.165-133.209.amzn2.x86_64 cannot be found at
/lib/modules/4.14.165-133.209.amzn2.x86_64/build or /lib/modules/4.14.165-133.209.amzn2.x86_64/source.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/0.20.0+e637b1e/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 4.14.165-133.209.amzn2.x86_64
Found kernel config at /host/boot/config-4.14.165-133.209.amzn2.x86_64
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-0.20.0%2Be637b1e-x86_64-4.14.165-133.209.amzn2.x86_64-1ed15745c0421d7dcbcff00ebd616d81.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community
Tue Mar 10 00:17:40 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Tue Mar 10 00:17:40 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Tue Mar 10 00:17:41 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Tue Mar 10 00:17:42 2020: Unable to load the driver. Exiting.
Tue Mar 10 00:17:42 2020: Runtime error: error opening device /dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

[nikhilgorantla]

Any update on this running into the same issue

[ahilmathew]

I ended up using helm install stable/falco --version 1.1.0 which has Falco version 0.18.0 which doesn't have this issue.

[ahilmathew]

@ahilmathew we have multiple fixes for the problems described here:

* #981

* #1050

In the meanwhile can you please try to use the falcosecurity/falco:master image? It contains those fixes until we are not ready to release 0.21.0.

thank you. Will try using the image as well.

[leodido]

Any update from people using the latest falcosecurity/falco:master ?

[nikhilgorantla]

falcosecurity/falco:master
falcosecurity/falco:latest
falcosecurity/falco:0.21.0
falcosecurity/falco:0.18.0

none of them work on eks

[djsly]

same for AKS, we were running 0.19.0 and just bumped to 0.21.0, pods are not starting

* Setting up /usr/src links from host
* Unloading falco-probe, if present
* Running dkms install for falco
Error! Could not find module source directory.
Directory: /usr/src/falco-latest does not exist.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/latest/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 4.15.0-1071-azure
Found kernel config at /host/boot/config-4.15.0-1071-azure
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-latest-x86_64-4.15.0-1071-azure-ed0721c105202e27d58e45a336be8299.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community
Fri Mar 27 16:06:20 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Fri Mar 27 16:06:20 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Fri Mar 27 16:06:21 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Fri Mar 27 16:06:21 2020: Unable to load the driver. Exiting.
Fri Mar 27 16:06:21 2020: Runtime error: error opening device /dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

[jjo]

Key in above error log is:

Directory: /usr/src/falco-latest does not exist.

as I just reported at #1123.

[blhagadorn]

I can also confirm, on IBM Cloud Kubernetes Service (IKS), I also had the same original reported issue. helm install stable/falco --version 1.1.0. Seemed to work for me.

[cw-sakamoto]

0.18.0(with stable/falco version 1.1.0) work on eks 1.14 but not work on 1.15.
0.21.0 doesn't work either.

[rodricarvalho]

0.18.0 works for me on AWS EC2 without any issue. I hope to be able to use the 0.21.0 version soon.
Just change the value of the tag to 0.18.0 in the values.yaml file to make works.

[cw-sakamoto]

0.22.1(helm chart 1.1.1) doesn't work on eks 1.15

Your kernel headers for kernel 4.14.173-137.229.amzn2.x86_64 cannot be found at
/lib/modules/4.14.173-137.229.amzn2.x86_64/build or /lib/modules/4.14.173-137.229.amzn2.x86_64/source.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 4.14.173-137.229.amzn2.x86_64
Found kernel config at /host/boot/config-4.14.173-137.229.amzn2.x86_64
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-a259b4bf49c3330d9ad6c3eed9eb1a31954259a6-x86_64-4.14.173-137.229.amzn2.x86_64-f0c8ced41ae4d0e71aa715068964ce9f.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community
Tue Apr 21 05:07:17 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Tue Apr 21 05:07:17 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Tue Apr 21 05:07:17 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Tue Apr 21 05:07:17 2020: Unable to load the driver. Exiting.
Tue Apr 21 05:07:17 2020: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

[loetn]

Getting also 404 not found when deploying as a normal daemonset, or installing via the helm chart on a kubeadm cluster 1.18.2. Tried 0.22.0, 0.22.1, latest, master. Using 0.18.0 still works.

[cw-sakamoto]

0.18.0 dose not work with eks 1.15 (eksctl 0.17.0).

$ kubectl logs falco-cdkjx
* Setting up /usr/src links from host
* Unloading falco-probe, if present
* Running dkms install for falco
Error! echo
Your kernel headers for kernel 4.14.173-137.229.amzn2.x86_64 cannot be found at
/lib/modules/4.14.173-137.229.amzn2.x86_64/build or /lib/modules/4.14.173-137.229.amzn2.x86_64/source.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/0.18.0/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 4.14.173-137.229.amzn2.x86_64
Found kernel config at /host/boot/config-4.14.173-137.229.amzn2.x86_64
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-0.18.0-x86_64-4.14.173-137.229.amzn2.x86_64-f0c8ced41ae4d0e71aa715068964ce9f.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the sysdig community
Wed Apr 22 00:43:58 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Wed Apr 22 00:43:58 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Wed Apr 22 00:43:58 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Wed Apr 22 00:43:58 2020: Unable to load the driver. Exiting.
Wed Apr 22 00:43:58 2020: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

[leodido]

@cw-sakamoto we now have a new set of prebuilt modules at https://dl.bintray.com/falcosecurity/driver/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/

For example, the module for amzn2 4.14.173 you are looking for is here.

[leodido]

@djsly we do not ship (neither support) drivers for AKS at the moment.

I opened an issue as reference/request here. It sounds like a good first contribution to do!

[leodido]

@rabidsloth the Falco driver your environment is looking for is now at https://dl.bintray.com/falcosecurity/driver/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/:falco_amazonlinux2_4.14.165-133.209.amzn2.x86_64_1.ko

[leodido]

Using the latest falcosecurity/falco:master image, which contains the new falco-driver-loader script (#1160), should solve the majority of the 404 here reported.

For this reason, I'm closing this issue. But feel free to open new ones and continue the discussion in case new errors of this kind happen.

/close

[poiana]

@leodido: Closing this issue.

In response to this:

Using the latest falcosecurity/falco:master image, which contains the new falco-driver-loader script (#1160), should solve the majority of the 404 here reported.

For this reason, I'm closing this issue. But feel free to open new ones and continue the discussion in case new errors of this kind happen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[djsly]

@djsly we do not ship (neither support) drivers for AKS at the moment.

I opened an issue as reference/request here. It sounds like a good first contribution to do!

@leodido thanks, while there might not be any official support for AKS yet, Falco was usable and runnable. Are we saying that the support for AKS is no longer possible until a new driver kit gets created ?

[leodido]

No @djsly, I was saying that we do not support prebuilt drivers for AKS.

Clearly the postinst scripts of the packages (same does the falco-driver-loader script) will always try to build the driver after the Falco installation. And that can succeed :)

[jessebye]

@leodido I'm running falcosecurity/falco:master but still receiving an error about unsupported driver. This is on EKS 1.15.

* Setting up /usr/src links from host
* Unloading falco module, if present
* Running dkms build failed, couldn't find /var/lib/dkms/falco/47374b2b73734d509f3c99890c80be5242021c3d/build/make.log
* Trying to load a system falco driver, if present
* Trying to find a prebuilt falco module for kernel 4.14.165-133.209.amzn2.x86_64
Detected an unsupported target system, please get in touch with the Falco community

Is there a way to point the falco-driver-loader script at the right URL? How can I tell what URL it is trying to use? It doesn't echo it in the logs...

[fntlnz]

@jessebye I know what's happening here. The master image is being used but we (on Thursday) changed the driver version in master without instructing test-infra to build for that version.
Infact we do have the module version for dev for your kernel here https://dl.bintray.com/falcosecurity/driver/dev/:falco_amazonlinux2_4.14.165-133.209.amzn2.x86_64_1.ko

As a workardound, you can open the falco-driver-loader script and change the DRIVER_VERSION variable value to dev.

I'm opening an issue to fix this for when people use the master image.

[leodido]

Imho the error @jessebye is encountering does not depend on the driver version.

I mean, the root cause is not the driver version. The root cause is the fact that falco-driver-loader is not able to build the driver locally on the EKS 1.15 of @jessebye.

The lookup on prebuilt drivers index is just a fallback.

Anyway, the workaround @fntlnz proposed should work without troubles.

[fntlnz]

I think that it might be due to errors while doing the symlinks. @jessebye are you running the latest Falco 0.22.0 ? And if so, is this an upgrade from an older version or a fresh new installation?

We still need to fix the pre-built stuff.

[jessebye]

We are running Falco 0.22.0. Fresh new installation.
On May 2, 2020, 7:17 AM -0700, Lorenzo Fontana [e-mail], wrote:

I think that it might be due to errors while doing the symlinks. @jessebye are you running the latest Falco 0.22.0 ? And if so, is this an upgrade from an older version or a fresh new installation?
We still need to fix the pre-built stuff.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[jessebye]

@leodido from what I can see, there _is_ a prebuilt driver for my Amazon Linux kernel, but it still fails to use it. So not only does it fail to build the driver (root cause) but then it fails to find the prebuilt image that it should be able to find. Does that make sense?

Also, not sure how to use that workaround... I guess we'd have to fork the repo and build our own custom docker image? I've been really trying to avoid that.

[ianhundere]

same issue here, but this was after updating to EKS 1.16 from EKS 1.15.

we were using 0.19.0 before this without issue.

using the latest falcosecurity/falco:master image did not resolve the issue.

Your kernel headers for kernel 4.14.173-137.229.amzn2.x86_64 cannot be found at
/lib/modules/4.14.173-137.229.amzn2.x86_64/build or /lib/modules/4.14.173-137.229.amzn2.x86_64/source.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 4.14.173-137.229.amzn2.x86_64
Found kernel config at /host/boot/config-4.14.173-137.229.amzn2.x86_64
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-a259b4bf49c3330d9ad6c3eed9eb1a31954259a6-x86_64-4.14.173-137.229.amzn2.x86_64-f0c8ced41ae4d0e71aa715068964ce9f.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community
Tue May  5 14:08:36 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Tue May  5 14:08:36 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Tue May  5 14:08:37 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Tue May  5 14:08:38 2020: Unable to load the driver. Exiting.
Tue May  5 14:08:38 2020: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

[victorrodriguez1984]

Hello Team, same erro on version 0.21.0 at IKS

image:
registry: docker.io
repository: falcosecurity/falco
tag: 0.21.0
pullPolicy: IfNotPresent

Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community
Sun Jun  7 17:28:33 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Sun Jun  7 17:28:33 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Sun Jun  7 17:28:33 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Sun Jun  7 17:28:34 2020: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Sun Jun  7 17:28:35 2020: Unable to load the driver. Exiting.
Sun Jun  7 17:28:35 2020: Runtime error: error opening device /dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

[fbongiovanni29]

Same error on GKE (ebpf enabled) version 0.23.0.

Note: It worked for me once, but I uninstalled/re-installed it to debug some issues and now I'm getting this:

* Setting up /usr/src links from host
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to dkms install falco module
* Running dkms build failed, couldn't find /var/lib/dkms/falco/96bd9bc560f67742738eb7255aeb4d03046b8045/build/make.log
* Trying to load a system falco driver, if present
* Trying to find locally a prebuilt falco module for kernel 4.14.138+, if present
* Trying to download prebuilt module from https://dl.bintray.com/falcosecurity/driver/96bd9bc560f67742738eb7255aeb4d03046b8045/falco_cos_4.14.138%2B_1.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco module and loading it or getting in touch with the Falco community
Mon Jun 22 19:25:50 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Mon Jun 22 19:25:50 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Mon Jun 22 19:25:50 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Mon Jun 22 19:25:51 2020: Unable to load the driver. Exiting.
Mon Jun 22 19:25:51 2020: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.

[rubensayshi]

we were running 0.23.0 on GKE without issues as well, but I just did a reinstall (and we bumped GKE to latest; 1.16.10-gke.8) and now we get more or less the same;

* Setting up /usr/src links from host
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to dkms install falco module
* Running dkms build failed, couldn't find /var/lib/dkms/falco/96bd9bc560f67742738eb7255aeb4d03046b8045/build/make.log
* Trying to load a system falco driver, if present
* Trying to find locally a prebuilt falco module for kernel 4.19.112+, if present
* Trying to download prebuilt module from https://dl.bintray.com/falcosecurity/driver/96bd9bc560f67742738eb7255aeb4d03046b8045/falco_cos_4.19.112%2B_1.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco module and loading it or getting in touch with the Falco community
Thu Jul  2 13:32:23 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Thu Jul  2 13:32:23 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Thu Jul  2 13:32:24 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Thu Jul  2 13:32:24 2020: Unable to load the driver. Exiting.
Thu Jul  2 13:32:24 2020: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.

[dinvlad]

We hit the same issue on COS-81 (unable to download falco_cos_4.19.112%2B_1.ko). It only works when run in privileged mode with ePBF enabled (https://github.com/falcosecurity/falco/issues/1299)