Falco: unable to download precompiled falco-probe-0.17.0-x86_64-4.14.138-114.102.amzn2.x86_64

[faarshad]

What happened:
On upgrading EKS nodes from 1.13 to 1.14(ami-08739803f18dcc019), the falco container went into CrashLoopBackOff status with the following error:

* Setting up /usr/src links from host
* Unloading falco-probe, if present
* Running dkms install for falco
Error! echo
Your kernel headers for kernel 4.14.138-114.102.amzn2.x86_64 cannot be found at
/lib/modules/4.14.138-114.102.amzn2.x86_64/build or /lib/modules/4.14.138-114.102.amzn2.x86_64/source.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/0.17.0/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 4.14.138-114.102.amzn2.x86_64
Found kernel config at /host/boot/config-4.14.138-114.102.amzn2.x86_64
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-0.17.0-x86_64-4.14.138-114.102.amzn2.x86_64-c62c9c7d342c26dbac88ad04bc036848.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the sysdig community
Tue Sep 17 16:59:41 2019: Falco initialized with configuration file /etc/falco/falco.yaml
Tue Sep 17 16:59:41 2019: Loading rules from file /etc/falco/falco_rules.yaml:
Tue Sep 17 16:59:41 2019: Loading rules from file /etc/falco/falco_rules.local.yaml:
Tue Sep 17 16:59:41 2019: Loading rules from file /etc/falco/rules.d/rules-php-fpm.yaml:
Tue Sep 17 16:59:42 2019: Unable to load the driver. Exiting.
Tue Sep 17 16:59:42 2019: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

What you expected to happen:
I was expecting the pre-compiled falco-probe to load but falco-probe-0.17.0-x86_64-4.14.138-114.102.amzn2.x86_64-c62c9c7d342c26dbac88ad04bc036848.ko is missing at https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-0.17.0-x86_64-4.14.138-114.102.amzn2.x86_64-c62c9c7d342c26dbac88ad04bc036848.ko

How to reproduce it (as minimally and precisely as possible):
Use ami-08739803f18dcc019 (us-east-1) in aws to deploy falco.

Anything else we need to know?:

Environment:

• Falco version (use falco --version):0.17.0
• System info
• Cloud provider or hardware configuration: AWS
• OS (e.g: cat /etc/os-release): Linux
• Kernel (e.g. uname -a):4.14.138-114.102.amzn2.x86_64
• Install tools (e.g. in kubernetes, rpm, deb, from source): kubernetes
• Others:

[faarshad]

/area kernel-module

[fntlnz]

Thanks for reporting @farshad-hobsons ! This is happening because there had been an issue with the build script that does the build.
In particular the sysdig dependency that we use and tagged here https://github.com/draios/sysdig/releases/tag/falco%2F0.17.0 does not contain a fix that was necessary for falco to build.

We are aware of that and are making a decision right now in the public falco slack.
We have two options:

• Wait to release Falco 0.18.0 and use the new sysdig code
• Release NOW a falco 0.17.1 and use the new sysdig code but keep the same falco code as per 0.17.0

[faarshad]

Thanks @fntlnz, appreciate it.

Is there a timeline in your mind for releasing the fix in 0.17.1.

[frankreno]

Context, I am getting this on an EKS cluster running 1.13.10.

So the feedback here at first made me believe this was only an issue with 0.17.0. So I tried older versions of the helm chart and had the same issues. I went so far as to install Helm version 0.7.6 and i get this:

Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-0.15.0-x86_64-4.14.138-114.102.amzn2.x86_64-c62c9c7d342c26dbac88ad04bc036848.ko
curl: (22) The requested URL returned error: 404 Not Found

It's referring to 0.15.0 of the binary. But its actually earlier in the path that seems to be the problem. The stable director which this refers to does not exist at all

https://s3.amazonaws.com/download.draios.com/stable

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>stable</Key>
<RequestId>2D9BC3072327864B</RequestId>
<HostId>
nD5xcdUtxzv1vxGNAYDkemVPdMSjR40r04Mzm5kmotjdXNufuu+3SYjVTkfaXbjMZaQu7c+miOs=
</HostId>
</Error>

Any other workarounds?

[fntlnz]

Thanks for the feedback @frankreno will look into that too when we release 0.17.1 to see if those are built for that kernel.

[fntlnz]

This probe is now available for 0.17.1 @farshad-hobsons @frankreno

https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-0.17.1-x86_64-4.14.138-114.102.amzn2.x86_64-c62c9c7d342c26dbac88ad04bc036848.ko

Please close this once you verify it yourself!

[frankreno]

Just tested and it works great! Thank much @fntlnz . Follow up question. ETA for a new helm chart release with a docker image bump?

[faarshad]

I have validated that this works on the 4.14.138-114.102.amzn2.x86_64 kernel.

Thanks @fntlnz for getting this fixed. This helps a lot of downstream falco fans!

[leodido]

@nestorsalceda friendly pinging you for the helm chart ^^^ :)

[bredamatt]

Hi, thank you for providing the link for the probe!

I have a Q. Lets say I'm on a VM on a private network only. The same private network hosts a registry server connected to a public interface (with internet access). This means I can run the Falco container on the registry server, but not on my private VM because the image starts downloading the precompiled module from an external URL.

How would I go about adding the pre-compiled module to the container that is meant to be run on the private VM? What I am asking more specifically, is whether I'll have to downlaod the .ko file on the registry server, and push this to my private VM and then refer to this when running the container somehow. Would be great with some docs on this!

Thanks.

Best,
M