Falco: Fail to build falco module in air-gapped environment

[EppO]

When deploying falco in an air-gapped environment, it can't compile the falco probe by itself even though kernel headers/devel packages are installed on the host.

How to reproduce it

Deploy falco 0.21.0 deployment manifests in a kubernetes cluster with strict egress restrictions (no Internet connectivity).

Expected behaviour

falco-probe.ko can be compiled at container startup using host kernel headers.

Screenshots

* Setting up /usr/src links from host
* Unloading falco-probe, if present
* Running dkms install for falco
Error! Could not find module source directory.
Directory: /usr/src/falco-latest does not exist.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/latest/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 3.10.0-1062.12.1.el7.x86_64
Found kernel config at /host/boot/config-3.10.0-1062.12.1.el7.x86_64
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-latest-x86_64-3.10.0-1062.12.1.el7.x86_64-eab5dab83905e02403bc54f09c828b8a.ko
curl: (7) Failed to connect to s3.amazonaws.com port 443: Connection timed out
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community
Fri Apr  3 16:28:25 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Fri Apr  3 16:28:25 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Fri Apr  3 16:28:26 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:

Environment

• Falco version: 0.21.0

• System info:

{
  "machine": "x86_64",
  "nodename": "infra-stack-falco-dbsg7",
  "release": "3.10.0-1062.12.1.el7.x86_64",
  "sysname": "Linux",
  "version": "#1 SMP Tue Feb 4 23:02:59 UTC 2020"
}

• Cloud provider or hardware configuration:
• OS:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel: 3.10.0-1062.12.1.el7.x86_64

• Installation method: Kubernetes

Additional context

Kernel related RPMs installed on all Kubernetes nodes:

• kernel-3.10.0-1062.12.1.el7.x86_64
• kernel-headers-3.10.0-1062.12.1.el7.x86_64
• kernel-devel-3.10.0-1062.12.1.el7.x86_64

When I try to compile manually the probe, it succeeds in the pod, but somehow the dmks build is failing (no logs found)

# cd /usr/src
# ls
debug  falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94  kernels
# cd falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94
# ls
Makefile  dkms.conf    dynamic_params_table.c  fillers_table.c  main.c  ppm_compat_unistd_32.h  ppm_events.c  ppm_events_public.h  ppm_fillers.h       ppm_ringbuffer.h  ppm_version.h
bpf   driver_config.h  event_table.c       flags_table.c    ppm.h   ppm_cputime.c       ppm_events.h  ppm_fillers.c        ppm_flag_helpers.h  ppm_syscall.h     syscall_table.c
# make
make -C /lib/modules/3.10.0-1062.12.1.el7.x86_64/build M=/usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94 modules
make[1]: Entering directory '/host/usr/src/kernels/3.10.0-1062.12.1.el7.x86_64'
  CC [M]  /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/main.o
  CC [M]  /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/dynamic_params_table.o
  CC [M]  /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/fillers_table.o
  CC [M]  /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/flags_table.o
  CC [M]  /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/ppm_events.o
/usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/ppm_events.o: warning: objtool: .altinstr_replacement+0x0: call without frame pointer save/setup
  CC [M]  /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/ppm_fillers.o
  CC [M]  /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/event_table.o
  CC [M]  /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/syscall_table.o
  CC [M]  /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/ppm_cputime.o
  LD [M]  /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/falco-probe.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/falco-probe.mod.o
  LD [M]  /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/falco-probe.ko
make[1]: Leaving directory '/host/usr/src/kernels/3.10.0-1062.12.1.el7.x86_64'
# ls
Makefile    dkms.conf       dynamic_params_table.o  falco-probe.ko     falco-probe.o    flags_table.c  main.o     ppm_compat_unistd_32.h  ppm_events.c  ppm_events_public.h  ppm_fillers.o   ppm_syscall.h    syscall_table.o
Module.symvers  driver_config.h     event_table.c       falco-probe.mod.c  fillers_table.c  flags_table.o  modules.order  ppm_cputime.c       ppm_events.h  ppm_fillers.c        ppm_flag_helpers.h  ppm_version.h
bpf     dynamic_params_table.c  event_table.o       falco-probe.mod.o  fillers_table.o  main.c     ppm.h      ppm_cputime.o       ppm_events.o  ppm_fillers.h        ppm_ringbuffer.h    syscall_table.c
# cat /var/lib/dkms/falco/latest/build/make.log
cat: /var/lib/dkms/falco/latest/build/make.log: No such file or directory
# cd /var/lib/dkms
# ls
dkms_dbversion  falco
# cd falco
# ls
be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94
# cd be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94
# ls
build  source

looks like /usr/src/falco/latest and /var/lib/dkms/falco/latest are both not created

[leodido]

I confirm this bug exists.

It’s related to the fact that the driver version now is decoupled from the
Falco version.

Fix should be straightforward and it will be provided as soon as possible.
Thanks for reporting the issue!

/assign @leodido
/assign @fntlnz

On Fri, 3 Apr 2020 at 18:38, Florent Monbillard [e-mail]
wrote:

When deploying falco in an air-gapped environment, it can't compile the
falco probe by itself even though kernel headers/devel packages are
installed on the host.

How to reproduce it

Deploy falco 0.21.0 deployment manifests in a kubernetes cluster with
strict egress restrictions (no Internet connectivity).

Expected behaviour

falco-probe.ko can be compiled at container startup using host kernel
headers.

Screenshots

• Setting up /usr/src links from host
• Unloading falco-probe, if present
• Running dkms install for falco
  Error! Could not find module source directory.
  Directory: /usr/src/falco-latest does not exist.
• Running dkms build failed, couldn't find /var/lib/dkms/falco/latest/build/make.log
• Trying to load a system falco-probe, if present
• Trying to find precompiled falco-probe for 3.10.0-1062.12.1.el7.x86_64
  Found kernel config at /host/boot/config-3.10.0-1062.12.1.el7.x86_64
• Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-latest-x86_64-3.10.0-1062.12.1.el7.x86_64-eab5dab83905e02403bc54f09c828b8a.ko
  curl: (7) Failed to connect to s3.amazonaws.com port 443: Connection timed out
  Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community
  Fri Apr 3 16:28:25 2020: Falco initialized with configuration file /etc/falco/falco.yaml
  Fri Apr 3 16:28:25 2020: Loading rules from file /etc/falco/falco_rules.yaml:
  Fri Apr 3 16:28:26 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:

Environment

-

Falco version: 0.21.0
-

System info:

{
"machine": "x86_64",
"nodename": "infra-stack-falco-dbsg7",
"release": "3.10.0-1062.12.1.el7.x86_64",
"sysname": "Linux",
"version": "#1 SMP Tue Feb 4 23:02:59 UTC 2020"
}

• Cloud provider or hardware configuration:
• OS:

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

-

Kernel: 3.10.0-1062.12.1.el7.x86_64
-

Installation method: Kubernetes

Additional context

Kernel related RPMs installed on all Kubernetes nodes:

• kernel-3.10.0-1062.12.1.el7.x86_64
• kernel-headers-3.10.0-1062.12.1.el7.x86_64
• kernel-devel-3.10.0-1062.12.1.el7.x86_64

When I try to compile manually the probe, it succeeds in the pod, but
somehow the dmks build is failing (no logs found)

cd /usr/src# ls

debug falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94 kernels# cd falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94# ls
Makefile dkms.conf dynamic_params_table.c fillers_table.c main.c ppm_compat_unistd_32.h ppm_events.c ppm_events_public.h ppm_fillers.h ppm_ringbuffer.h ppm_version.h
bpf driver_config.h event_table.c flags_table.c ppm.h ppm_cputime.c ppm_events.h ppm_fillers.c ppm_flag_helpers.h ppm_syscall.h syscall_table.c# make
make -C /lib/modules/3.10.0-1062.12.1.el7.x86_64/build M=/usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94 modules
make[1]: Entering directory '/host/usr/src/kernels/3.10.0-1062.12.1.el7.x86_64'
CC [M] /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/main.o
CC [M] /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/dynamic_params_table.o
CC [M] /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/fillers_table.o
CC [M] /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/flags_table.o
CC [M] /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/ppm_events.o
/usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/ppm_events.o: warning: objtool: .altinstr_replacement+0x0: call without frame pointer save/setup
CC [M] /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/ppm_fillers.o
CC [M] /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/event_table.o
CC [M] /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/syscall_table.o
CC [M] /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/ppm_cputime.o
LD [M] /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/falco-probe.o
Building modules, stage 2.
MODPOST 1 modules
CC /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/falco-probe.mod.o
LD [M] /usr/src/falco-be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94/falco-probe.ko
make[1]: Leaving directory '/host/usr/src/kernels/3.10.0-1062.12.1.el7.x86_64'# ls
Makefile dkms.conf dynamic_params_table.o falco-probe.ko falco-probe.o flags_table.c main.o ppm_compat_unistd_32.h ppm_events.c ppm_events_public.h ppm_fillers.o ppm_syscall.h syscall_table.o
Module.symvers driver_config.h event_table.c falco-probe.mod.c fillers_table.c flags_table.o modules.order ppm_cputime.c ppm_events.h ppm_fillers.c ppm_flag_helpers.h ppm_version.h
bpf dynamic_params_table.c event_table.o falco-probe.mod.o fillers_table.o main.c ppm.h ppm_cputime.o ppm_events.o ppm_fillers.h ppm_ringbuffer.h syscall_table.c# cat /var/lib/dkms/falco/latest/build/make.log
cat: /var/lib/dkms/falco/latest/build/make.log: No such file or directory# cd /var/lib/dkms# ls
dkms_dbversion falco# cd falco# ls
be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94# cd be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94# ls
build source

looks like /usr/src/falco/latest and /var/lib/dkms/falco/latest are both
not created

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/falcosecurity/falco/issues/1127, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAA5J452KMPTS56SHPN2GUTRKYGG5ANCNFSM4L4H6PIA
.

>

L.

[leodido]

Hello @EppO thanks for reporting this.

Anyway, this issue should have been fixed by https://github.com/falcosecurity/falco/pull/1111

Would you please try using falcosecurity/falco:master image?

Otherwise, you can wait the end of this week for Falco 0.22.0 that will contain that fix.

[EppO]

fixed in 0.22.0!!! thanks :)