Falco: Unable to start Falco with a custom eBPF Probe

[danmx]

Describe the bug

I’m trying to start falco using the official helm chart but with custom minimal image (which just contain an extra ebpf probe). It seems I can’t run the image.

How to reproduce it

Dockerfile which builds the image:

FROM amazonlinux:2 AS build

ARG FALCO_VERSION=0.20.0+d77080a
ARG KERNELRELEASE=4.14.154-128.181.amzn2.x86_64
ARG BUILD_TYPE=release
ARG BUILD_DRIVER=OFF
ARG BUILD_BPF=ON
ARG BUILD_WARNINGS_AS_ERRORS=ON
ARG MAKE_JOBS=4

ENV BUILD_TYPE=${BUILD_TYPE}
ENV BUILD_DRIVER=${BUILD_DRIVER}
ENV BUILD_BPF=${BUILD_BPF}
ENV BUILD_WARNINGS_AS_ERRORS=${BUILD_WARNINGS_AS_ERRORS}
ENV MAKE_JOBS=${MAKE_JOBS}
ENV FALCO_VERSION=${FALCO_VERSION}
ENV KERNELRELEASE=${KERNELRELEASE}
ENV KERNELDIR=/usr/src/kernels/${KERNELRELEASE}/

# build toolchain
RUN yum -y update && yum clean all
RUN yum install -y \
        gcc \
        gcc-c++ \
        git \
        wget \
        perl-Digest-SHA \
        make \
        autoconf \
        automake \
        pkg-config \
        patch \
        libcurl-devel \
        zlib-devel \
        libyaml-devel \
        ncurses-devel \
        libtool \
        glibc-static \
        libstdc++-static \
        elfutils-libelf-devel \
        clang \
        llvm \
        tar \
        gzip \
        kernel-devel-${KERNELRELEASE} \
    && yum clean all \
    && rm -rf /var/cache/yum \
    && mkdir -p /falco/build

ARG CMAKE_VERSION="3.5.1"
ARG CMAKE_HASH="93d651a754bcf6f0124669646391dd5774c0fc4d407c384e3ae76ef9a60477e8"
COPY ./dependencies/cmake-3.5.1.tar.gz /tmp/
WORKDIR /tmp
RUN echo "${CMAKE_HASH}  cmake-${CMAKE_VERSION}.tar.gz" | shasum -a 256 -c
RUN tar xz -f cmake-${CMAKE_VERSION}.tar.gz && \
    cd cmake-${CMAKE_VERSION} && \
    ./bootstrap --system-curl && \
    make -j${MAKE_JOBS} && \
    make install && \
    rm -rf /tmp/cmake-${CMAKE_VERSION}*

COPY vendor/falco /falco

WORKDIR /falco/build

RUN cmake \
        -DCMAKE_BUILD_TYPE="$BUILD_TYPE" \
        -DCMAKE_INSTALL_PREFIX=/usr \
        -DBUILD_DRIVER="$BUILD_DRIVER" \
        -DBUILD_BPF="$BUILD_BPF" \
        -DBUILD_WARNINGS_AS_ERRORS="$BUILD_WARNINGS_AS_ERRORS" \
        -DFALCO_VERSION="$FALCO_VERSION" \
        -DDRAIOS_DEBUG_FLAGS="$DRAIOS_DEBUG_FLAGS" \
        -DUSE_BUNDLED_DEPS=ON \
        ..

# build eBPF Probe
RUN make bpf

FROM falcosecurity/falco:0.20.0-minimal AS main

# copy eBPF probe
COPY --chown=0:0 --from=build /falco/build/driver/bpf/probe.o /root/.sysdig/falco-probe-bpf.o

CMD ["/usr/bin/falco", "-o", "time_format_iso_8601=true"]

Falco version vendored using braid:

{
  "config_version": 1,
  "mirrors": {
    "vendor/falco": {
      "url": "git://github.com/falcosecurity/falco.git",
      "tag": "0.20.0",
      "revision": "d77080a8c2db0c290b5d7552bd5b49e1b23fb5ad"
    }
  }
}

Using suggested CMake just pulled and saved it.

Using official helm chart (stable/helm) copied using braid:

{
  "config_version": 1,
  "mirrors": {
    "mirrored/falco": {
      "url": "git://github.com/helm/charts.git",
      "branch": "master",
      "path": "stable/falco",
      "revision": "28993e1abbdf1d9f17e75cbcf4447e53607a4b09"
    }
  }
}

with custom parameters:

image:
  registry: my-private-registry
  repository: falco
  tag: 0.20.0-dev15
containerd:
  enabled: false
rbac:
  create: false
serviceAccount:
  create: false
podSecurityPolicy:
  create: false
extraArgs:
  - --disable-source
  - k8s_audit
ebpf:
  enabled: true
falco:
  timeFormatISO8601: true
  jsonOutput: true
tolerations: []
daemonset:
  env:
    FALCO_BPF_PROBE: ""

Expected behaviour

Falco deploys and starts.

Screenshots

Logs from k8s:

2020-03-03T10:20:44.036855347Z 2020-03-03T10:20:44+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2020-03-03T10:20:44.0368956Z 2020-03-03T10:20:44+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2020-03-03T10:20:44.641893214Z 2020-03-03T10:20:44+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2020-03-03T10:20:46.935844926Z 2020-03-03T10:20:46+0000: Unable to load the driver. Exiting.
2020-03-03T10:20:48.744051928Z 2020-03-03T10:20:48+0000: Runtime error: failed to open event raw_syscalls/sys_enter. Exiting.  

Environment

• Falco version: 0.20.0+d77080a
  
• System info:

{"machine":"x86_64","nodename":"foobar.eu-west-1.compute.internal","release":"4.14.154-128.181.amzn2.x86_64","sysname":"Linux","version":"#1 SMP Sat Nov 16 21:49:00 UTC 2019"}

• Cloud provider or hardware configuration: AWS
• OS: Amazon Linux 2
  
• Kernel: 4.14.154-128.181.amzn2.x86_64
  
• Installation method:
  
  Falco from official minimal container image.
  eBPF probe compiled from source code.
  Deployed using the helm chart.

Additional context

[JoupainMD]

I have been going through the same issue (same ecosystem, Amazon Linux kube etc ..) and have been able to fix it doing this in the container:

mount -t debugfs nodev /sys/kernel/debug

as mentioned here in the falco-probe-loader.

https://github.com/falcosecurity/falco/blob/2126616529e7015ff88653b7491dc1937d7e54e5/scripts/falco-probe-loader#L206-L208

You can also mount the host volume in kube like this:

        volumeMounts:
        - name: docker-socket
          mountPath: "/host/var/run/docker.sock"
        - name: procdir
          mountPath: "/host/proc"
          readOnly: true
        - name: dev
          mountPath: "/host/dev"
          readOnly: true
        - name: rules
          mountPath: "/etc/falco/falco_rules.local.yaml"
          subPath: falco_rules.local.yaml
        - name: debugfs
          mountPath: "/sys/kernel/debug"
        - name: probedir
          mountPath: "/falco-bpf-probe"
      volumes:
      - hostPath:
          path: "/var/run/docker.sock"
        name: docker-socket
      - hostPath:
          path: "/proc"
        name: procdir
      - hostPath:
          path: "/dev"
        name: dev
      - hostPath:
          path: "/sys/kernel/debug"
        name: debugfs**
      - name: rules
        configMap:
          name: falco-rules
      - name: probedir
        emptyDir: {}

[danmx]

This solved the issue