Falco: High CPU usage when grpc is enable

[scallopboat]

Describe the bug

When grpc is enabled in falco 0.21.0 there is a huge increase in cpu usage on some pods on kops cluster, k8s 1.16, debian nodes.

grpc disabled
NAME CPU(cores) MEMORY(bytes)
falco-htqvd 7m 16Mi
falco-hzxvx 7m 16Mi
falco-lw9f7 7m 16Mi
falco-nmvqq 7m 16Mi
falco-phz9d 8m 16Mi
falco-scbcr 9m 16Mi
falco-snwqj 8m 16Mi
falco-svswf 7m 16Mi
falco-tgsq8 8m 16Mi

grpc enabled
NAME CPU(cores) MEMORY(bytes)
falco-48zhv 8m 17Mi
falco-8prq5 7m 17Mi
falco-9wtl4 8m 17Mi
falco-ckfl4 1930m 18Mi
falco-fwlzc 1833m 17Mi
falco-gpx6g 7m 17Mi
falco-hrfv8 1825m 18Mi
falco-tv4f2 1004m 17Mi
falco-wx8dx 1003m 17Mi
How to reproduce it

Install falco using the stable helm chart with grpc enabled.
`grpc:
enabled: true
bind_address: "0.0.0.0:5060"
threadiness: 8
private_key: /etc/falco/certs/server.key
cert_chain: /etc/falco/certs/server.crt
root_certs: /etc/falco/certs/ca.crt

grpc_output:
enabled: true`

Expected behaviour

CPU usage should be consistent across all nodes, and should be at a reasonable level.

Screenshots

Environment

k8s 1.16 deployed via kops, standard debian nodes.

• Falco version:
  0.21.0
• System info:
  
• Cloud provider or hardware configuration:
  AWS
• OS:
  Distributor ID: Debian Description: Debian GNU/Linux 9.11 (stretch) Release: 9.11 Codename: stretch
• Kernel:
  4.4.0-1052-aws
• Installation method:
  Kubernetes

Additional context

[fntlnz]

Thanks for reporting @scallopboat – I did some analysis on this and I found that the major bottleneck we have here is the try_pop method on the queue we use to get Falco outputs to the gRPC server ( output::queue )

I don’t see any problems in the push counterpart here and from what I can see by profiling it looks like we call it an enormous amount of time for each iteration. We need to refine the logic here to avoid this.

[scallopboat]

@fntlnz thanks for pinpointing the issue. If everyone is cool with it, I'd like to take a stab at fixing this bug.

[fntlnz]

We are discussing this issue at the community call today if you'd like to join @scallopboat - you can find zoom link and time here

[DannyPat44]

I am running into what I think is this issue on our deployments and I was wondering if someone could help me confirm that this issue is the root cause. We run a Falco Deployment on our GKE cluster (1.14.10-gke.36) in both our Production and Stage Environments. I recently added the Falco Exporter to our Production cluster and have enabled the GRPC server to support it. I am running into an issue where Falco seems to be pinned at a high CPU usage (something not happening in our Staging Cluster where GRCP server is not enabled even though it has similar loads.)

Both Falco and the exporter will start together fine. But after a while, I will see Falco Server spit out the following errors and the servers seem to shutdown causing the exporter to enter a crash loop it can't recover from.

Let me know if I can help test the open fix PR or contribute in any way!

[fntlnz]

Hi @DannyPat44 @scallopboat - there's a set of changes to fix this issue, anyone minds testing it?

Here is the PR: https://github.com/falcosecurity/falco/pull/1241

Here are the artifacts (rpm,deb,tar.gz) p
ackages/falco-0.23.0-44+cd858c0-x86_64.deb
packages/falco-0.23.0-44+cd858c0-x86_64.rpm
packages/falco-0.23.0-44+cd858c0-x86_64.tar.gz

This is the updated client-go to use it client side: https://github.com/falcosecurity/client-go/pull/45

I prepared a docker image to test this (be aware it's not official): docker pull docker.io/fntlnz/falco-dev:pr-1241

For Falco Exporter, @leogr is taking care of that in https://github.com/falcosecurity/falco-exporter/issues/30

Again @leogr for the event-generator https://github.com/falcosecurity/event-generator/pull/31

All this stuff is scheduled to release next week with Falco 0.24.0

[scallopboat]

@fntlnz I downloaded the .deb file and some ad hoc tests against the feat/grpc-unix-socket branch of falco-exporter. I'm seeing about ~5% cpu usage on my machine, and was able to see rule violations coming through the exporter so it looks like a winner from my perspective.

[leodido]

Thanks for try it out and confirming Chris :)

PR #1241 is going to be merged soon! Great work everyone

On Thu, 11 Jun 2020 at 07:42, Chris Meyer [e-mail] wrote:

@fntlnz https://github.com/fntlnz I downloaded the .deb file and some
ad hoc tests against the feat/grpc-unix-socket branch of falco-exporter.
I'm seeing about ~5% cpu usage on my machine, and was able to see rule
violations coming through the exporter so it looks like a winner from my
perspective.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/falcosecurity/falco/issues/1126#issuecomment-642420933,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAA5J43PRTM6T3TUPTV4YIDRWBVCZANCNFSM4LYS5IAQ
.

>

L.