Falco: Falco crashing on Kubernetes with connection timeout

[hus787]

What happened:

Falco crashing. ~Once every 15mins.

What you expected to happen:

Falco not crash and handle and prevent connection timeouts gracefully.

How to reproduce it (as minimally and precisely as possible):

helm install --name falco -f deployment/falco_config.yaml stable/falco
cat deployment/falco_config.yaml

ebpf:
  enabled: true
resources:
  requests:
    cpu: 50m
    memory: 128Mi
  limits:
    memory: 600Mi
image:
  tag: dev
falco:
  jsonOutput: true
fakeEventGenerator:
  enabled: false
customRules:
  rules-fluentd.yaml: |-
    - rule: Clear Log Activities
      append: true
      condition: and not k8s.ns.name=kube-system

Environment:

• Falco version (use falco --version):
• System info
• Cloud provider or hardware configuration: Google (GKE)
• OS (e.g: cat /etc/os-release): COS (build 11647.121.0)
• Install tools (e.g. in kubernetes, rpm, deb, from source): Kubernetes 1.13.5-gke.10
• Node type: g1-small

[fntlnz]

@hus787 thanks for opening! Can you post the system info output please?

[hus787]

Welcome :)

can you provide the command whose specific output you would like to see? I don't understand what you specifically mean here by system.

[leodido]

@hus787 I think @fntlnz was referring to the output provided by
falco --support | jq .system_info and falco --version in first instance :)

[hus787]

falco --support | jq .system_info

{
  "machine": "x86_64",
  "nodename": "gke-REDACTED-cluster-default-pool-5f3e8946-1sxg",
  "release": "4.14.127+",
  "sysname": "Linux",
  "version": "#1 SMP Tue Jun 18 18:32:10 PDT 2019"
}

falco --version

falco version 0.1.2811dev

[fntlnz]

/area integrations

[fntlnz]

@gnosek do you have any pointer on this?

[fntlnz]

@hus787 can you try with a newer release? 0.15.3 is out.

We also need more details, for example can you post here the output when it crashes?
Any information will be helpful.

[fntlnz]

/triage needs-information

[hus787]

I can't use 0.15.3 on COOS because it doesn't contain the fix for https://github.com/falcosecurity/falco/issues/425#issuecomment-497986111

Dev apparently fixes the issue.

Output:

Mon Jul 1 09:35:58 2019: Runtime error: Socket handler (k8s_daemonset_handler_event) an error occurred while connecting to https://10.104.0.1: Connection timed out. Exiting.

[fntlnz]

oh @hus787 that had been solved in 0.15.3 ! You just have to specify the environment variable in your daemonset.

Read this comment for reference: https://github.com/falcosecurity/falco/issues/650#issuecomment-501541258

I'm sorry that is not in the docs, we need to get it updated here's the issue: https://github.com/falcosecurity/falco-website/issues/37

[mfdii]

I'm guessing there's a "bug" in the logic that's used to connect to the k8s cluster. If there's a timeout, falco shouldn't crash (IMO). It should log that the connection timed out, retry, and then continue logging the retry messages until it's successful.

I've seen this myself from time to time on COS but it doesn't appear to be consistent in how often pods crash.

[fntlnz]

An exponential backoff seems a good and easy solution to fix this one.

[fntlnz]

Hi @hus787 can you try with the current dev branch? We did a lot of fixes in the webserver recently.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[fntlnz]

I'm closing since @hus787 didn't get back to us in 5 months and we did a lot of improvements to the webserver in the meanwhile.