Falco: K8S audit unable to reach Falco due to invalid port in webhook endpoint

[bzurkowski]

Describe the bug

After installing Falco with dynamic K8S auditing enabled, K8S API Server is unable to reach Falco service due to invalid port in Falco webhook endpoint.

Falco Helm chart creates an audit sink with webhook endpoint defined using K8S service reference instead of url reference. According to the documentation that is the correct approach for in-cluster webhook setup, however, for some reason, when using service reference, the specified webhook port (8765) is not respected by the API server - it attempts to communicate with the webhook service using HTTPS on port 443 anyway:

Error in audit plugin 'dynamic_webhook' affecting 1 audit events: Post https://falco.falco.svc:443/k8s-audit?timeout=30s: http: server gave HTTP response to HTTPS client
Impacted events:
AUDIT: id="4d6c692b-b4f6-4228-8461-2fb8a1a82eaa" stage="ResponseComplete" ip="10.10.10.31" method="get" user="system:kube-controller-manager" groups="\"system:authenticated\"" as="<self>" asgroups="<lookup>" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s" response="200"

To solve the issue, I have modified the audit sink with fixed url reference like this:

    clientConfig:
      url: "http://$FALCO_SERVICE_CLUSTERIP:8765/k8s_audit"

Later on, I found out that this solution is actually described in the examples directory.

How to reproduce it

1. Enable dynamic auditing in API Server.
2. Deploy Falco using Helm chart with dynamic K8S auditing enables:

helm install stable/falco \
      --name falco \
      --namespace falco \
      --set auditLog.enabled=true \
      --set auditLog.dynamicBackend.enabled=true \
      --set falco.jsonOutput=true \
      --set falco.httpOutput.enabled=true \
      --set falco.httpOutput.url=<http-output-endpoint>

1. Observe API Server logs.

Expected behavior

Falco installed using Helm chart works out-of-the-box without the need to hack the generated audit sink. Also, better information about the issue (if confirmed).

Environment

• Falco version:

Falco version: 0.18.0

• System info:

{
  "machine": "x86_64",
  "nodename": "falco-5gqdj",
  "release": "3.10.0-693.el7.x86_64",
  "sysname": "Linux",
  "version": "#1 SMP Tue Aug 22 21:09:27 UTC 2017"
}

• Cloud provider or hardware configuration: Kubernetes 1.15
• OS:

NAME="CentOS Linux"
VERSION="7 (Core)"

• Kernel:

Linux node1 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Installation method: Helm chart

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[leodido]

Recently, @leogr has worked a bit on improving the Helm chart. He also used the updated Helm chart in this blog post.

So, it's probably worth giving it a new try ...

[leogr]

Not sure if the updated Helm chart can help with this. Anyway, I was not able to reproduce this issue (quickly tested on a local kind cluster with the latest helm chart), the webhook configuration using the service reference just worked for me.

@bzurkowski Could you retry with the latest version?

[bzurkowski]

@leogr Sure, I will test the latest Helm chart and let you know the result. One question. What Kubernetes version did you use in your setup?

[leogr]

@leogr Sure, I will test the latest Helm chart and let you know the result. One question. What Kubernetes version did you use in your setup?

Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2020-01-14T00:09:19Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

Also, the latest chart uses Falco 0.21.0.

[bzurkowski]

@leogr I retested using Helm chart with Falco 0.21.0 release. The same problem occurs. I guess, the issue might be solved by a newer Kubernetes version? Kubernetes 1.16 vs. 1.17.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[leogr]

Keep pls

On Tue, Jun 23, 2020, 20:32 stale[bot] [e-mail] wrote:

This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs. Thank you
for your contributions.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/falcosecurity/falco/issues/1026#issuecomment-648340473,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAZ34FOBNGQ75JFQ7QDOXYTRYDYNFANCNFSM4KMFRNGQ
.

[lundbird]

I can confirm that on Kube 15.12 with falco 0.21.0 I encounter the same issue which is fixed by manually setting auditLog.dynamicBackend.url

[fntlnz]

This was recently fixed https://github.com/falcosecurity/charts/blob/master/falco/README.md#enabling-k8s-audit-event-support

I suggest everyone here to use the latest helm chart from here https://github.com/falcosecurity/charts
Or the k8s manifests from here https://github.com/falcosecurity/evolution/tree/master/deploy/kubernetes/kernel-and-k8s-audit

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Issues labeled "cncf", "roadmap" and "help wanted" will not be automatically closed. Please refer to a maintainer to get such label added if you think this should be kept open.

[leogr]

Since this is a helm chart related problem - if I'm not wrong - I'm closing this in favor of https://github.com/falcosecurity/charts/issues/7