Falco: Falcos K8s Audit feature won't work anymore in the future (K8s version >= 1.19)

[PhilipSchmid]

Hi there,

Motivation

As you instruct to do so when someone wants to use the Falco K8s Audit feature (see here), one need to enable dynamic auditing backends via the following kube-apiserver parameters first:

--audit-dynamic-configuration
--feature-gates=DynamicAuditing=true
--runtime-config=auditregistration.k8s.io/v1alpha1=true

While that's a valid approach for e.g. K8s versions 1.17 & 1.18, I think it doesn't work anymore in version 1.19. The v1.19.0-alpha.3 release notes says:

The alpha DynamicAuditing feature gate and auditregistration.k8s.io/v1alpha1 API have been removed and are no longer supported. (#91502, @deads2k) [SIG API Machinery, Auth and Testing]

Here the linked Github issue: https://github.com/kubernetes/kubernetes/pull/91502
Here the "discussion": https://groups.google.com/forum/#!topic/kubernetes-sig-auth/aV_nXpa5uWU

Feature

The Falco K8s audit feature should still be supported and possible in the future.

Alternatives

The linked mailing list post states the same functionality would also be possible via the existing audit webhook support. Would might be an option for Falco to migrate to?

Thanks!

Regards,
Philip

[leogr]

IIRC the existing audit webhook could work with some caveats.

AFAIK, to enable the webhook (in a non-dynamic fashion), you need to change the arguments to the kube-apiserver process to add --audit-webhook-config argument and provide a webhook configuration (as like as we did in this very old example).

Unfortunately, we need the Falco service IP in the aforementioned configuration, but - since Falco runs inside the cluster - the IP is unknown beforehand (and a cluster DNS cannot be used, since the apiserver is expecting something that's outside the cluster). There're workarounds for that (like restarting the apiserver), but no one seems to be a reliable solution, IMHO.

Moreover, it seems the Falco's web server has a problem with SSL ( see https://github.com/falcosecurity/falco/issues/860 and https://github.com/falcosecurity/charts/pull/71), although it should work even without SSL enabled (not sure).

That being said, if there was a reliable way to configure the webhook, Falco should work. In that case, it's just a matter of enabling the helm chart to work so (ofc, our example manifest files will need to be updated accordingly).

/cc @maxgio92
/cc @nibalizer

[antoinedeschenes]

SSL isn't required when hardcoding the webhook service IP address.

Since apiserver is running in host-networking mode, a NodePort service or having the falco pod in host-networking mode could make sense as an alternative to setting the service IP in the k8s api configuration after deployment

[leogr]

It's a good topic to talk about in tomorrow's community call, it would be great if you'll join us.

[maxgio92]

As discussed with @leogr the biggest problem is probably the inability to fully configure the control plane in managed solutions.
The dynamic audit proxy proposal seems the more clean but is just a proposal...

[ananth07reddy]

@leogr Any updates on this ticket?

[leogr]

Hey @ananth07reddy

Not yet, unfortunately. Moreover, since the issues discussed above (and also the fact that the current webserver implementation comes with some caveats), at this point, I believe we should completely re-think the way this "input source" works and evaluate several different approaches.

That being said, Falco's K8s Audit is a feature we still want to support, but I guess we need some help on that.
Any volunteers? :smile_cat:

[antoinedeschenes]

I'll be testing the Falco webserver exposed as a NodePort soon.

[antoinedeschenes]

k8s auditing still works and will still work as expected with the pre k8s-1.13 method that didn't use AuditSink

Just setup the falco service as a NodePort (or run directly on the host) and kube-apiserver will be able to reach it without setting a ClusterIP manually in the cluster config.

One needs to run api server with:

    --audit-log-max-backups=1
    --audit-log-max-size: 10
    --audit-log-path: /var/log/k8s-audit.log
    --audit-policy-file: /var/lib/k8s-audit/audit-policy.yaml
    --audit-webhook-batch-max-wait: 5s
    --audit-webhook-config-file: /var/lib/k8s-audit/webhook-config.yaml

webhook-config.yaml should look like:

apiVersion: v1
kind: Config
clusters:
- name: falco
  cluster:
    # certificate-authority: /path/to/ca.crt # for https
    server: http://127.0.0.1:32765/k8s-audit
contexts:
- context:
    cluster: falco
    user: ""
  name: default-context
current-context: default-context
preferences: {}
users: []

audit-policy.yaml: use https://raw.githubusercontent.com/falcosecurity/evolution/master/examples/k8s_audit_config/audit-policy.yaml

I have a falcosecurity/charts PR with the nodePort change and kops documentation:
https://github.com/antoinedeschenes/falcocharts/tree/falco-webserver-ssl/falco#manual-setup-on-kops

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[PhilipSchmid]

/remove-lifecycle stale