Kubeadm: CIS Compliance for kubeadm

[jaxxstorm]

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST:

I'd like to start working on some of the default configuration that kubeadm generates for the manifest files. Currently, the defaults aren't especially secure.

The CIS benchmarks for kubernetes are publicly available, and a lot of the issues with a kubeadm generated cluster can be fixed by updating the default configurations here: https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/controlplane/manifests.go

Some of these configuration options require a bit more work than others to enable, but some (like audit log configuration) are very simple to get started with.

Any thoughts welcomed.

[fabriziopandini]

@jaxxstorm AFAIK audit will be fixed on 1.10 as well as etcd ca.

However this is a really interesting benchmark, and IMO it will be great if you can transform this issue into a checklist of actionable items.

PS. this could be also a relevant topic for kubeadm office hours meetings

[timothysc]

/assign @liztio

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[jaxxstorm]

/remove-lifecycle rotten

[neolit123]

@jaxxstorm
kubeadm is passing the k8s conformance tests.
https://k8s-testgrid.appspot.com/sig-testing-kind#conformance,%20master%20(dev)

so my question here would be why isn't CIS compliance as part of conformance tests and what does the CNCF thinks about that?

cc @spiffxp

the CIS benchmarks for kubernetes are publicly available

i cannot seem to find a link for this.

if we end up with CIS related conformance tests failing for kubeadm, actions will be taken.
until then, please enumerate items in manifests.go that you think are problematic and we can discuss them.

[neolit123]

^ cc @raesene

[luxas]

I'm gonna own/track this work for v1.14.
kube-bench hasn't updated their benchmarks since v1.11, I'll get in touch with the maintainers.
I've ran the v1.11 benchmarks on a local kubeadm cluster, and it looks pretty good.
We'll do some minor changes, and patch some false negatives in the CIS spec itself.

i cannot seem to find a link for this.

https://github.com/aquasecurity/kube-bench/tree/master/cfg

so my question here would be why isn't CIS compliance as part of conformance tests and what does the CNCF thinks about that?

The CIS benchmark would be additive to conformance. In other words, you can be conformant k8s without passing all the benchmarks, but once you do, you'll comply with this extra "profile". The profiles works on k8s conformance isn't fully proposed/finished yet.

I'll put together a list of actionable items, in fact, I have a rough draft of this locally in my computer.
cc @lizrice FYI

[luxas]

also xref this old issue we had: https://github.com/aquasecurity/kube-bench/issues/65

[raesene]

So Kube-bench hasn't updated past 1.11, I'd imagine, 'cause the CIS spec. hasn't been updated yet for 1.12 or 1.13.

In the past we've not done a new version of the CIS benchmark for every Kubernetes release, although we could look to do that I guess, as hopefully there shouldn't be too many changes for each one.

I've already started some work on adapting items where compliance with the CIS benchmark would result in broken clusters, so for example 2.1.1 in the current version recommends setting --allow-privileged to false. We can handle requirements for restricting privileged containers using PSP now, so it makes sense to remove that recommendation.

Feel free to ping me on items you think might need changed, I've kicked off the process of the next version on the CIS site. Also you can sign up at https://workbench.cisecurity.org if you'd like to look at the draft / make recommendations there.

[neolit123]

hi, what is the state of this issue?

[lizrice]

I think it would be fair to say that updates to the CIS Benchmark are nearing completion, thanks in large part to @raesene. In kube-bench we stick to what's published in the benchmark, so when that's ready we'll release a new set of test config files to match.

[neolit123]

moving out of the 1.14 milestone for kubeadm.
1.11 is going outside of support skew with the 1.14 release.

[neolit123]

/assign @yastij

[neolit123]

spoke with @yastij on zoom and we might get this enabled in testgrid/prow in the near future.

[fabriziopandini]

@neolit123 @yastij I'm happy to help in setting up test machinery (hopefully re-using kinder)

[neolit123]

added P0 prio as per the 1.16 planning.

[neolit123]

in 2019 the topic of CIS compliance was brought in the kubeadm office hours discussing some technical aspects, proposing changes to the benchmark itself. we are not aware if any changes were made in kube-bench since then.

in 2019 during a steering committee meeting, there was a discussion whether CIS is officially approved as a k8s-wide method of security benchmark. the present members at the time did not confirm that.

closing in favor of:
https://github.com/kubernetes/kubeadm/issues/1649

where kubeadm can provide a guide of documenting how to create a CIS compliant cluster.
but if different versions of kubeadm require different guides, this might be a bit too difficult for the kubeadm developers to maintain, so contributions are welcome.

/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

in 2019 the topic of CIS compliance was brought in the kubeadm office hours discussing some technical aspects, proposing changes to the benchmark itself. we are not aware if any changes were made in kube-bench since then.

in 2019 during a steering committee meeting, there was a discussion whether CIS is officially approved as a k8s-wide method of security benchmark. the present members at the time did not confirm that.

closing in favor of:
https://github.com/kubernetes/kubeadm/issues/1649

where kubeadm can provide a guide of documenting how to create a CIS compliant cluster.
but if different versions of kubeadm require different guides, this might be a bit too difficult for the kubeadm developers to maintain, so contributions are welcome.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.