Enhancements: Distroless images

[justaugustus]

Enhancement Description

• One-line enhancement description (can be used as a release note): Rebase Kubernetes Main Master and Node Images to distroless/static
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1729-rebase-images-to-distroless
• Primary contact (assignee): @justaugustus
• Responsible SIGs: SIG Release, Product Security Committee
• Enhancement target (which target equals to which milestone):
  
  
  
  • Alpha release target: 1.15
  
  
  • Beta release target: 1.19
  
  
  • Stable release target: ???
  
  

_Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement_

Will update. Tracks https://github.com/kubernetes/kubernetes/issues/70249.
/area security
/sig release

[k8s-ci-robot]

@justaugustus: The label(s) area/security cannot be applied, because the repository doesn't have them

In response to this:

Enhancement Description

• One-line enhancement description (can be used as a release note):
• Kubernetes Enhancement Proposal: (link to kubernetes/enhancements file, if none yet, link to PR)
• Primary contact (assignee):
• Responsible SIGs: SIG Release
• Enhancement target (which target equals to which milestone):
  
  
  
  • Alpha release target (x.y)
  
  
  • Beta release target (x.y)
  
  
  • Stable release target (x.y)
  
  

_Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement_

Will update. Tracks https://github.com/kubernetes/kubernetes/issues/70249.
/area security
/sig release

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[justaugustus]

/committee product-security

[justaugustus]

/assign

Spoke w/ @tallclair earlier and I'm going to take this one over since @dims and I have been making some forward progress here, tracked in https://github.com/kubernetes/kubernetes/issues/70249 + https://github.com/kubernetes/kubernetes/issues/58012 and some other threads that I need to tie together.

cc: @yuwenma @dekkagaijin

@palnabarun -- This is going to require an exception, which I'm not quite ready to file, but will next week. Just putting it on your radar in the meantime...

/milestone v1.19
/stage beta

[palnabarun]

@justaugustus -- Thanks for the ping! :+1:

I have added this enhancement to the tracking sheet and noted that an exception would be filed in the very near future.

I'm assuming this is the KEP for this enhancement.

[Conan-Kudo]

Are we using these images because we think that they're better? What makes them better? The KEP seems to just say "thinner and lighter" and mumbles about attack surface. Do we actually have real problems we're trying to solve here? Forgive me if I missed some discussion that can be pointed to about this, the PR about the KEP didn't seem to have any of that...

[tallclair]

Motivations:

• "thinner and lighter" - not a big win once you take shared base layers into account, but it's a bit better
• mumble mumble attack surface - maybe, but most (all?) of the dependencies that get pulled in by heavier images aren't actually running, so it would be hard to exploit a vulnerability in them
• fewer tools available to the attacker - even if an attacker gets code execution, the code that can be executed is limited
• CVE noise - Kubernetes users often (and should) run a vulnerability scanner against containers in the cluster, and keeping control plane images free of known CVEs is an unnecessary hassle.

[Conan-Kudo]

@tallclair Honestly, if they own the image, they own the kubelet. If all your stuff lives in Kubernetes, I think it's pretty much game over anyway.

[Conan-Kudo]

Erk, I mean if they own the kubelet (the actively running thing in the image), then it doesn't matter. It's game over, and what else your image has doesn't matter.

[justaugustus]

That does not and should not deter us from making efforts to ensure the images are more secure.

[Conan-Kudo]

Sure, but I still don't see how "distroless" images do that. Part of my skepticism comes from having looked into how those images are built. It's hard to trust images that just download and inject goop into the image that would be hard to reliably audit in the first place. 😉

[dims]

@Conan-Kudo can you please give a specific example in kubernetes/kubernetes that you are uncomfortable with?

[Conan-Kudo]

It's not bad at the k8s level. My problem is mostly with how the actual base images are constructed. The "fetching and installing packages while not respecting dependencies" and the occasional "fetching and injecting from the internet" means that it's hard for me to trust it. It may be fine for k8s, given that most of it is in Go, but the inherent broken dependencies thing in distroless images makes me wary.

[Conan-Kudo]

The above also worries me for another reason: they chose Debian as the base of distroless, and yet continue to hacksaw it instead of working in Debian to fix things. I personally favor Fedora for my stuff, but when I find issues with images being too big in Fedora, instead of doing hacksaw maneuvers, I actually go and try to fix it in Fedora itself. The distro even has a minimization project to cut down the dependency web for containers to the absolute bare minimum when desired. In my view, this is the right approach to solving this problem. A Fedora variant of "distroless" would actually be ~40MB, based on their own tooling with no hacks.

[dims]

"they chose Debian" .. you are talking about https://github.com/GoogleContainerTools/distroless/search?q=stretch&unscoped_q=stretch ?

@Conan-Kudo you can and should have your own CI/CD pipeline with your own images if you are doing anything meaningful and not rely on someone else's image. period. If our k8s makefiles/dockerfiles do not support it, then please report a bug and we will fix it. AFAIK, all our plumbing allows injecting base images of your own. So please do that.

[Conan-Kudo]

@dims Yes. And of course, I have my own pipelines for making images. I'm not lazy. 😉

But this is why I brought up the Fedora example. Because of the work by @asamalik and others, I have been able to make really tiny images in my own pipelines without hacksaw techniques, and that means the images are much easier to verify, validate, and audit. 😄

And while the tooling allows injecting my own base for this, _defaults matter_. Lots of people don't change a thing here, and that's why I'm talking about this at all.

[dims]

@Conan-Kudo i'd request you to follow this up with an issue in GoogleContainerTools/distroless since you are worried about the techniques they use. If this is a request for kubernetes to switch over from those images to something else, we need to file a KEP to discuss options/possibilities etc. It's going to require a community decision and KEP's are the way we do it.

[Conan-Kudo]

I have no illusions that the GCP team would care about what I think. I may consider filing a KEP, once I'm a little more comfortable with my understanding of the process.

[tallclair]

@Conan-Kudo I think your concerns may be with the distroless images in general, but maybe aren't applicable to the distroless:static image that we're actually using. The static image doesn't have any executables or libraries, and therefore doesn't have any dependencies that can be broken. It is meant as a base for statically compiled binaries only (no dependencies).

[Conan-Kudo]

@tallclair Perhaps, it'd also depend on how the Go compiler was sourced, but that does go quite a long way to alleviate issues... In general, I think the distroless images are a poor choice for reliable containers (for the reasons I outlined above), but naturally it's possible to mitigate those...

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[dekkagaijin]

/remove-lifecycle stale

[kikisdeliveryservice]

Hi @justaugustus :wave:

Enhancements Lead here :smile: Do you plan on any work (beta/GA) for 1.20?

Thanks!
Kirsten

[kikisdeliveryservice]

/milestone clear

(removing this enhancement issue from the v1.19 milestone as the milestone is complete)

[kikisdeliveryservice]

Hi @justaugustus

Circling back around - do you want this included in 1.20?

Thanks,
Kirsten

[justaugustus]

@kikisdeliveryservice -- yep, I'm continuing (and hopefully wrapping up) work on this in v1.20
/milestone v1.20

[kikisdeliveryservice]

hey @justaugustus

Looking at the kep the graduation criteria need to be updated it only refs 1.15: https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1729-rebase-images-to-distroless#graduation-criteria

I see that we have a list of rebased images refed in graduation criteria: https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1729-rebase-images-to-distroless#rebased-images

Is there something else that needs to be added then?

Does this also need a testing plan? Or are the existing tests passing fine? Should we update the KEP to say that if that's the case?

As a reminder freeze is October 6th (but you know that already 😆 )

Thanks!
Kirsten

[kikisdeliveryservice]

ping @justaugustus could you address the comments/questions above?

[kikisdeliveryservice]

@justaugustus Enhancements Freeze is EOD PST tomorrow October 6th. Could you lmk the answers to my Qs above?

[kikisdeliveryservice]

Hi @justaugustus

Enhancements Freeze is now in effect. Unfortunately, your KEP still has outstanding issues. If you wish to be included in the 1.20 Release, please submit an Exception Request as soon as possible.

Best,
Kirsten
1.20 Enhancements Lead

[justaugustus]

@kikisdeliveryservice -- Apologies for the hold here; I've been busy with KubeCon prep.
Filed an exception for 1.20 here: https://groups.google.com/g/kubernetes-sig-release/c/mYGxI8ixeTI

[BenTheElder]

@tallclair Perhaps, it'd also depend on how the Go compiler was sourced, but that does go quite a long way to alleviate issues...

The Go compiler is not inside the distroless images and the binaries we add in images built on top of them have CGO disabled.
There are no libraries and no executables in the distroless:static based images, only data files like ca-certificates.

It's actually important that these containers have no binaries / shared libraries in them ... the whole idea is that even if an attacker somehow gained execution in a Kubernetes binary in one of these images, they have no userspace to work with, so we don't want even a 40MB userspace, we want NO userspace.

It may be fine for k8s, given that most of it is in Go, but the inherent broken dependencies thing in distroless images makes me wary.

ALL of what is in the images migrated so far is in go and only go, and is 100% statically linked. These images add tzdata and ca-certificates. Tracking debian's packages for these seems like a very reasonable approach for both projects.

---

@justaugustus has anyone written up a solution for kube-proxy yet? I have some thoughts...

[justaugustus]

@justaugustus has anyone written up a solution for kube-proxy yet? I have some thoughts...

I'm fairly certain no one has thought through that part to completion. I'm all ears!

[kikisdeliveryservice]

Your Exception Request was approved so moved this back into 1.20 :+1:

[justaugustus]

Thanks Kirsten!

[kikisdeliveryservice]

@justaugustus we both know you know this but I'm telling you anyway :wink:

Since your Enhancement is scheduled to be in 1.20, please keep in mind the important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thanks!
Kirsten

[eagleusb]

Hello @justaugustus :wave:, 1.20 Docs shadow here.

Does this enhancement work planned for 1.20 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.20 branch in the k/website repo.

This PR can be just a placeholder at this time and must be created before Nov 6th

Also take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[kikisdeliveryservice]

Spoke with @justaugustus and the deliverables for this are to update the KEP to reflect the relevant images used for release (current KEP is overbroad & uses non-essential ones) as well as make updates similar to: https://github.com/kubernetes/sig-release/pull/1084

@eagleusb there will be no k/website docs for this issue in 1.20

[annajung]

Hi @justaugustus

Please keep in mind the important upcoming dates:

• ~Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline~ Docs Not Required
• Thursday, Nov 12th: Week 9 - Code Freeze
• ~Monday, Nov 23rd: Week 11 - Docs PR Ready for Review~ Docs Not Required

As a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

[kikisdeliveryservice]

Hey @justaugustus :smile_cat:

Just a reminder that Code Freeze is coming up in 2 days on Thursday, November 12th. All PRs must be merged by that date, otherwise an Exception is required.

Do you have any PRs open for this?

Thanks!
Kirsten

[justaugustus]

I'll wrap release docs for this in v1.21. Just ran out of time here with KubeCon looming.
/milestone v1.21