Enhancements: API Audit Logging

[roberthbailey]

API Audit Logging

• The Kubernetes apiserver can generate audit logs for all requests, and export to a pluggable backend.
• Primary contact (assignee): Tim Allclair @tallclair
• Responsible SIGs: sig-auth
• Design proposal link (community repo): https://github.com/kubernetes/community/blob/master/contributors/design-proposals/auditing.md
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @sttts @soltysh @x13n
• Approver (likely from SIG/area to which feature belongs): @sttts @tallclair
• Feature target (which target equals to which milestone):
  
  
  
  • [x] Alpha release target (1.7)
  
  
  • [x] Beta release target (1.8) - https://github.com/kubernetes/kubernetes/issues/48561
  
  
  • [ ] Stable release target (1.12) - https://github.com/kubernetes/kubernetes/pull/65891
  
  
• Docs: https://github.com/kubernetes/kubernetes.github.io/blob/master/docs/tasks/debug-application-cluster/audit.md

[amouat]

What's the current status of this? It's not clear to me who's working on what or what the next steps are. @soltysh came up with a working PR that has had a lot of feedback and discussion: https://github.com/kubernetes/kubernetes/pull/27087

[sttts]

@amouat in the mentioned PR @soltysh introduces what we call "basic auditing", basically access.log-style logging only without any deeper api knowledge. To my knowledge mainly log-rotation is an open issue.

https://github.com/kubernetes/kubernetes/pull/29443 is the continuation by me and @soltysh describing more "advanced auditing" where the basic audit output would just be a special case. This feature issue is about the latter and will link to that proposal PR once it's more complete.

[amouat]

Thanks!

I have to say this process is very confusing. The discussion has moved from issue #2203, to PR #27087 to this issue and then to #29443, with no clear indication on each where the current discussion is happening, or what the next steps are :(

Many thanks for you work on this though, I don't mean to sound ungrateful towards a great OS project. I just wanted to check that this issue was still moving forward.

[janetkuo]

@soltysh @sttts Are the docs ready? Please update the docs to https://github.com/kubernetes/kubernetes.github.io, and then add PR numbers and have the docs box checked in the issue description

[sttts]

@janetkuo this feature is postponed to 1.5, in 1.4 we only have https://github.com/kubernetes/kubernetes/pull/27087 as a first step. Unfortunately, I lack the permissions to change the milestone.

[soltysh]

I've changed both the labels and milestone. Although it would be good to have at least the small part documented. I'll create a PR right away.

[soltysh]

Created https://github.com/kubernetes/kubernetes.github.io/pull/1168 for the basic audit part.

[goltermann]

I added the alpha-in-1.4 label, as we got _some_ of this done in 1.4. It might be a stretch to call it alpha, but I don't want to lose that we shipped some working pieces of this for 1.4.

[soltysh]

Yeah, the _some_ is quite a stretch here, but I'm ok with it.

[idvoretskyi]

@soltysh @sttts can you provide the actual status of the feature for the 1.5 release (is it alpha, beta, etc)?

[soltysh]

Unfortunately this is stuck in alpha, no work has been done recently with it :sob:

[davidopp]

Maybe of interest to @kubernetes/sig-instrumentation ?

[idvoretskyi]

@soltysh @davidopp so, I'll target this one to the next milestone.

[sandys]

hey guys - this is very important for us since we are planning to financial services application on k8s. I realize that this may take a while to make it in. I hope im not destroying the conversation here.. but what are people using today to do this kind of logging ?

A lot of people use bastion hosts to run kubectl - are you guys logging commands on that server, etc ? it would be good to know some practical examples.

[soltysh]

This feature is in its very early stage, the docs for the current state are here.

[idvoretskyi]

@timothysc why this feature has been added to 1.7 milestone? We haven't had any public agreement at this thread to have it for 1.7 during the release timeframe, while the features freeze deadline has already passed.

[timstclair]

The feature is already implemented (alpha stage) in 1.7. I think it was an oversight that it wasn't originally tagged in the v1.7 release.

[idvoretskyi]

@timstclair thank you for clarifying. Please, update the feature description with the new issue template.

[ericchiang]

Updating label to stage/beta for the 1.8 release. Beta goals here: https://github.com/kubernetes/kubernetes/issues/48561

[soltysh]

This come up during the discussion we've had today. The initial description mentions we want to promote this to GA in the next release (1.9). But there are no clear requirements in place for this to happen. Before we commit to an actual release I'd like us to define those requirements. Of top of my head there is:

1. SLO - we need to be able to define one, but for that tests are needed to be performed.
2. Properly working audit in multi-apiserver and federated environment.
3. Filters (first of all we need to decide how they should work and what to expect from them and update the proposal with this see here and only then start the implementation).

@sttts @ericchiang @crassirostris @piosz @loburm thoughts?

[soltysh]

I've updated the milestone, since this was mentioned to be 1.8 beta feature.

[tallclair]

I agree with the need to define the requirements to get this to GA. However, I disagree that filters should be a requirement without some specific real-world data (i.e. customers asking for this with a specific use case). From what I've seen so far, the policy can cover 90% of the uses for filters. Also, I think audit filters can rollout out after advanced auditing launches to GA.

[sttts]

2) The multi-apiserver scenario including fereration should at least be sketched out such that we know to have a way forward later. In fact, I would like to see that even now with beta.
3) +1 that filters can be added into v1 in a later release.

[crassirostris]

The multi-apiserver scenario including fereration should at least be sketched out such that we know to have a way forward later. In fact, I would like to see that even now with beta.

I created an issue earlier to have a place to discuss this problem: https://github.com/kubernetes/kubernetes/issues/50076

[soltysh]

+1 that filters can be added into v1 in a later release.

+1

I'm guessing we have a clear reqs for GA, in that case. That is:

1. SLO
2. multi-apiserver + federation covered

[soltysh]

Since I think we have all the bits necessary for 1.8 milestone I'll go ahead and modify it to 1.9 now. So that folks responsible for tracking features get this off their list.

[2rs2ts]

Not sure if this is the right place to raise this issue but the log rotation implemented by the apiserver seems to be incompatible with logrotate and there doesn't seem to be a way to turn it off. This feature would probably be better off if it could work with logrotate.

[crassirostris]

@2rs2ts Could you please create a separate issue for that in kubernetes repo? Let's continue the discussion there

[zacharysarah]

@roberthbailey :wave: Please open a documentation PR and add a link to the tracking spreadsheet. Thanks in advance!

[crassirostris]

@zacharysarah Opened a PR (https://github.com/kubernetes/website/pull/6427), added a link to the spreadsheet

Thanks for reminding!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[crassirostris]

/remove-lifecycle stale

[justaugustus]

@timstclair @crassirostris @soltysh
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

cc @idvoretskyi

[crassirostris]

/cc @loburm

[justaugustus]

@tallclair @x13n @CaoShuFeng --
Feature Freeze is today. Are we planning on graduating this feature in Kubernetes 1.12?
If so, can you make sure everything is up-to-date, so I can include it on the 1.12 Feature tracking spreadsheet?

[loburm]

@justaugustus yes this is in plans. PR is already in review:
https://github.com/kubernetes/kubernetes/pull/65891

[justaugustus]

Thanks for the update!

/remove-stage beta
/stage stable

[zparnold]

Hey there! @roberthbailey I'm the wrangler for the Docs this release. Is there any chance I could have you open up a docs PR against the release-1.12 branch as a placeholder? That gives us more confidence in the feature shipping in this release and gives me something to work with when we start doing reviews/edits. Thanks! If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it?

[roberthbailey]

@tallclair is the primary assignee; I just created the initial issue.

[tallclair]

@loburm @x13n @CaoShuFeng - Can one of you volunteer to own the v1.12 docs for this feature?

[CaoShuFeng]

Can one of you volunteer to own the v1.12 docs for this feature?

I will do it.

[CaoShuFeng]

Can one of you volunteer to own the v1.12 docs for this feature?

I found that these two pull requests need document:
https://github.com/kubernetes/kubernetes/pull/65862
https://github.com/kubernetes/kubernetes/pull/65763
I will update the document once they get merged.

The dynamic audit documentation is here: https://github.com/kubernetes/website/pull/9947

[zparnold]

Thank you!

On Tue, Aug 21, 2018 at 10:13 PM CaoShuFeng [e-mail]
wrote:

Can one of you volunteer to own the v1.12 docs for this feature?

I found that these two pull requests need document:
kubernetes/kubernetes#65862
https://github.com/kubernetes/kubernetes/pull/65862
kubernetes/kubernetes#65763
https://github.com/kubernetes/kubernetes/pull/65763
I will update the document
https://github.com/kubernetes/website/pull/9953 once they get merged.

The dynamic audit documentation is here: kubernetes/website#9947
https://github.com/kubernetes/website/pull/9947

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/features/issues/22#issuecomment-414895057,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE81SPi0lMeSJ3iqUIkvBoJhy9XZhtlBks5uTMxxgaJpZM4JK333
.

[justaugustus]

@CaoShuFeng @tallclair --
Any update on docs status for this feature? Are we still planning to land it for 1.12?
At this point, code freeze is upon us, and docs are due on 9/7 (2 days).
If we don't here anything back regarding this feature ASAP, we'll need to remove it from the milestone.

cc: @zparnold @jimangel @tfogo

[CaoShuFeng]

The document is ready for review: https://github.com/kubernetes/website/pull/9947

https://github.com/kubernetes/kubernetes/pull/65763 not included yet.

[justaugustus]

Thanks for the update!

[justaugustus]

Dropping this from the milestone per the feedback here: https://github.com/kubernetes/website/pull/9947#issuecomment-418939135

/milestone v1.13

[tallclair]

As this has graduated to stable I'm going to close this feature (woohoo!)

Future enhancements should be tracked as separate features (e.g. Dynamic Audit Configuration).

Thanks to everyone who worked on this! :tada:

[kacole2]

/milestone clear