Enhancements: Pid Limiting

[derekwaynecarr]

Enhancement Description

• Ability to isolate pid resources pod-to-pod and node-to-pod
• Primary contact (assignee): @derekwaynecarr
• Responsible SIGs: sig-node
• Design proposal link (community repo): https://github.com/kubernetes/enhancements/pull/755
• Link to e2e and/or unit tests: https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/pids_test.go
• KEP: 20190129-pid-limiting
• Reviewer(s) - @dashpole @sjenning
• Approver (likely from SIG/area to which enhancement belongs): @dashpole @dchen1107

• Enhancement target (which target equals to which milestone):

  • Alpha 1.10
  • Beta 1.15
  • Stable release target 1.20

• Documentation PR: https://github.com/kubernetes/website/pull/23929

[derekwaynecarr]

/sig node

[derekwaynecarr]

/assign @dashpole @dchen1107

[ameukam]

/kind feature

[ameukam]

Hello @derekwaynecarr, 1.14 enhancement shadow here. Code Freeze is March 7th and all PRs must be merged by then to your issue to make the 1.14 release. What open K/K PRs do you still have that need to merge? Thanks

[simplytunde]

Hey @derekwaynecarr. Just a friendly reminder we're looking for a PR against k/website (branch dev-1.14) due by Friday, March 1. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[sjenning]

https://github.com/kubernetes/kubernetes/pull/73651 implements node-to-pid isolation for 1.14.
@RobertKrawitz could you open a PR against kubernetes/website for documenting this. Doesn't have to be the full thing yet. I think @simplytunde just needs the PR for release tracking atm.

[RobertKrawitz]

https://github.com/kubernetes/website/pull/12932 is a placeholder.

[RobertKrawitz]

Above PR is complete and waiting for review/approval.

[derekwaynecarr]

https://github.com/kubernetes/website/pull/13006 as well

[kacole2]

Hello @derekwaynecarr, I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet.

/milestone clear
/tracked no

[derekwaynecarr]

we are graduating the node to pod pid isolation to beta for 1.15 per the kep.

/milestone 1.15

[k8s-ci-robot]

@derekwaynecarr: The provided milestone is not valid for this repository. Milestones in this repository: [keps-beta, keps-ga, v1.15, v1.16]

Use /milestone clear to clear the milestone.

In response to this:

we are graduating the node to pod pid isolation to beta for 1.15 per the kep.

/milestone 1.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[derekwaynecarr]

/milestone v1.15

[derekwaynecarr]

link to merged kep
https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/20190129-pid-limiting.md#version-115

[daminisatya]

Hey, @derekwaynecarr @dashpole @dchen1107 I'm the v1.15 docs release shadow.

I see that you are targeting the beta version of this enhancement for the 1.15 release. Does this require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions! 😄

[RobertKrawitz]

For node PID limiting, see https://github.com/kubernetes/website/pull/14425

[kacole2]

Hi . Code Freeze is Thursday, May 30th 2019 @ EOD PST. All enhancements going into the release must be code-complete, including tests, and have docs PRs open.

Please list all current k/k PRs so they can be tracked going into freeze. If the PRs aren't merged by freeze, this feature will slip for the 1.15 release cycle. Only release-blocking issues and PRs will be allowed in the milestone.

If you know this will slip, please reply back and let us know. Thanks!

[kacole2]

Hi @RobertKrawitz @derekwaynecarr , today is code freeze for the 1.15 release cycle. I do not see a reply for any k/k PRs to track for this merge. The only one mentioned previously is https://github.com/kubernetes/kubernetes/pull/73651 that's been merged. Are there any others to track? After this point, only release-blocking issues and PRs will be allowed in the milestone with an exception.

[RobertKrawitz]

For node PID limiting, the graduate support PR is https://github.com/kubernetes/kubernetes/pull/76221 (which is merged).

[kacole2]

Hi @derekwaynecarr @RobertKrawitz , I'm the 1.16 Enhancement Lead. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.6 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[iot86]

I thought some test results concerning using --pod-max-pids flag might be appropriate. I created 2 java apps one which creates threads as fast as it can and the other slowly creates threads. I ran both apps on a linux centos node that did not have the pid restriction set and then ran the same apps on a node that did not have any limits on pids.

Results:

On the node with no pid limit I quickly caught the OutOfMemoryError "Unable to create native thread" with the java app that quickly spun up threads. It also effected the 2nd java app creating threads slowly.

...
Thread 29442 is running
Thread 29443 is running
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at MainNoSleep.main(MainNoSleep.java:31)

// The app creating threads slowly is also effected
...
Thread 15 is running
Thread 16 is running
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at MainSleep.main(MainSleep.java:31)
[centos@ip-10-250-216-28 ~]$

When I ran the same test on the node that had pid limits in place the app that was spinning up threads like crazy was stopped after about 100 threads matching the pid limit previously set of 100. "The error was the same java.lang.OutOfMemoryError: unable to create new native thread" but the second app creating threads slowly was running fine. So it appears that using the pid limit does allow multiple container to use host resources efficiently.

...
Thread 94 is running
Thread 95 is running
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at MainNoSleep.main(MainNoSleep.java:31)
Thread 96 is running

// the second app continues to run fine
...
Thread 33 is running
Thread 34 is running
[centos@ip-10-250-216-28 ~]$

[annajung]

Hello @derekwaynecarr @RobertKrawitz , 1.17 Enhancement Shadow here! 🙂

I wanted to reach out to see *if this enhancement will be graduating to alpha/beta/stable in 1.17?

*
Please let me know so that this enhancement can be added to 1.17 tracking sheet.

Please note that the KEP is missing test plan.

Thank you!

🔔Friendly Reminder

• The current release schedule is
  
  
  
  • Monday, September 23 - Release Cycle Begins
  
  
  • Tuesday, October 15, EOD PST - Enhancements Freeze
  
  
  • Thursday, November 14, EOD PST - Code Freeze
  
  
  • Tuesday, November 19 - Docs must be completed and reviewed
  
  
  • Monday, December 9 - Kubernetes 1.17.0 Released
  
  

• A Kubernetes Enhancement Proposal (KEP) must meet the following criteria before Enhancement Freeze to be accepted into the release

  • PR is merged in
  • In an implementable state
  • Include test plan and graduation criteria

• All relevant k/k PRs should be listed in this issue

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[palnabarun]

Hey there @derekwaynecarr @RobertKrawitz -- 1.18 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.18 or having a major change in its current level?

The current release schedule is:

• Monday, January 6th - Release Cycle Begins
• Tuesday, January 28th EOD PST - Enhancements Freeze
• Thursday, March 5th, EOD PST - Code Freeze
• Monday, March 16th - Docs must be completed and reviewed
• Tuesday, March 24th - Kubernetes 1.18.0 Released

To be included in the release,

1. The KEP PR must be merged
2. The KEP must be in an implementable state
3. The KEP must have test plans and graduation criteria.

If you would like to include this enhancement, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

We'll be tracking enhancements here: http://bit.ly/k8s-1-18-enhancements

Thanks! :)

[palnabarun]

@derekwaynecarr @RobertKrawitz Just a friendly reminder, we are just 7 days away from the Enhancement Freeze (Tuesday, January 28th).

[palnabarun]

@derekwaynecarr @RobertKrawitz Just a friendly reminder, we are just 2 days away from the Enhancement Freeze (3 PM Pacific Time, Tuesday, January 28th).

[palnabarun]

Unfortunately, the deadline for the 1.18 Enhancement freeze has passed. For now, this is being removed from the milestone. If there is a need to get this in, please file an enhancement exception.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[kikisdeliveryservice]

Hi @derekwaynecarr @RobertKrawitz

1.19 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating in 1.19?

In order to have this part of the release:

The KEP PR must be merged in an implementable state
The KEP must have test plans
The KEP must have graduation criteria.

The current release schedule is:

Monday, April 13: Week 1 - Release cycle begins
Tuesday, May 19: Week 6 - Enhancements Freeze
Thursday, June 25: Week 11 - Code Freeze
Thursday, July 9: Week 14 - Docs must be completed and reviewed
Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released

Please let me know and I'll add it to the 1.19 tracking sheet (http://bit.ly/k8s-1-19-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[kikisdeliveryservice]

As a reminder, enhancements freeze is tomorrow May 19th EOD PST. In order to be included in 1.19 all KEPS must be implementable with graduation criteria and a test plan.

Thanks.

[kikisdeliveryservice]

Unfortunately the deadline for the 1.19 Enhancement freeze has passed. For now this is being removed from the milestone and 1.19 tracking sheet. If there is a need to get this in, please file an enhancement exception.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[kikisdeliveryservice]

Hi @derekwaynecarr @RobertKrawitz

Enhancements Lead here. Are there any plans for this 1.20?

Thanks!
Kirsten

[sjenning]

This is going GA for 1.20 https://github.com/kubernetes/kubernetes/pull/94140

[kikisdeliveryservice]

/milestone v1.20

[sftim]

As per https://github.com/kubernetes/enhancements/issues/757#ref-issue-701936366 I also recommend documenting the feature.

[kikisdeliveryservice]

Documentation PR already open: https://github.com/kubernetes/website/pull/23929

[MorrisLaw]

Hey @derekwaynecarr @RobertKrawitz - 1.20 Enhancements Shadow here 👋

Just a friendly reminder that the Enhancements Freeze deadline is October 6th.

[MorrisLaw]

Hi @derekwaynecarr,

Since your Enhancement is scheduled to be in 1.20, please keep in mind the important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Regards,
Jeremy

[kikisdeliveryservice]

Hey @derekwaynecarr @sjenning

Is https://github.com/kubernetes/kubernetes/pull/94140 the only k/k pr required to move this to GA? Is all work done? Code freeze is tomorrow.

Thanks
Kirsten

[kikisdeliveryservice]

Confirmed with both Derek and Seth that this is completed. :+1:

[hh]

Just a quick check that in the process of promoting this feature to GA, was there any change to the API surface area?

Particularly swagger.json?

We have a process for detecting new GA API operations, but it currently looks primarily at the generated OpenAPI spec.

[hh]

it tests only GA, non-optional features or APIs (e.g., no alpha or beta endpoints, no feature flags required, no deprecated features)

^ From: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#conformance-test-requirements

@dims Can I get your check-in on this? It doesn't seem to touch the OpenAPI spec, but if looks non-optional:

https://github.com/kubernetes/kubernetes/pull/94140/files#diff-71e3b98f9a6bbf5b8421e26a7ba0c079f397cd8d49abacdad943c66a4f44f03dR685-R686

+ SupportPodPidsLimit: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.21
+ SupportNodePidsLimit:{Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.21

[derekwaynecarr]

There is no end user api change in the promotion. The feature has been on by default via kubelet Config for multiple releases.

[hh]

Thanks @derekwaynecarr !

[SergeyKanzhelev]

@derekwaynecarr I wonder if the pid limit test should have been promoted to conformance as part of the GA process? It seems to be satisfying all requirements. I understand now there is a validation for the APIs to be conformance tested. I also see conformance tests for things like config maps. I wonder whether graduation criteria in general should include promoting a test to conformance whenever possible.

On KEP I see the graduation criteria that doesn't say anything about conformance tests:

GA

• assuming no negative user feedback, promote after 1 release at beta.

The only note on desire to increase coverage of conformance tests I found here:

NB: This should be viewed as a living document in a few key areas:

The desired set of conformant behaviors is not adequately expressed by the current set of e2e tests, as such this document is currently intended to guide us in the addition of new e2e tests than can fill this gap