Enhancements: EndpointSlice API

[freehan]

Enhancement Description

• One-line enhancement description (can be used as a release note): New Endpoint API as eventual replacement of the current core/v1 Endpoints API.
• Primary contact (assignee): @freehan
• Responsible SIGs: sig-network, sig-scalability
• Kubernetes Enhancement Proposal PR: #1086
• Kubernetes Enhancement Proposal:https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/0752-endpointslices
• Link to e2e and/or unit tests: TBD
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @thockin @wojtek-t
• Approver (likely from SIG/area to which enhancement belongs): @thockin @wojtek-t
• Enhancement target (which target equals to which milestone):
  
  
  
  • Alpha release target (1.15)
  
  
  • Beta release target (1.16)
  
  
  • Stable release target (1.17)
  
  

[liggitt]

was this intended to be targeted at 1.14 or 1.15?

[liggitt]

moving to 1.15, since there's no design in progress yet
/milestone v1.15

[k8s-ci-robot]

@liggitt: The provided milestone is not valid for this repository. Milestones in this repository: [keps-beta, keps-ga, v1.13, v1.14]

Use /milestone clear to clear the milestone.

In response to this:

moving to 1.15, since there's no design in progress yet
/milestone v1.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wojtek-t]

We would like to have the KEP approved in 1.14 timeframe, so that we would be able to reach alpha in 1.15.

[liggitt]

sounds good; that means the actual changes are targeting 1.15, so I think the 1.15 milestone is accurate

[wojtek-t]

Agree.

[kacole2]

Hello @wojtek-t @freehan, I'm the Enhancement Lead for 1.15. Is this feature going to be targeting alpha in 1.15? I do not see a KEP which is required. Please let me know so it can be tracked properly and added to the spreadsheet.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[wojtek-t]

Yes - we're targetting 1.15 for Alpha release. Will keep it issue updated.

[mrbobbytables]

@freehan, @wojtek-t Kubernetes 1.15 Enhancement Freeze is 4/30/2019. To be included in the Kubernetes 1.15 milestone, KEPs are required to be merged and in an "Implementable" state with proper test plans and graduation criteria. Please submit any PRs needed to make this KEP adhere to inclusion criteria. If this will slip from the 1.15 milestone, please let us know so we can make appropriate tracking changes.

[freehan]

The proposal is already out. It has been shared with the community and still in discussion. KEP is not merged yet.

https://docs.google.com/document/d/1sLJfolOeEVzK5oOviRmtHOHmke8qtteljQPaDUEukxY/edit#heading=h.3v0n1vmx9wtr

[mrbobbytables]

@freehan @wojtek-t The Enhancement Freeze for Kubernetes 1.15 has passed and this did not meet the deadline. This is now being removed from the 1.15 milestone and the tracking sheet. If there is a need for this to be in 1.15, please file an Enhancement Exception. Thank you.

[kacole2]

Hi @freehan @wojtek-t , I'm the 1.16 Enhancement Lead. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[freehan]

@kacole2 I do not have permission to add it into spreadsheet. Would you mind adding me to it?

KEP in flight: https://github.com/kubernetes/enhancements/pull/1086

[freehan]

KEP merged. Will start pushing PRs.

[daminisatya]

@freehan @wojtek-t

I'm one of the v1.16 docs shadows.
Does this enhancement (or the work planned for v1.16) require any new docs (or modifications to existing docs)? If not, can you please update the 1.16 Enhancement Tracker Sheet (or let me know and I’ll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday, August 23rd, it can just be a placeholder PR at this time. Let me know if you have any questions!

[freehan]

Yes. We will try to catch the deadline to add docs.

[simplytunde]

@freehan This Friday is the Docs placeholder PR deadline. Please try to get one in against k/website dev-1.16 branch.

[robscott]

@simplytunde Thanks for the reminder, we're aiming to have a WIP PR out today.

[simplytunde]

ok thanks @robscott

[kacole2]

@robscott @freehan I see https://github.com/kubernetes/kubernetes/pull/81430, https://github.com/kubernetes/kubernetes/pull/81795, https://github.com/kubernetes/kubernetes/pull/81048, https://github.com/kubernetes/kubernetes/pull/80766 are still open. code freeze for 1.16 is on Thursday 8/29. Are there any more outstanding k/k PRs that still need to be merged for this to go Alpha?

[robscott]

Hey @kacole2, that's the extent of the k/k PRs needed for this. Hoping to get them merged in soon, just waiting for a final round of reviews on the biggest ones. I think we're still on pace to get everything in before code freeze.

[kacole2]

Hi @robscott it looks as though https://github.com/kubernetes/kubernetes/pull/81795 didn't merge before code freeze and it's not in the Tide Merge Pool. This feature is going to be bumped from v1.16. If you would still like to have this be a part of the 1.16 release, please file an exception

[freehan]

@kacole2 The main PRs implementing the feature have been merged. IIUC, as this is alpha, kubectl support (https://github.com/kubernetes/kubernetes/pull/81795) is not required.

Does it still require an exception?

[lachie83]

I've had a chat to @robscott and we have agreed to add this to the milestone and get it merged into 1.16 as there was a little confusion regarding labeling yesterday. @freehan no need for an exception

[mrbobbytables]

Hey there @freehan @robscott -- 1.17 Enhancements lead here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

/milestone clear

[robscott]

Hey @mrbobbytables, thanks for checking in! We're aiming for beta for Endpoint Slices in 1.17. We'll reference this in any PRs related to that - at this point it looks like any changes will be primarily performance related.

[mrbobbytables]

Great, thanks for the quick response @robscott :) I'll add it to the tracking sheet 👍
/milestone v1.17
/stage beta

[VineethReddy02]

@robscott @freehan

I'm one of the v1.17 docs shadows.
Does this enhancement (or the work planned for v1.17) require any new docs (or modifications to existing docs)? If not, can you please update the 1.17 Enhancement Tracker Sheet (or let me know and I’ll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.17) due by Friday, Nov 8th, it can just be a placeholder PR at this time. Let me know if you have any questions!

Thanks!

[robscott]

Hey @VineethReddy02, thanks for checking in! This will require some changes to the docs, including:

• Removing the documentation covering enabling EndpointSlices since they'll be enabled by default.
• Updating the other documentation to indicate that EndpointSlices will be enabled by default.
• Adding documentation to cover the new label, address type, and appProtocol field that will be added as part of the 1.17 release.

Unfortunately I don't have access to modify that tracker sheet, so if you're able to that'd be great.

[VineethReddy02]

@robscott
I will update the 1.17 release tracker sheet. Can you please create a placeholder PR against k8s/website(branch dev-1.17) where all the required docs work will be carried out and update me on that.

Thanks for the update!

[VineethReddy02]

Hey, @robscott Just a friendly reminder we're looking for a PR against k/website (branch dev-1.17) due by Friday, Nov 8th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!
We are 5 days away from the deadline. Thanks!

[robscott]

@VineethReddy02 thanks for the reminder! Just added a docs PR.

[mrbobbytables]

Hey there @robscott 1.17 Enhancement lead here. 👋 Code Freeze is right around the corner (November 14th).

It looks like there are a lot of outstanding PRs 🙈 Do you think they'll be merged in time? If they don't happen to make it in, the Enhancement will be removed from the milestone. 😬

Only release-blocking issues and PRs will be allowed in the milestone after the freeze.

Thanks!

[robscott]

Hey @mrbobbytables, thanks for checking in! I think we're still on pace, but definitely have been working to get reviews in. I think I've been referencing this issue on more PRs than I need to, since I've generally been including a reference to this on any PR related to EndpointSlices. The only PRs that need to make it into beta according to our KEP are being tracked on the main beta PR: https://github.com/kubernetes/kubernetes/pull/84390. One of the PRs listed there has been approved and not yet merged, which would bring us to 5/8 merged, and 2 of the remaining 3 are in final rounds of review.

[mrbobbytables]

Ahhh, cool -- thanks for letting me know. It's easy to get lost in what actually needs to be tracked. 👍

[mrbobbytables]

Hey there @robscott tomorrow is code freeze for the 1.17 release cycle. It looks like two (https://github.com/kubernetes/kubernetes/pull/84089 and https://github.com/kubernetes/kubernetes/pull/84208) of the k/k PRs have not yet been merged. 😬 We're flagging this as At Risk in the 1.17 Enhancement Tracking Sheet.
Do you think they will be merged by the EoD of the 14th (Thursday)? After that point, only release-blocking issues and PRs will be allowed in the milestone with an exception.

[robscott]

Hey @mrbobbytables, definitely working hard to make that a reality. https://github.com/kubernetes/kubernetes/pull/84208 is approved and hopefully should be merged shortly. My goal is still to have the beta PR ready to go in time for code freeze :crossed_fingers: .

[robscott]

/retitle EndpointSlice API

[johnbelamaric]

@robscott I am shadowing for release team enhancements. Any plan to graduate in 1.18?

[robscott]

Hey @johnbelamaric, no plans to graduate beyond beta in 1.18, but we are hoping to enable the controller by default in 1.18 (that was not the case in 1.17 despite the beta status). Here's the relevant PR: https://github.com/kubernetes/kubernetes/pull/86137.

[robscott]

/assign

[johnbelamaric]

Thanks Rob

[johnbelamaric]

/milestone clear

[johnbelamaric]

/milestone v1.18

@robscott we will track the change for default at the release team level. thanks for linking the PR.

[irvifa]

Hello @freehan @robscott I'm one of the v1.18 docs shadows.
Does this enhancement for (or the work planned for v1.18) require any new docs (or modifications to existing docs)? If not, can you please update the 1.18 Enhancement Tracker Sheet (or let me know and I'll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.18) due by Friday, Feb 28th., it can just be a placeholder PR at this time. Let me know if you have any questions!

[robscott]

Hey @irvifa there are some small changes related to EndpointSlices and Service Topology that will need docs updates. I'll plan to have a draft ready in time.

[johnbelamaric]

@robscott Is https://github.com/kubernetes/kubernetes/pull/86137 the only PR that was needed, other than docs?

[robscott]

There will be one more PR, but it will already be tracked on the Service Topology enhancement, not sure if it's needed here as well.

[irvifa]

Hi @robscott 👋 Thank you for your response, please let me know if you already create a draft PR so I can change the tracking sheet.

[irvifa]

Hello @robscott I'm one of the v1.18 docs shadows.

Does this enhancement for (or the work planned for v1.18) require any new docs (or modifications to existing docs)? If not, can you please update the 1.18 Enhancement Tracker Sheet (or let me know and I'll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.18) due by Friday, Feb 28th, which is roughly 1 weeks from now. It can just be a placeholder PR at this time. Let me know if you have any questions!

[robscott]

Hey @irvifa, I will need to update the docs for this. I'll work on getting a placeholder PR in shortly. The actual implementation work for EndpointSlices in 1.18 is complete now.

[robscott]

@irvifa I have a docs PR ready for review now: https://github.com/kubernetes/website/pull/19316

[johnbelamaric]

@robscott Since the service topo piece is pushed, are we code complete on this now?

[robscott]

@johnbelamaric yep, code complete, docs waiting for review.

[irvifa]

Hi @robscott thanks for your update..

[palnabarun]

/milestone clear

(removing this enhancement issue from the v1.18 milestone as the milestone is complete)

[robscott]

@palnabarun Would love to have this as part of the 1.19 milestone as well. I've outlined the changes I'd like to include in 1.19 here: https://github.com/kubernetes/enhancements/pull/1713

[johnbelamaric]

Thanks @robscott I will add this to the tracking sheet.

[palnabarun]

/milestone v1.19

[divya-mohan0209]

Hello @robscott , @freehan , I'm one of the v1.19 docs shadows.

Could you please let me know if this piece of enhancement work planned for 1.19 release requires any new docs (or modifications to existing docs)?

If not, can you please update the 1.19 Enhancement Tracker Sheet (or let me know and I'll do so)

If yes, just a friendly reminder we're looking for a PR against k/website (branch dev-1.19) due by Friday, June 12. It can just be a placeholder PR at this time.

Please do let me know if you have any questions!

Regards,
Divya

[robscott]

Hey @divya-mohan0209, thanks for checking in! This feature will need some docs updates. I'll make sure we get a placeholder PR in place by June 12.

[wojtek-t]

@freehan @robscott - can you please

1. transform the KEP to the new format (should be simple, just extracting metadata to kep.yaml: see template: https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template )
2. fill in PRR questionaire and assign me as PRR approver (I'm taking all network PRRs this cycle)

[divya-mohan0209]

Hello @robscott , a gentle reminder on the above PR. Please could you let me know if you were able to raise a draft PR for the documentation bit required in this release?

[robscott]

Hey @divya-mohan0209, thanks for the reminder! I've created a docs PR for 1.19 now here: https://github.com/kubernetes/website/pull/21589.

[kikisdeliveryservice]

Hi @robscott !

To follow-up on the email sent to k-dev on Monday, I wanted to let you know that Code Freeze has been extended to Thursday, July 9th. You can see the revised schedule here: https://github.com/kubernetes/sig-release/tree/master/releases/release-1.19
We expect all PRs to be merged by that time. Please let me know if you have any questions. 😄

Best,

Kirsten

[palnabarun]

Hi @robscott :wave: -- 1.19 Enhancements Lead here,

Is https://github.com/kubernetes/kubernetes/pull/91637 the only implementation PR for this enhancement? If it is otherwise, can you please link all the implementation PRs here?

Thank you. :slightly_smiling_face:

[robscott]

Hey @palnabarun, there's also this PR for a Windows kube-proxy implementation: https://github.com/kubernetes/kubernetes/issues/90315. Once they both merge we'll need a tiny PR to flip a feature gate from alpha to beta.

[palnabarun]

@robscott -- Thank you for the update. :slightly_smiling_face:

Please link the feature flip PR whenever ready. :+1:

[palnabarun]

Hi @robscott :wave:, I see that https://github.com/kubernetes/kubernetes/pull/90909 (closing https://github.com/kubernetes/kubernetes/issues/90315) is already merged but https://github.com/kubernetes/kubernetes/pull/91637 has a failing test. Do you think that the PR will be merged before the Code Freeze on Thursday, July 9th?

Please note that the feature gate PR (https://github.com/kubernetes/kubernetes/pull/92736) also needs to go in before the freeze.

Thank you. :slightly_smiling_face:

---

Code Freeze begins on Thursday, July 9th EOD PST

[robscott]

Hey @palnabarun, kubernetes/kubernetes#91637 is in the merge queue now and I'm hoping to get approval for the feature gate PR today or tomorrow: kubernetes/kubernetes#92736. I think we should be on track still.

[johnbelamaric]

Looks like both are now in the queue.

[kikisdeliveryservice]

Congrats @robscott looks like the PRs finally merged! I will update the tracking sheet accordingly.

[robscott]

Thanks to everyone for the help!

[kikisdeliveryservice]

/milestone clear

(removing this enhancement issue from the v1.19 milestone as the milestone is complete)

[kikisdeliveryservice]

Hi @robscott

Enhancements Lead here. Are there any plans to graduate this to stable in 1.20?

Thanks!
Kirsten

[kikisdeliveryservice]

Just following up: any plans to graduate this to stable in 1.20? Enhancements Freeze is October 6th.

Thanks,
Kirsten

[robscott]

Thanks for following up @kikisdeliveryservice! We're hoping to graduate some components to GA in 1.20, I don't think there will be any significant changes beyond that. I'll work on updating the KEP soon.

[kikisdeliveryservice]

Hi @robscott just to confirm, we don't need to track this in 1.20? (sorry for my confusion)

[robscott]

Sorry for the confusion @kikisdeliveryservice! There are no significant changes to the feature but I think we'll need to track it since we're planning to graduate it to GA.

[wojtek-t]

The answer is "yes - we should track it".

@robscott - It doesn't matter how many changes we're doing underneath. What matters is that we are graduating the API (and the endpointslice controller to GA). So that's actually a big one (no matter if there will be changes underneath or not).

[kikisdeliveryservice]

Thanks @wojtek-t @robscott

Can we then update the graduation criteria to be clear about what is reqd for 1.20?
https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/20190603-endpointslices#graduation-criteria

Note (just ftr (:smile: ) this needs to be done by Enhancements Freeze Oct 6th (also if you could update to new format that would be great)

[robscott]

Will do @kikisdeliveryservice, thanks for the reminder! I think it's already updated to use the new format, unless there's an even newer update I missed :)

[kikisdeliveryservice]

@robscott So close!!! The dir name and KEP name should be 752-endpointslices :smile:

[robscott]

Woops, thanks @kikisdeliveryservice! I've created a PR to update that and include GA graduation: https://github.com/kubernetes/enhancements/pull/2032

[kikisdeliveryservice]

Nice! The PR looks good. It will just need to merge by next Tuesday, October 6th

Almost there!! :muscle:

[robscott]

@kikisdeliveryservice #2032 has merged, I think we're good on this one for the KEP deadline

[kikisdeliveryservice]

Hi @robscott

Just dropping a note about upcoming impt dates:

Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thanks!
Kirsten

[kcmartin]

Hello @robscott , 1.20 Docs shadow here.

Does this enhancement work planned for 1.20 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.20 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Nov 6th

Also take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[kcmartin]

Hi @robscott

The docs placeholder deadline is almost here. Please make sure to create a placeholder PR against the dev-1.20 branch in the k/website before the deadline

Also, please keep in mind the important upcoming dates:

Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze
Monday, Nov 23rd: Week 11 - Docs PR Ready for Review

As a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

[robscott]

Hey @kcmartin, I've created https://github.com/kubernetes/website/pull/24935 for docs here.

[kikisdeliveryservice]

Hey @robscott

I see that kubernetes/kubernetes#96318 is still open but being actively worked on. Just a reminder that Code Freeze is coming up in 2 days on Thursday, November 12th. All PRs must be merged by that date, otherwise an Exception is required.

Best,
Kirsten

[kikisdeliveryservice]

@robscott just to confirm is kubernetes/kubernetes#96440 the required PR?

[robscott]

@kikisdeliveryservice Yep, sorry for the confusion. Got caught by the bug that's breaking test runs for some reason.

[kikisdeliveryservice]

Looks like the PR finally got past the test-infra bug and is fully approved. Seems to have failed the e2e-gce-ubuntu-containerd tho: https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/96440/pull-kubernetes-e2e-gce-ubuntu-containerd/1326986000102068224/

Not sure if that needs a retest or something else.

[kikisdeliveryservice]

Reading: https://github.com/kubernetes/kubernetes/pull/96318#issuecomment-725008391

It looks like this won't be GA in 1.20, so it will stay beta and the required PR for 1.20 has merged: https://github.com/kubernetes/kubernetes/pull/96440

[kikisdeliveryservice]

@robscott could you update the kep.yaml to reflect this: https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/0752-endpointslices/kep.yaml

Also does this change your docs PR at all?

[robscott]

Yep, I'll work on updating the KEP to reflect that tomorrow. The docs PR will need some work but it was already fairly small.

[robscott]

Update to the KEP has merged: https://github.com/kubernetes/enhancements/pull/2152