Enhancements: SCTP support for Services, Pod, Endpoint, and NetworkPolicy

[janosi]

Feature Description

• One-line feature description (can be used as a release note): SCTP is now supported as additional protocol (alpha) alongside TCP and UDP in Pod, Service, Endpoint, and NetworkPolicy.
• Primary contact (assignee): @janosi
• Responsible SIGs: sig-network
• Design proposal link (community repo): SCTP support, https://github.com/kubernetes/community/pull/2276
• KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/0015-20180614-SCTP-support.md
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs): @thockin
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target (x.y) 1.12
  
  
  • Beta release target (x.y)
  
  
  • Stable release target (x.y)
  
  

[zparnold]

/milestone v1.12

[justaugustus]

/assign @janosi
/sig network
/kind feature

[justaugustus]

/stage alpha

[ameukam]

Hi folks,
Kubernetes 1.13 is going to be a 'stable' release since the cycle is only 10 weeks. We encourage no big alpha features and only consider adding this feature if you have a high level of confidence it will make code slush by 11/09. Are there plans for this enhancement to graduate to alpha/beta/stable within the 1.13 release cycle? If not, can you please remove it from the 1.12 milestone or add it to 1.13?

We are also now encouraging that every new enhancement aligns with a KEP. If a KEP has been created, please link to it in the original post. Please take the opportunity to develop a KEP.

[janosi]

Hello @ameukam,

I am not sure I understand :) The feature's implementation has been merged into 1.12 with the PR @justaugustus referenced above, and it is indeed in that release as alpha feature, including the documentation
PR: https://github.com/kubernetes/kubernetes/pull/64973
KEP: https://github.com/kubernetes/community/pull/2276
Doc: https://github.com/kubernetes/website/pull/10279

Thanks!

[ameukam]

Hi @janosi, apologies for your confusion. The idea is to identify what is the target of this enhancement
for the next milestone ? Do you want to keep as alpha for v1.13 ? or graduate it to beta ?
As i said earlier, the cycle for 1.13 will be only 10 weeks. So it's up to you to decide if it can make it to the Code Slush as beta feature.

[guineveresaenger]

@janosi if you are saying that #64973 has fixed this issue, should this issue be closed?

[justaugustus]

Feature issues remain open through to GA stage.

[janosi]

@ameukam Thank you for the clarification! I am fine to keep it as alpha in 1.13 - thought it would be great to understand how much time/releases I have to mature to beta/GA before my feature is removed. Thank you!

[kacole2]

/milestone clear

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ameukam]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[kacole2]

I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[subrnath]

SCTP support still seems to be in alpha in release 1.15 as per https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/. What is the road map for the SCTP support to move to GA?

[danwinship]

/reopen
/remove-lifecycle rotten

The actual implementation is done, but we need to add tests and confirm some implementations before moving to beta/GA.

[k8s-ci-robot]

@danwinship: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten

The actual implementation is done, but we need to add tests and confirm some implementations before moving to beta/GA.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[danwinship]

filed https://github.com/kubernetes/enhancements/pull/1250

[annajung]

Hello @danwinship @janosi, 1.17 Enhancement Shadow here! 🙂

I wanted to reach out to see *if this enhancement will be graduating to alpha/beta/stable in 1.17?

*
Please let me know so that this enhancement can be added to 1.17 tracking sheet.

Please note that open PR #1250 does not include test plan which is required to be accepted to the release

Thank you!

🔔Friendly Reminder

• The current release schedule is
  
  
  
  • Monday, September 23 - Release Cycle Begins
  
  
  • Tuesday, October 15, EOD PST - Enhancements Freeze
  
  
  • Thursday, November 14, EOD PST - Code Freeze
  
  
  • Tuesday, November 19 - Docs must be completed and reviewed
  
  
  • Monday, December 9 - Kubernetes 1.17.0 Released
  
  

• A Kubernetes Enhancement Proposal (KEP) must meet the following criteria before Enhancement Freeze to be accepted into the release

  • PR is merged in
  • In an implementable state
  • Include test plan and graduation criteria

• All relevant k/k PRs should be listed in this issue

[danwinship]

@annajung Yes, the plan is to graduate to Beta for 1.17. I will update #1250; the additions there are basically all Test Plan, but I put the test plan in the Graduation Criteria section because there wasn't a "Test Plan" section in its outline because it was started from an older version of the template

[annajung]

@danwinship Thanks for the clarification! I will add this enhancement to the tracking sheet to be tracked 👍

[annajung]

Hey @danwinship Could you please post links to the tests in testgrid and keep track of any tests added for this enhancement?

Thank you!

[annajung]

Hey @danwinship , unfortunately deadline for 1.17 enhancement freeze has passed and looks like the KEP w/ test plan is still open. I will be removing this enhancement from the 1.17 milestone.

Please note that you can file an enhancement exception if you need to get this in for 1.17

/milestone clear

[palnabarun]

Hey there @danwinship @janosi -- 1.18 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to beta in 1.18 or having a major change in its current level?

The current release schedule is:

• Monday, January 6th - Release Cycle Begins
• Tuesday, January 28th EOD PST - Enhancements Freeze
• Thursday, March 5th, EOD PST - Code Freeze
• Monday, March 16th - Docs must be completed and reviewed
• Tuesday, March 24th - Kubernetes 1.18.0 Released

If you would like to include this enhancement, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

We'll be tracking enhancements here: http://bit.ly/k8s-1-18-enhancements

Thanks! :)

[palnabarun]

@danwinship @janosi Just a friendly reminder, we are just 7 days away from the Enhancement Freeze (Tuesday, January 28th).

[janosi]

@palnabarun @danwinship I wonder if there is anything to do with the KEP itself. Do you see any gap?

[palnabarun]

@janosi The KEP satisfies all criteria for inclusion into the release cycle. I don't see any gap with respect to the KEP.

However, the person(s)/SIG(s) responsible for the enhancements has to give a nod if there is any work planned for graduation in the current release cycle.

[palnabarun]

Also, we are just 2 days away from the Enhancement Freeze (3 PM Pacific Time, Tuesday, January 28th).

[palnabarun]

The KEP linked in the PR was linking to a non-existent document, fixed it.

[janosi]

@palnabarun @danwinship Yes, I plan to work on the e2e tests in 1.18 timeframe. Hopefully I can create those in 1.18, so this one can graduate to beta.

[palnabarun]

@janosi Awesome! I am going to set this to tracked in the tracking sheet.

[palnabarun]

/milestone v1.18

[palnabarun]

Hi @janosi, just a friendly reminder that the Code Freeze will go into effect on Thursday 5th March.

Can you please link all the k/k PRs or any other PRs which should be tracked for this enhancement?

Thank You :)

[irvifa]

Hello @danwinship @janosi I'm one of the v1.18 docs shadows.
Does this enhancement for (or the work planned for v1.18) require any new docs (or modifications to existing docs)? If not, can you please update the 1.18 Enhancement Tracker Sheet (or let me know and I'll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.18) due by Friday, Feb 28th., it can just be a placeholder PR at this time. Let me know if you have any questions!

[danwinship]

@irvifa nothing about the feature is changing; the only difference is that it will be Beta instead of Alpha.

Hm... I guess it's normal to switch feature gates from default-off to default-on when going to Beta but we haven't talked about doing that at all...

[irvifa]

Hi 👋🏻 @danwinship thank you for your response. Any way will this explanation need to be mentioned explicitly in the docs? If it is then please open a placeholder PR..

[janosi]

@palnabarun this is the PR to cover the basic e2e test cases. It is WIP currently https://github.com/kubernetes/kubernetes/pull/88196

[irvifa]

Hi 👋🏻 @danwinship just a friendly reminder, do you already decided on the document's format? Since the deadline is roughly 1 week from now. Thanks!

[danwinship]

This should be handled exactly like any other feature going from alpha to beta. I have no idea what the docs team does for that.

[palnabarun]

@janosi Thanks for the link to the e2e test PR. Do you have any more PR's that we should track?

[palnabarun]

FYI, we are really close to Code Freeze (5th March).

[danwinship]

e2e tests are the only thing being added in this cycle; the actual implementation of the feature is already complete

[VineethReddy02]

@danwinship Can you please raise a placeholder PR. I assume moving a feature from alpha to beta needs some docs changes at least changing the user-facing docs from alpha to beta. Doc's placeholder PR deadline is tomorrow. Please raise a placeholder PR in k8s/website repo against the dev-1.18 branch.

Thanks! :)

[jeremyrickard]

/milestone clear

[VineethReddy02]

Hi, @danwinship as there is no docs placeholder PR for this enhancement and didn't receive any update on the docs front. we have crossed the docs placeholder PR deadline. So we are removing this enhancement from the 1.18 release. If you want to request an exception. Please refer to https://github.com/kubernetes/sig-release/blob/master/releases/EXCEPTIONS.md

Thanks!

[janosi]

@VineethReddy02 @danwinship
Doc PR to move SCTP to beta: https://github.com/kubernetes/website/pull/19440

[janosi]

@VineethReddy02 @danwinship I have just sent the exception request to the necessary mail lists. Unfortunately, I missed to add a subject :(

[palnabarun]

/milestone v1.18

[janosi]

test-infra PR to implement a new periodic job for the basic SCTP e2e test cases https://github.com/kubernetes/test-infra/pull/16595

[palnabarun]

Hi @janosi @danwinship,

Today EOD is Code Freeze.

Do you think the PR's would get reviewed by the deadline?

If not, please file an exception request.

[janosi]

Hello @palnabarun

On this PR I wait for the first review from the assigned reviewers @jingax10 @bowei
https://github.com/kubernetes/test-infra/pull/16595

On this PR I fixed all review comments I got so far from @aojea and @danwinship :
https://github.com/kubernetes/kubernetes/pull/88196

I do not know whether it is realistic or not to merge these today.

[kikisdeliveryservice]

Hi @janosi

As both PRs are not fully reviewed/approved and still needs-ok-to-test/ci runs, removing from the milestone. Feel free to file an exception request, if you feel this should make it in and please let us know if you plan on re-targetting this for 1.19.

Best,
The Enhancements Team.

/milestone clear

[janosi]

PR to set the feature flag to true by default once all other PRs are merged: https://github.com/kubernetes/kubernetes/pull/88932

[palnabarun]

Hi @janosi -- 1.19 Enhancements Lead here, I wanted to check in if you think this enhancement would graduate in 1.19?

---

The current release schedule is:

• Monday, April 13: Week 1 - Release cycle begins
• Tuesday, May 19: Week 6 - Enhancements Freeze
• Thursday, June 25: Week 11 - Code Freeze
• Thursday, July 9: Week 14 - Docs must be completed and reviewed
• Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released

[janosi]

Hello @palnabarun I do not know. One of my PRs would need some attention
https://github.com/kubernetes/kubernetes/pull/90365

Once that one is merged the other one could follow.
Then we would have some parts of the e2e tests that are required to become beta. Though it was so, that first the test cases should run before this feature could become beta. The original thought was, that the test cases would be merged at the beginning of 1.19, so those could run enough times to have some data to support the moving to beta. But the tests were not merged at the beginning of 1.19, so I do no know anymore, Thank you for asking!

[palnabarun]

@janosi -- Unfortunately, the deadline for the 1.19 Enhancement freeze has passed. For now, this is being removed from the milestone and 1.19 tracking sheet. If there is a need to get this in, please file an enhancement exception.

[palnabarun]

Hi @janosi @danwinship, your exception request was approved. :slightly_smiling_face:

I have updated the tracking sheet accordingly. Also, please link here any implementation PRs (k/k or otherwise) that you have.

/milestone v1.19

[janosi]

Hello @palnabarun, great news! Thank you! :)
There are no real implementation PRs for this.
We have a "change the feature flag to true" PR https://github.com/kubernetes/kubernetes/pull/88932/
And there should be a new "update the docs PR" as the previous one was closed when we could not make it for 1.18.

[savitharaghunathan]

Hi @janosi - I am Savitha, 1.19 Docs lead. Does this enhancement work planned for 1.19 require any new docs (or modifications to existing docs)? If not, can you please update the 1.19 Enhancement Tracker Sheet, or let me know, I can do it for you :)
If docs are required, just a friendly reminder we're looking for a PR against k/website (branch dev-1.19) due by Friday, June 12, it can just be a placeholder PR at this time. Let me know if you have any questions!

[janosi]

Hi @savitharaghunathan - It is nice to meet you! Yes, doc update is required, the relevant PR is https://github.com/kubernetes/website/pull/21672
Could you please help in updating the 1.19 Enhancement Tracker Sheet? That would be awesome. Thank you!

[savitharaghunathan]

@janosi Thank you for the PR, I have updated the enhancement tracker sheet accordingly.

[palnabarun]

Hi @janosi -- just wanted to check in about the progress of the enhancement.

I saw that https://github.com/kubernetes/kubernetes/pull/88932 has been merged :tada: . Do you have any other PRs for the graduation of this enhancement? Or, is the graduation criteria is complete for this cycle?

The release timeline has been revised recently, more details of which can be found here.

Please let me know if you have any questions. :slightly_smiling_face:

---

The revised release schedule is:

• Thursday, July 9th: Week 13 - Code Freeze
• Thursday, July 16th: Week 14 - Docs must be completed and reviewed
• Tuesday, August 25th: Week 20 - Kubernetes v1.19.0 released

[janosi]

Hello @palnabarun. There is the doc PR only waiting for further possible comments or merge, nothing else. Thank you!

[palnabarun]

Hi @janosi :wave:, thank you for the update. :slightly_smiling_face:

[kikisdeliveryservice]

Hi @janosi

Enhancements Lead here. Are there any plans to graduate this to stable in 1.20?

Thanks!
Kirsten

[danwinship]

Yes; all the e2e tests have merged now, so per the KEP we just need to get some passing results from network plugins and then this can move to GA

[kikisdeliveryservice]

Great thanks for the update!

/milestone v1.20

[antoninbas]

@danwinship here are the sonobuoy results for the Antrea plugin when running [Feature:SCTPConnectivity] e2e tests:

https://downloads.antrea.io/tmp/sonobuoy-results/202009252249_sonobuoy_7979918d-3e1b-4fb4-80a1-c56933312b88.tar.gz

with some extra details on how I ran the tests:

> kubectl version
Client Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.6-beta.0", GitCommit:"e7f962ba86f4ce7033828210ca3556393c377bcc", GitTreeState:"clean", BuildDate:"2020-01-15T08:26:26Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

> ./sonobuoy version
Sonobuoy Version: v0.19.0
MinimumKubeVersion: 1.17.0
MaximumKubeVersion: 1.19.99
GitSHA: e03f9ee353717ccc5f58c902633553e34b2fe46a

> kubectl apply -f https://github.com/vmware-tanzu/antrea/releases/download/v0.10.0/antrea.yml

> kubectl get nodes -o wide
NAME                STATUS   ROLES    AGE     VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-node-master     Ready    master   13m     v1.19.2   192.168.77.100   <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.13
k8s-node-worker-1   Ready    <none>   10m     v1.19.2   192.168.77.101   <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.13
k8s-node-worker-2   Ready    <none>   7m32s   v1.19.2   192.168.77.102   <none>        Ubuntu 18.04.3 LTS   4.15.0-72-generic   docker://19.3.13

> ./sonobuoy run --e2e-focus="SCTPConnectivity" --e2e-skip="" --kube-conformance-image-version=v1.20.0-alpha.1 --wait
INFO[0000] created object                                name=sonobuoy namespace= resource=namespaces
INFO[0000] created object                                name=sonobuoy-serviceaccount namespace=sonobuoy resource=serviceaccounts
INFO[0000] created object                                name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterrolebindings
INFO[0000] created object                                name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterroles
INFO[0000] created object                                name=sonobuoy-config-cm namespace=sonobuoy resource=configmaps
INFO[0000] created object                                name=sonobuoy-plugins-cm namespace=sonobuoy resource=configmaps
INFO[0001] created object                                name=sonobuoy namespace=sonobuoy resource=pods
INFO[0001] created object                                name=sonobuoy-aggregator namespace=sonobuoy resource=services

> ./sonobuoy retrieve
202009252249_sonobuoy_7979918d-3e1b-4fb4-80a1-c56933312b88.tar.gz

> ./sonobuoy results 202009252249_sonobuoy_7979918d-3e1b-4fb4-80a1-c56933312b88.tar.gz
Plugin: e2e
Status: passed
Total: 5230
Passed: 5
Failed: 0
Skipped: 5225

Plugin: systemd-logs
Status: passed
Total: 3
Passed: 3
Failed: 0
Skipped: 0

I used the v1.20.0-alpha.1 conformance image for sonobuoy to ensure that the SCTP tests which are not part of 1.19 are picked up correctly. The following tests were run and passed:

    - name: '[sig-network] NetworkPolicy [Feature:SCTPConnectivity][LinuxOnly][Disruptive]
        NetworkPolicy between server and client using SCTP should enforce policy to
        allow traffic only from a pod in a different namespace based on PodSelector
        and NamespaceSelector [Feature:NetworkPolicy]'
      status: passed
    - name: '[sig-network] Networking Granular Checks: Services should function for
        pod-Service: sctp [Feature:SCTPConnectivity][Disruptive]'
      status: passed
    - name: '[sig-network] Networking should function for pod-pod: sctp [Feature:SCTPConnectivity][Disruptive]'
      status: passed
    - name: '[sig-network] NetworkPolicy [Feature:SCTPConnectivity][LinuxOnly][Disruptive]
        NetworkPolicy between server and client using SCTP should enforce policy based
        on Ports [Feature:NetworkPolicy]'
      status: passed
    - name: '[sig-network] NetworkPolicy [Feature:SCTPConnectivity][LinuxOnly][Disruptive]
        NetworkPolicy between server and client using SCTP should support a ''default-deny''
        policy [Feature:NetworkPolicy]'
      status: passed

Let me know if I need to upload these somewhere else. I am happy to provide more information / run additional tests if necessary.

[danwinship]

Output of running the current git master e2e.test --ginkgo.focus=SCTP against openshift-sdn is at https://gist.github.com/danwinship/d7e6918bd15cc46f3c4c6181f38a00aa.

{"msg":"Test Suite starting","total":9,"completed":0,"skipped":0,"failed":0}
{"msg":"PASSED [sig-network] Networking Granular Checks: Services should function for pod-Service: sctp [Feature:SCTPConnectivity][Disruptive]","total":9,"completed":1,"skipped":391,"failed":0}
{"msg":"PASSED [sig-network] NetworkPolicy [Feature:SCTPConnectivity][LinuxOnly][Disruptive] NetworkPolicy between server and client using SCTP should enforce policy to allow traffic only from a pod in a different namespace based on PodSelector and NamespaceSelector [Feature:NetworkPolicy]","total":9,"completed":2,"skipped":1575,"failed":0}
{"msg":"PASSED [sig-network] NetworkPolicy [Feature:SCTPConnectivity][LinuxOnly][Disruptive] NetworkPolicy between server and client using SCTP should enforce policy based on Ports [Feature:NetworkPolicy]","total":9,"completed":3,"skipped":2392,"failed":0}
{"msg":"PASSED [sig-network] NetworkPolicy [LinuxOnly] NetworkPolicy between server and client should not allow access by TCP when a policy specifies only SCTP [Feature:NetworkPolicy] [Feature:SCTP]","total":9,"completed":4,"skipped":3208,"failed":0}
{"msg":"PASSED [sig-network] Networking should function for pod-pod: sctp [Feature:SCTPConnectivity][Disruptive]","total":9,"completed":5,"skipped":3217,"failed":0}
{"msg":"PASSED [sig-network] NetworkPolicy [Feature:SCTPConnectivity][LinuxOnly][Disruptive] NetworkPolicy between server and client using SCTP should support a 'default-deny' policy [Feature:NetworkPolicy]","total":9,"completed":6,"skipped":4416,"failed":0}
{"msg":"PASSED [sig-network] SCTP [Feature:SCTP] [LinuxOnly] should allow creating a basic SCTP service with pod and endpoints","total":9,"completed":7,"skipped":5080,"failed":0}
{"msg":"Test Suite completed","total":9,"completed":7,"skipped":5219,"failed":0}

(The two SCTP-related tests that got skipped were It should create a Pod with SCTP HostPort (which only works with kubenet) and It should create a ClusterIP Service with SCTP ports (which only works if you're using the default kube-proxy metrics port, which this cluster isn't). But those are both [Feature:SCTP] not [Feature:SCTPConnectivity] and are tested by the existing SCTP periodic job and aren't part of the "need at least two plugins to pass the SCTPConnectivity tests" requirement.)

[kinarashah]

Hi all,

1.20 Enhancement shadow here 👋

Since this Enhancement is scheduled to be in 1.20, please keep in mind these important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thank you!

[danwinship]

Updated to GA in k/k, https://github.com/kubernetes/kubernetes/pull/95566.
Docs PR is https://github.com/kubernetes/website/pull/24593

[danwinship]

So code and docs are merged and I filed a PR (#2107) to update the KEP to "implemented". Should I make that PR be "Closes: #614" or does this issue stay open until the release team is done with it?

[SomtochiAma]

Sorry @danwinship! You are all good.

[kikisdeliveryservice]

This is awesome @danwinship !!!

I see the PR: https://github.com/kubernetes/enhancements/pull/2107 marking it as GA which looks good.

Even after that merges, we'll leave this issue open for tracking purposes and close it out when the release is finished.

Thanks again!!