Enhancements: Add IPv4/IPv6 dual-stack support

[leblancd]

Feature Description

• One-line feature description (can be used as a release note):
  IPv4/IPv6 dual-stack support and awareness for Kubernetes pods, nodes, and services
• Primary contact (assignee): @leblancd
• Responsible SIGs: sig-network
• Design proposal link (community repo): Add IPv4/IPv6 dual stack KEP (old)
• KEP PR: https://github.com/kubernetes/enhancements/pull/808
• KEP: 20180612-ipv4-ipv6-dual-stack
• Link to e2e and/or unit tests: TBD
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @thockin @dcbw @luxas
• Approver (likely from SIG/area to which feature belongs): @thockin
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target 1.11
  
  
  • Beta release target 1.20
  
  
  • Stable release target (x.y)
  
  

Corresponding kubernetes/kubernetes Issue: https://github.com/kubernetes/kubernetes/issues/62822

[leblancd]

Cross Reference with kubernetes/kubernetes: Issue #62822

[justaugustus]

Thanks for the update!

/assign @leblancd
/kind feature
/sig network
/milestone 1.11

[idvoretskyi]

@leblancd any design document available?

/cc @thockin @dcbw @luxas @kubernetes/sig-network-feature-requests

[leblancd]

@idvoretskyi - No design doc yet, but we'll start collaborating on one shortly.

[sb1975]

Does this mean Kubernetes Ingress will support Dual-Stack ?
Does this mean CNI ( Calico) would need to run Dual stack ( both BIRD and BIRD6 daemons for example) ?

[leblancd]

@sb1975 - Regarding dual-stack ingress support, that's something we'll need to hash out, but here are my preliminary thoughts:

• Dual stack ingress support will mostly depend upon which ingress controller you use (whether it's supported and how it's implemented). Existing ingress controllers will probably need some changes to support dual-stack.
• I expect that the ingress configuration for a typical ingress controller won't change (the config might e.g. still map an L7 address to a service name / service port, with no mention of V4/V6 family)
• In the case where a service has endpoint pods that are dual-stack, the ingress controller(s) might need changes to map ingress packets based on the packets' family, i.e. map IPv4 ingress packets to an IPv4 endpoint, and map IPv6 ingress packets to an IPv6 endpoint. For the purposes of load-balance weighting, a dual-stack endpoint should count as a single endpoint target.

- We might want to consider FUTURE support for having an ingress controller map across V4/V6 families (map ingress IPv4 packets to an IPv6 backend, and vice versa), but our initial development will be for strict dual-stack (i.e. separate, independent stacks).

Regarding Calico and other CNI plugins:

• The CNI plugins won't HAVE TO run in dual-stack mode if a cluster scenario doesn't require dual-stack, they should still be able to run IPv4-only or IPv6-only (if the plugin supports it).
• Dual-stack support will probably require changes in the various CNI plugins, but that work is considered outside of the scope of this Kubernetes issue (we're focusing on making Kubernetes work for any arbitrary dual-stack plugin, probably using the bridge plugin as a reference), and the CNI work will be done separately on a case-by-case basis.
• For Calico specifically, I'm not an expert but I believe that a single BIRD daemon can be configured to handle both IPv4 and IPv6 routes (search for "template bgp" here: http://bird.network.cz/?get_doc&v=20&f=bird-3.html#ss3.1). That said, although Calico already supports dual-stack addresses at the pod, there might be changes required to get the BGP routing working for both families.

[sb1975]

@leblancd : So here is the scenario :

1. Lets say we will use NGINX ingress controller
2. I am exposing my services via Ingress.
3. I am running my pods configured on dual-stack
4. I am trying to reach the service remotely using A and AAAA dns-records.
   Hope all of these
5. In summary : I want to connect to pod interfaces using either IPv4 or IPv6 addresses, as resolved by my own queries for A and/or AAAA records for the pod service name.
   Can I get involved in this initiative to test,documentation,architecture: but need some guidance.
   How do I get to know about the progress of this please.

[leblancd]

@sb1975 - Good question re. the NGINX ingress controller with dual-stack. I'm not an expert on the NGINX ingress controller (maybe someone more familiar can jump in), but here's how I would see the work flow:

• When you try to reach the Kube services from outside, your DNS controller should resolve the service with A and AAAA DNS records for the ingress controller. It would be your client/app's choice to use the A vs. the AAAA record to reach the ingress controller. So external access to the ingress controller would be dual-stack.
• At the NGINX ingress controller, NGINX would then look at the L7 URL (regardless of request being in an IPv4 or IPv6 packet) and load balance that to upstream endpoints. If the ingress controller load balancer is configured with ipv6=on (which is default, see https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#configuring-http-load-balancing-using-dns), and the service endpoint(s) are dual-stack, then the upstream configuration should have both IPv4 and IPv6 entries for each dual-stack endpoint. As designed, the NGINX load balancer treats the IPv4 entry and the IPv6 entry for an endpoint as separate servers. (See the line in the fore-mentioned doc "If a domain name resolves to several IP addresses, the addresses are saved to the upstream configuration and load balanced.") This can be considered good news or bad news. The good news is that load balancing will be done across IPv4 and IPv6 endpoints (giving you some redundancy), where e.g. an incoming IPv4 request could get mapped to either an IPv4 or an IPv6 endpoint. But the potential bad news is with load balancing granularity: a connection to an IPv4 endpoint and a connection to the corresponding IPv6 endpoint will be treated (for load balancing considerations) as 2 loads to separate endpoints, rather than 2 separate loads to the same endpoint. If this load-balancing granularity is a concern, then someone could disable load balancing to IPv6 (or to IPv4, if there's a config knob for that?), so that load balancing would be to IPv4-only endpoints. OR, maybe NGINX load balancer can be modified to treat a connection to an IPv4 address and a connection to its corresponding IPv6 address as 2 loads to the same endpoint.

As for helping and getting involved, this would be greatly appreciated! We're about to start working in earnest on dual-stack (it's been a little delayed by the work in getting CI working for IPv6-only). I'm hoping to come out with an outline for a spec (Google Doc or KEPs WIP doc) soon, and would be looking for help in reviewing, and maybe writing some sections. We'll also DEFINITELY need help with official documentation (beyond the design spec), and with defining and implementing dual-stack E2E tests. Some of the areas which I'm still a bit sketchy on for the design include:

• How are health/liveness/readiness probes affected or handled with dual-stack
• Will there be an impact on network policies?
• Load balancer concerns?
• Cloud provider plugin concerns?
• L3/L4 ingress concerns?
  If you've thought about any of these, maybe you could help with those sections?

We're also considering an intermediate "dual-stack at the edge" (with IPv6-only inside the cluster) approach, where access from outside the cluster to K8s services would be dual-stack, but this would be mapped (e.g. via NGINX ingress controller) to IPv6-only endpoints inside the cluster (or use stateless NAT46). Pods and services in the cluster would need to be all IPv6, but the big advantage would be that dual-stack external access would be available much more quickly from a time-to-market perspective.

[caseydavenport]

/milestone 1.12

[justaugustus]

@leblancd / @caseydavenport - I'm noticing a lot of discussion here and a milestone change.
Should this be pulled from the 1.11 milestone?

[leblancd]

@justaugustus - Yes, this should be moved to 1.12. Do I need to delete a row in the release spreadsheet, or is there anything I need to do to get this changed?

[justaugustus]

@leblancd I've got it covered. Thanks for following up! :)

[justaugustus]

@leblancd @kubernetes/sig-network-feature-requests --

This feature was removed from the previous milestone, so we'd like to check in and see if there are any plans for this in Kubernetes 1.12.

If so, please ensure that this issue is up-to-date with ALL of the following information:

• One-line feature description (can be used as a release note):
• Primary contact (assignee):
• Responsible SIGs:
• Design proposal link (community repo):
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
• Approver (likely from SIG/area to which feature belongs):
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target (x.y)
  
  
  • Beta release target (x.y)
  
  
  • Stable release target (x.y)
  
  

Set the following:

• Description
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

/cc @justaugustus @kacole2 @robertsandoval @rajendar38

[justaugustus]

@leblancd --
Feature Freeze is today. Are you planning on graduating this to Beta in Kubernetes 1.12?
If so, can you make sure everything is up-to-date, so I can include it on the 1.12 Feature tracking spreadsheet?

[leblancd]

Hi @justaugustus - Beta status will need to slip into Kubernetes 1.13. We are making (albeit slow) progress on the design KEP (https://github.com/kubernetes/community/pull/2254), and we're getting close to re-engaging with the CI test PR, but the Kubernetes 1.12 target was a bit too optimistic.

I'll update the description/summary above with the information you requested earlier. Thank you for your patience.

[justaugustus]

/remove-stage alpha
/stage beta

[justaugustus]

No worries, @leblancd. Thanks for the update!

[navjotsingh83]

Hi, @justaugustus @leblancd

I just read the update that the beta is moved to 1.13 for dual stack. What is the expected release date of 1.13? We are actually looking for dual stack support. Its a go-nogo decision for our product to move to containers.

[leblancd]

@navjotsingh83 - I don't think the release date for Kubernetes 1.13 has been solidified. I don't see 1.13 listed in the Kubernetes releases documentation.

[AishSundar]

@navjotsingh83 @leblancd 1.13 release schedule is published. Its a short release cycle with Code freeze on Nov 15th. Do you think its enough time to graduate this feature to Beta. Can you plz update this issue with your level of confidence, whats pending in terms of code, test and docs completion?

[AishSundar]

As per discussion in the SIG Network meeting, though there will considerable work done on this feature in 1.13, it is not expected to go to Beta in 1.13. removing milestone accordingly.

/milestone clear

[AishSundar]

@kacole2 to remove this from 1.13 enhancements spreadsheet

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[caseydavenport]

/remove-lifecycle stale

[claurence]

@leblancd Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

[KevinAtDesignworx]

@leblancd Wanted to follow up on your prior comment relative to creating a delineation at the edge of the cluster for IPv4/IPv6:

“We're also considering an intermediate "dual-stack at the edge" (with IPv6-only inside the cluster) approach, where access from outside the cluster to K8s services would be dual-stack, but this would be mapped (e.g. via NGINX ingress controller) to IPv6-only endpoints inside the cluster (or use stateless NAT46). Pods and services in the cluster would need to be all IPv6, but the big advantage would be that dual-stack external access would be available much more quickly from a time-to-market perspective.”

This use case would be a good one for a current project, so wanted to see your thoughts around timeframe, see if there was anything myself or someone in our group could contribute to help out with this quicker time to market path.

[steebchen]

@KevinAtDesignworx If the edge-dual-stack but internal ipv6-only approach can still reach external ipv4 requests from inside a container (i.e. curl -v 93.184.216.34 -H "Host: example.com"), I genuinely think it's the best approach. If your infrastructure can use ipv6, why bother use ipv4 except at the edge for compatibility reasons. However, if this approach means that I can not reach legacy websites solely using ipv4 from inside my cluster, I'm not so sure anymore.

[schmitch]

well there is 464XLAT so ipv6 only inside the container would be feasable.

[leblancd]

@KevinAtDesignworx - If using an ingress controller would work in your scenario, it's possible to configure an NGINX ingress controller for dual-stack operation from outside (proxying to single-family inside the cluster): https://github.com/leblancd/kube-v6#installing-a-dual-stack-ingress-controller-on-an-ipv6-only-kubernetes-cluster

The ingress controllers would need to run on the host network on each node, so the controllers wold need to be set up as a daemonset (one ingress controller on each node). This assumes:

• Your nodes are dual-stack (as opposed to pods and services being single-family)
• Your /etc/hosts on each node has IPv6 entries, and only IPv6 entries (no IPv4 addresses) for that node's hostname.

This would be in addition to a NAT64/DNS64 for connections from V6 clients inside the cluster to external IPv4-only servers.

Stateless NAT46 is also an option, but I haven't tried that, so I don't have any config guides for that.

[kacole2]

@leblancd any work planned here for 1.15? Looks like a KEP hasn't been accepted yet at this point either. Thanks!

[GeorgeGuo2018]

@leblancd Wanted to follow up on your prior comment relative to creating a delineation at the edge of the cluster for IPv4/IPv6:

“We're also considering an intermediate "dual-stack at the edge" (with IPv6-only inside the cluster) approach, where access from outside the cluster to K8s services would be dual-stack, but this would be mapped (e.g. via NGINX ingress controller) to IPv6-only endpoints inside the cluster (or use stateless NAT46). Pods and services in the cluster would need to be all IPv6, but the big advantage would be that dual-stack external access would be available much more quickly from a time-to-market perspective.”

This use case would be a good one for a current project, so wanted to see your thoughts around timeframe, see if there was anything myself or someone in our group could contribute to help out with this quicker time to market path.

From inside a container(which is only ipv6) sendding out a curl request (i.e. curl -v 93.184.216.34 -H "Host: example.com") to the outside of the cluster. I think it will give out an error of unknow destination or destination unreachable, unless there exist an ipv4 route on the host where the container exist.

[schmitch]

@GeorgeGuo2018 if k8s would implement DNS64/NAT64 it would work. it heavily depends on how far k8s will go into 464xlat/plat solutions and what would need to be handled at edge routers, etc...

actually I think it would be possible by using a DaemonSet/Deployment that uses host networking and Tayga inside the kube-system namespace so that the internal DNS64 would use tayga to go outside the network.

[chrisnew]

Sounds like a solution to me.

We run an IPv6-only network internally and NAT64/DNS64 works quite well for us. For some legacy stuff where there was no IPv6 support at all, we ended up using clatd directly where it was needed. (In our case directly on a VM.)

[lachie83]

@kacole2 - I would like this tracked for 1.15. I'm working to get the following PR merged - https://github.com/kubernetes/enhancements/pull/808

Specifically for 1.15 we would be adding support for the following:

• API type modifications
  
  
  
  • Kubernetes types
  
  
  • CRI types
  
  
• dual-stack pod networking (multi-IP pod)
• kubenet multi-family support

[cmluciano]

cc @caseydavenport for milestone tracking ^

[lachie83]

@kacole2 the KEP is now merged. Let me know if there is anything else we need to get this tracked in 1.15

[simplytunde]

Hey @leblancd @lachie83 Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[GeorgeGuo2018]

@kacole2 the KEP is now merged. Let me know if there is anything else we need to get this tracked in 1.15

@lachie83 Hi,Lachie,did you mean the IPv4/IPv6 dual-stack support this KEP was finished?

[GeorgeGuo2018]

@kacole2 the KEP is now merged. Let me know if there is anything else we need to get this tracked in 1.15

Actually, I want to figure out whether dual stack support will surely be add in k8s 1.15.

[simplytunde]

@leblancd The placeholder PR against k8s.io dev-1.15 is due Thursday May 30th.

[GeorgeGuo2018]

@leblancd The placeholder PR against k8s.io dev-1.15 is due Thursday May 30th.

Could I consider that dual-stack support will be available in release-1.15?

[simplytunde]

@GeorgeGuo2018 It is still on the enhancement sheet for 1.15 but only enhancement lead @kacole2 can provide you with better details on that.

[kacole2]

Hi @lachie83 @leblancd. Code Freeze is Thursday, May 30th 2019 @ EOD PST. All enhancements going into the release must be code-complete, including tests, and have docs PRs open.

Please list all current k/k PRs so they can be tracked going into freeze. If the PRs aren't merged by freeze, this feature will slip for the 1.15 release cycle. Only release-blocking issues and PRs will be allowed in the milestone.

I see kubernetes/kubernetes#62822 in the original post is still open. Are there other PRs we are expecting to be merged as well?

If you know this will slip, please reply back and let us know. Thanks!

[lachie83]

@simplytunde - Appreciate the heads up. I am working on getting the docs PR together this week.

[lachie83]

@GeorgeGuo2018 - This is going to be a multi-release KEP. We plan on landing phase 1 in 1.15. Please take a look at the implementation plan in the KEP for further detail - https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/20180612-ipv4-ipv6-dual-stack.md#implementation-plan.

[lachie83]

@simplytunde - I've created the initial placeholder docs PR here with a WIP https://github.com/kubernetes/website/pull/14600. I plan to complete and have it ready for review over the next couple of days.

[lachie83]

@kacole2 Thanks for the ping. I've updated the 1.15 enhancements speadsheet with the k/k PR that we are tracking (https://github.com/kubernetes/kubernetes/pull/73977) along with the draft docs PR (https://github.com/kubernetes/website/pull/14600). We are still currently on track to get this PR merged before code freeze. LMK if I'm missing anything else

[lachie83]

@kacole2 after discussion with @claurence and the release team we've decided to remove this from the 1.15 milestone. Please go ahead and remove it and update the spreadsheet as appropriate. Thanks for all your assistance thus far.

[kacole2]

/milestone clear

[lachie83]

@simplytunde I've also commented on the docs PR. Can you please make sure that's removed from the 1.15 milestone also?

[evillgenius75]

Hi @lachie83 @leblancd , I'm the 1.16 Enhancement Shadow. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[sftim]

https://github.com/kubernetes/dns/issues/315 covers adding IPv6 / AAAA to the DNS service discovery specification.

[kacole2]

@lachie83 @leblancd any idea if this will be graduating in 1.16 to track it?

[lachie83]

@evillgenius75 @kacole2 This needs to be tracked in 1.16. This feature will be in alpha state. We will be implementing phase 1 and phase 2 as defined in the KEP is 1.16

Tracking KEP

• https://github.com/kubernetes/enhancements/pull/808

Merged k/k PRs (currently in master will be in 1.16)

• [MERGED] Phase 1 - https://github.com/kubernetes/kubernetes/pull/73977
• [MERGED] Phase 1 e2e tests - https://github.com/kubernetes/kubernetes/pull/78801
• [MERGED] Phase 1 e2e test cleanup - https://github.com/kubernetes/kubernetes/pull/80001
• Phase 2 Services/Endpoints - https://github.com/kubernetes/kubernetes/pull/79386
• Phase 2 kube-proxy - https://github.com/kubernetes/kubernetes/pull/79576

Associated PRs

• Support for IPv6 https://github.com/kubernetes-incubator/ip-masq-agent/pull/45
• Phase 1 docs – will be reopened once we finalize tooling examples - https://github.com/kubernetes/website/pull/14600
• [MERGED] clean up - node dropDisabledFields - https://github.com/kubernetes/kubernetes/pull/79741
• Support multiple Mask Sizes for cluster cidrs - https://github.com/kubernetes/kubernetes/pull/79993
• E2e Prow Job for dualstack https://github.com/kubernetes/test-infra/pull/12966

[simplytunde]

Hey, @leblancd I'm the v1.16 docs release lead.

Does this enhancement (or the work planned for v1.16) require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday,August 23rd. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[lachie83]

@simplytunde here is the docs PR - https://github.com/kubernetes/website/pull/16010

[kacole2]

@lachie83 friendly reminder code freeze for 1.16 is on Thursday 8/29. (as if you didn't know that). Looks like these PRs are still outstanding:
Phase 2 Services/Endpoints - kubernetes/kubernetes#79386
Phase 2 kube-proxy - kubernetes/kubernetes#79576

Associated:
Support multiple Mask Sizes for cluster cidrs - kubernetes/kubernetes#79993
E2e Prow Job for dualstack kubernetes/test-infra#12966

[kacole2]

Hi @lachie83 @leblancd it looks as though https://github.com/kubernetes/kubernetes/pull/79576 and https://github.com/kubernetes/kubernetes/pull/79993 didn't merge before code freeze and it's not in the Tide Merge Pool. This feature is going to be bumped from v1.16. If you would still like to have this be a part of the 1.16 release, please file an exception

[lachie83]

@kacole2 Apologies for the delays in response. The primary PR were were tracking was https://github.com/kubernetes/kubernetes/pull/79386. As for kubernetes/kubernetes#79576 we made a decision to defer that to 1.17 and instead focus on https://github.com/kubernetes/kubernetes/pull/82091 (in agreement with sig-network) which fulfills the same phase2 goals that were laid out in the KEP. The other related PR was tracked in this release was https://github.com/kubernetes/kubernetes/pull/80485 which is also merged. kubernetes/kubernetes#79993 has also been deferred to 1.17

[mrbobbytables]

Hey there @lachie83 @leblancd -- 1.17 Enhancements lead here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

/milestone clear

[lachie83]

Hi Bob. Thanks for reaching out. I'm still planning phase 3 of this enhancement which will round out the enhancement to completion. This enhancement will still be in alpha at the end of this release but there will be phase 3 related work that will land in k/k as part of 1.17.

[lachie83]

Here is a list of high level deliverables for 1.17 for dual-stack. I will update this list throughout the release.

• IPtables dual-stack proxier - https://github.com/kubernetes/kubernetes/pull/79576
• downward api support for Pod.Status.PodIPs - https://github.com/kubernetes/kubernetes/pull/83123
• e2e test one CNI plugin
• Hostfile changes (IPv6 address for hostname needs to be added /etc/hosts) - https://github.com/kubernetes/kubernetes/pull/83123
• Kind dual-stack - https://github.com/kubernetes-sigs/kind/pull/692
• Update e2e suite - https://github.com/kubernetes/kubernetes/pull/80725
• Endpoint slice support
• Dual-Stack support for Kubeadm - https://github.com/kubernetes/kubeadm/issues/1612

[mrbobbytables]

Much appreciated, thank you kindly @lachie83 ❤️ I'll go ahead and add it to the tracking sheet.

/milestone v1.17

[lachie83]

@mrbobbytables I've also added a PR to detail the work listed above as part of phase 3 in the KEP after communicating the plan via sig-network. The KEP itself is still in the implementable state and these changes are merely documenting the planned work as part of 1.17 specifically.

[sftim]

At some point, I'd like to ensure that https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/ covers IPv6 DNS. https://github.com/kubernetes/website/issues/15434 tracks that change; mentioning it here to note a cross-reference.

[lachie83]

Updated KEP to add phase 2 e2e tests - https://github.com/kubernetes/enhancements/pull/1311

[irvifa]

Hello @lachie83 I'm one of the v1.17 docs shadows.
Does this enhancement for (or the work planned for v1.17) require any new docs (or modifications to existing docs)? If not, can you please update the 1.17 Enhancement Tracker Sheet (or let me know and I'll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.17) due by Friday, November 8th, it can just be a placeholder PR at this time. Let me know if you have any questions!

[irvifa]

@lachie83

Since we're approaching Docs placeholder PR deadline on Nov 8th. Please try to get one in against k/website dev-1.17 branch.

[mrbobbytables]

Hey there @lachie83, I know you're keepin' tabs, but I need to pop in and mention it anyway 🙈
Code freeze is just around the corner (November 14th). How are things looking? Is everything on track to be merged before then?

Thanks!

[lachie83]

Hey @mrbobbytables! Thanks for the ping. We are tracking the following PRs to land in 1.17. There may be one or two more PRs associated with this change that come in too. These changes will need docs. I will raise a placeholder docs PR

• Dualstack support for kube-proxy iptables mode pr/82462
• Dualstack downward api pr/82462
• add ipvs metaproxier endpointslice support pr/84089
• Allow multiple node cidr masks in kube-controller-manager pr/79993

[lachie83]

@irvifa - Here is the placeholder docs PR. https://github.com/kubernetes/website/pull/17457

[irvifa]

Cool thanks 🎉 @lachie83

[mrbobbytables]

@lachie83 tomorrow is the code freeze for the 1.17 release cycle. It looks like the k/k PRs have not yet been merged. 😬 We're flagging this as At Risk in the 1.17 Enhancement Tracking Sheet.
Do you think they will be merged by the EoD of the 14th (Thursday)? After that point, only release-blocking issues and PRs will be allowed in the milestone with an exception.

[lachie83]

Thanks Bob - I'll be discussing this with sig-network today and will provide an update.

[lachie83]

Hey @mrbobbytables. Here is a list of PRs that we are working on getting merged by EoD today and have been approved by sig-network.

• https://github.com/kubernetes/kubernetes/pull/85246
• https://github.com/kubernetes/kubernetes/pull/83123
• https://github.com/kubernetes/kubernetes/pull/79993

The remaining PR is most likely going to be punted to 1.18 - https://github.com/kubernetes/kubernetes/pull/82462

[lachie83]

@mrbobbytables just confirming that all stated PRs above have been merged and that we are indeed going to punt kubernetes/kubernetes#82462 to 1.18. This enhancement can still be tracked as these PRs add meaning changes to the dualstack behavior in 1.17. Now I just need to get the docs PR ready! We are hoping to land kubernetes/kubernetes#82462 in 1.18 and progress this work to beta

[mrbobbytables]

Great, thanks @lachie83!

[CharlieTLe]

And it's live! https://kubernetes.io/blog/2019/12/09/kubernetes-1-17-release-announcement/

[lachie83]

We plan to move this enhancement to beta in 1.18. Enhancement graduation criteria and test plans can be found in the KEP along with this PR - https://github.com/kubernetes/enhancements/pull/1429

[lachie83]

/milestone 1.18

[k8s-ci-robot]

@lachie83: The provided milestone is not valid for this repository. Milestones in this repository: [keps-beta, keps-ga, v1.17, v1.18, v1.19, v1.20, v1.21]

Use /milestone clear to clear the milestone.

In response to this:

/milestone 1.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[lachie83]

/milestone v1.18

[jeremyrickard]

We plan to move this enhancement to beta in 1.18. Enhancement graduation criteria and test plans can be found in the KEP along with this PR - #1429

Thanks for the update @lachie83, I've marked this as tracked in the 1.18 spreadsheet!

[lachie83]

Please track the following PR as part of the work to land in 1.18. https://github.com/kubernetes/kubernetes/pull/82462

Adding other related PRs for tracking:
https://github.com/kubernetes/test-infra/pull/15893
https://github.com/kubernetes-sigs/kind/pull/692

[jeremyrickard]

Thanks @lachie83!

[palnabarun]

Hi @lachie83, do you have any other PR's that we should track other than the above-mentioned ones?

[sethmccombs]

Hello, @lachie83 @leblancd - I'm a Docs shadow on the 1.18 release team.

Does this enhancement work planned for 1.18 require any new docs or modifications to existing docs?

If not, can you please update the 1.18 Enhancement Tracker Sheet (or let me know and I'll do so)

If doc updates are required, reminder that the placeholder PRs against k/website (branch dev-1.18) are due by Friday, Feb 28th.

Let me know if you have any questions!

[sftim]

If anyone wants help documenting IPV6 or dual-stack stuff for v1.18, give me a nudge. I may be able to help.

[jeremyrickard]

Hey @lachie83,

Looks like kubernetes-sigs/kind#692 hasn't merged yet. Is that critical for your Beta graduation?

[lachie83]

Hey @jeremyrickard @sethmccombs we're going to have to pull this from graduating to beta given this PR https://github.com/kubernetes/kubernetes/pull/86895. Until we have a reasonable way forward I don't think it is wise to move this to beta for 1.18

[palnabarun]

/milestone clear

[palnabarun]

@lachie83 Thank you for the update. I've removed this enhancement from the milestone. Looking forward to this on 1.19. :)

[lachie83]

I would like to confirm that the state of the dualstack enhancement remains in alpha in 1.18. I am currently working with the community to assess the work planned to be completed in 1.19. It's likely this enhancement will still remain in alpha state in 1.19 however I would like to confirm. I will also take an action on getting the docs updated to reflect the enhancement state in the 1.18 docs.

[sftim]

If there are pages on the website that show dual-stack Kubernetes as beta, please file those against k/website as priority/important-soon bugs.

[palnabarun]

Hi @lachie83 -- 1.19 Enhancements Lead here, I wanted to check in if you think this enhancement would graduate in 1.19?

---

The current release schedule is:

• Monday, April 13: Week 1 - Release cycle begins
• Tuesday, May 19: Week 6 - Enhancements Freeze
• Thursday, June 25: Week 11 - Code Freeze
• Thursday, July 9: Week 14 - Docs must be completed and reviewed
• Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released

[lachie83]

If there are pages on the website that show dual-stack Kubernetes as beta, please file those against k/website as priority/important-soon bugs.

@sftim I've raised two PRs to address the release labelling in 1.17 and 1.18

[lachie83]

@palnabarun We are working to get the dualstack KEP updated in the 1.19 release timeframe however we don't currently think that we will be landing code changes in the 1.19 release. We have one blocking issue with the work that's already been done (thanks to having it in alpha state). The blocking issue is https://github.com/kubernetes/kubernetes/pull/86895. We plan to address that via the follow KEP update https://github.com/kubernetes/enhancements/pull/1679 but it's going to take time get consensus on the proposed change. At this stage the dualstack enhancement will remain in alpha state until we address this blocking issue with the current implementation. I will provide updates as things progress.

[palnabarun]

Thank you, Lachie for the updates. I appreciate all the efforts! :slightly_smiling_face:

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[Reamer]

/remove-lifecycle stale

[lachie83]

We would like this enhancement to be tracked in 1.20. It will be reimplemented in alpha state according to the updated kep - https://github.com/kubernetes/enhancements/pull/1679. Please track the following PR for the implementation - https://github.com/kubernetes/kubernetes/pull/91824. We are planning to complete the review and merge the PR early in the 1.20 release cycle.

[dcbw]

Latest dual-stack graduation to Beta status as discussed in Sept 17th's SIG Network meeting, for those playing along at home:

• Must resolve kubelet dual-stack health-check ambiguity; via https://github.com/kubernetes/enhancements/pull/1975
• Must finish and merge the existing dual-stack services PR (under active review); via https://github.com/kubernetes/kubernetes/pull/91824
• Must finalize behavior of dual-stack node IPs; via https://github.com/kubernetes/enhancements/pull/1665
• Should determine what (if any) changes required for Service's Spec.LoadBalancerIPs field

All these items are being actively worked on, and 1.20 is still the target for dual-stack API Beta graduation. However despite our best efforts there is always a chance something will not be resolved in time, and if so, SIG Network will decide whether to continue graduation to Beta or not in our public meetings. All are welcome to join.

[lachie83]

@dcbw thank you very much for the update (sorry I couldn't make the call). Does it make sense to get this to enhancement to beta in 1.20 or simply remain in alpha? If we want to go to beta does the graduation criteria in the KEP still make sense given that this is a reimplementation https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/20180612-ipv4-ipv6-dual-stack.md#graduation-criteria

[russellb]

@dcbw thank you very much for the update (sorry I couldn't make the call). Does it make sense to get this to enhancement to beta in 1.20 or simply remain in alpha? If we want to go to beta does the graduation criteria in the KEP still make sense given that this is a reimplementation https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/20180612-ipv4-ipv6-dual-stack.md#graduation-criteria

It's not really a reimplementation, though. All of the previous work is still valid and the work in 1.20 is building on top of it to finalize the last changes needed that have been identified. My interpretation of the sig-network discussion is that the list @dcbw posted is the set of remaining known issues needed to be resolved for graduation.

[kikisdeliveryservice]

Hi all,

1.20 Enhancements Lead here, I'm going to set this as tracked please update me if anything changes :)

As a reminder Enhancements Freeze is October 6th.

As a note the KEP is using an old format we have updated to : https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template

Best,
Kirsten

/milestone v1.20

[bridgetkromhout]

Hi, @russellb -

It's not really a reimplementation, though. All of the previous work is still valid and the work in 1.20 is building on top of it to finalize the last changes needed that have been identified.

Given the API changes in https://github.com/kubernetes/kubernetes/pull/91824, enough is different that marking dual-stack as alpha for 1.20 will allow room for any further re-implementations that prove necessary. I know we're all eager for beta, but let's first land the PR with +9,319 −3,261 and let the dust settle. :)

[dcbw]

Given the API changes in kubernetes/kubernetes#91824, enough is different that marking dual-stack as alpha for 1.20 will allow room for any further re-implementations that prove necessary. I know we're all eager for beta, but let's first land the PR with +9,319 −3,261 and let the dust settle. :)

@bridgetkromhout yeah, we need to land https://github.com/kubernetes/kubernetes/pull/91824 before we can make any determination about API readiness. I really hope we can do that ASAP.

[kinarashah]

Hi all,

1.20 Enhancement shadow here 👋

Since this Enhancement is scheduled to be in 1.20, please keep in mind these important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thank you!

[lachie83]

Hi @kinarashah @kikisdeliveryservice - I have confirmed on the sig-network call that we need this reclassified to alpha for 1.20. It's a complete reimplementation that needs time to soak and be tested in alpha stage.

[reylejano-rxm]

Hello @lachie83, 1.20 Docs shadow here.

Does this enhancement work planned for 1.20 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against the dev-1.20 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Nov 6th.

Also take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[lachie83]

Thanks @reylejano-rxm - we've opened kubernetes/website#24725

[reylejano-rxm]

Hi @lachie83

Thanks for creating the docs PR!

Please keep in mind the important upcoming dates:

• Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline DONE!
• Thursday, Nov 12th: Week 9 - Code Freeze
• Monday, Nov 23rd: Week 11 - Docs PR Ready for Review

As a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

[kikisdeliveryservice]

Hi @kinarashah @kikisdeliveryservice - I have confirmed on the sig-network call that we need this reclassified to alpha for 1.20. It's a complete reimplementation that needs time to soak and be tested in alpha stage.

Hey @lachie83

Given the above, I presume that this is still intended for alpha as-is? I don't see any outstanding PRs that need to merge / work was already merged.

_Just a reminder that Code Freeze is coming up in 2 days on Thursday, November 12th. All PRs must be merged by that date, otherwise an Exception is required._

Thanks!
Kirsten

[bridgetkromhout]

Hi, @kikisdeliveryservice - yes, IPv4/IPv6 dual-stack support (reimplemented) will be alpha for 1.20.

Here's the progress we have for this enhancement:

1) Code is merged from https://github.com/kubernetes/kubernetes/pull/91824 - will be alpha for 1.20
2) Documentation updates covering that code change are in https://github.com/kubernetes/website/pull/24725/ - reviewed and merged into the dev-1.20 branch

Is there anything else needed for 1.20 that we haven't completed on this enhancement?

[kikisdeliveryservice]

@bridgetkromhout Thanks for the clear update, you're all good!

[chenwng]

It looks like LoadBalancerIP in ServiceSpec is not part of the dual-stack implementation yet. Is there any plan to support it or did I miss it?

[lachie83]

Hi @chenwng - Changes to cloud provider code for Loadbalancers are out of scope currently as defined in the KEP here - https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/20180612-ipv4-ipv6-dual-stack.md#load-balancer-operation.

You can help by providing your use case and suggested changes to understand and decide if we need to make any modifications to the KEP.

[aramase]

@chenwng There is a KEP being worked on for LoadBalancerIPs in dual-stack clusters - https://github.com/kubernetes/enhancements/pull/1992

[chenwng]

Thanks for the info, @aramase , @lachie83 .