Kubeadm: Dual-Stack support for Kubeadm

[Arvinderpal]

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST

In order to support the forthcoming dual-stack work, certain changes must be made to kubeadm so that it can create dual-stack clusters. The dual-stack KEP (https://github.com/kubernetes/enhancements/pull/808) has a section on changes required in kubeadm.

Phase-1 Tasks:

• [x] kubeadm feature-gate https://github.com/kubernetes/kubernetes/pull/80145
• [x] Propagate kubeadm feature-gate to all k8s components https://github.com/kubernetes/kubernetes/pull/80531
• [x] --cluster-cidr support for comma delimited list of IPs https://github.com/kubernetes/kubernetes/pull/79033
• [x] e2e tests for --cluster-cidr/PodSubnet changes https://github.com/kubernetes/kubernetes/pull/80947

Phase-2 Tasks:

• [x] service-cluster-ip-range support for dual-stack. https://github.com/kubernetes/kubernetes/pull/82473
• [x] e2e tests for service-cluster-ip-range (these are in kind upstream)
• [ ] Add documentation on enabling dual-stack via kubadm
  https://github.com/kubernetes/website/pull/26675

Additional Tasks:

• [x] Option node-cidr-mask-size to kube-controller manager needs to support a list of CIDRs. This currently defaults to /24 (for both ipv4 and ipv6). Func calcNodeCidrSize must be updated to handle dual-stack. https://github.com/kubernetes/kubernetes/pull/85609
• [x] don't hardcode the ipv4 mask size to 24?
  https://github.com/kubernetes/kubeadm/issues/2327
  https://github.com/kubernetes/kubernetes/issues/90922
  fixed in:
  https://github.com/kubernetes/kubernetes/commit/8b52995d328a50169924eb27d8d03cb47a9ceac6#diff-d870edeb820f49abf2f52cd357a8a5396c7cd1a36e024d339146e512186c7297

TBD:

• [ ] Add apiserver-advertise-address support for dual-stack

[neolit123]

thanks for logging the ticket @Arvinderpal
@aojea has been interested in implementing IPV6 support in kind, which uses kubeadm under the hood and adding testing signal for that.

@aojea please take a look and communicate with @Arvinderpal who wishes to add dual stack support in kubeadm.

/assign @aojea @Arvinderpal

[k8s-ci-robot]

@neolit123: GitHub didn't allow me to assign the following users: aojea.

Note that only kubernetes members and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

thanks for logging the ticket @Arvinderpal
@aojea has been interested in implementing IPV6 support in kind, which uses kubeadm under the hood and adding testing signal for that.

@aojea please take a look and communicate with @Arvinderpal who wishes to add dual stack support in kubeadm.

/assign @aojea @Arvinderpal

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

the IPV6 PRs merged in kind.
https://github.com/kubernetes-sigs/kind/pull/636
also for test signal:
https://github.com/kubernetes/test-infra/pull/13123
(should appear here https://k8s-testgrid.appspot.com/sig-testing-kind#kind%20(IPv6),%20master%20(dev))

[BenTheElder]

been keeping an eye on the dual stack work for exactly this @neolit123 👀

[aojea]

/cc

[Arvinderpal]

@neolit123 I would like to add a doc for setting up dual-stack with kubeadm. I have a draft ready. Where would be a good place to add this?

[neolit123]

hi @Arvinderpal we decided to do that for the beta release. is dual-stack going beta in 1.17?
if not possibly we should delay these docs until the beta.
if yes the location IMO should be in this doc in a new section, possibly under this one:
https://github.com/kubernetes/website/blob/master/content/en/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm.md#initializing-your-control-plane-node

also i'm on PTO this week and please direct any review requests to the rest of the group. :)

[SataQiu]

/cc

[sftim]

I'm concerned about a /24 default netblock allocation for IPv6. That's really large; it'd be more realistic to default to a /64 and let cluster operators override that if they know better.

[neolit123]

@sftim

I'm concerned about a /24 default netblock allocation for IPv6. That's really large; it'd be more realistic to default to a /64 and let cluster operators override that if they know better.

@Arvinderpal should correct me, but i think such decisions are on the plate of SIG Network.

[Arvinderpal]

I totally agree that /24 is not appropriate for IPv6. It's more a carryover from IPv4.

@Arvinderpal should correct me, but i think such decisions are on the plate of SIG Network.
Yes, sig-network is the right place.

[aojea]

IIRC /24 is a legacy thing and we can't change it until this merge https://github.com/kubernetes/kubernetes/pull/79993

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/remove-lifecycle stale

[neolit123]

@Arvinderpal @aojea hi, are we going to see Beta graduation for this feature in core k8s for 1.19?

[aojea]

@Arvinderpal @aojea hi, are we going to see Beta graduation for this feature in core k8s for 1.19?

KEP is going to be updated soon , seems this is the intention https://github.com/kubernetes/kubernetes/pull/86895#issuecomment-606924359

[neolit123]

thanks, @Arvinderpal would you be able to contribute any related changes to kubeadm for 1.19?

[Arvinderpal]

@neolit123 I have not been following the recent developments in dual-stack; however, if there are specific kubeadm changes needed, I can certainly make the time to help.

[neolit123]

@Arvinderpal thanks. hopefully you will have the time if the feature moves forward as users will have the expectation that kubeadm supports the latest incarnation. otherwise we have to explain the discrepancy to them.

[fabriziopandini]

@aojea are you planning to get dual-stack support in kind?

[aojea]

@aojea are you planning to get dual-stack support in kind?

absolutely, I have also a PR working https://github.com/kubernetes-sigs/kind/pull/692, but we've decided to hold until the dual-stack feature is more stable. There are still lots of things to solve in the dual-stack area ...

[neolit123]

help-wanted with resolving this item:

don't hardcode the ipv4 mask size to 24?

@sidharthsurana recently logged the same issue here:
https://github.com/kubernetes/kubeadm/issues/2327

[aojea]

I'll take it @neolit123

[aojea]

so,

e2e tests for service-cluster-ip-range

These are being tested implicitly with the presubmit and periodic jobs https://testgrid.k8s.io/sig-network-kind#sig-network-kind,%20dual,%20master

Add documentation on enabling dual-stack via kubadm

This seems the only part missing

Add apiserver-advertise-address support for dual-stack

Sig network started to discuss it recently ... that is not going to be resolved soon and is not clear how to move forward

[aojea]

/assign
so I don´t forget,
I will be the backup if nobody picks it

[pacoxu]

/assign

so I don´t forget,

I will be the backup if nobody picks it

May I pick it，😄 @aojea ？

I have done a similar thing before on dual-stack 1.18 which is a little different.

[aojea]

/unassign
/assign @pacoxu
Appreciate it,
Thanks

[pacoxu]

Action Items in 1.21：

• [x] https://github.com/kubernetes/kubernetes/pull/99294: feature gate alpha to beta cmd/kubeadm/app/features/features.go, it need wait for pkg/features/kube_features.go change.
• [x] https://github.com/kubernetes/website/pull/26675 Documentation: Add documentation on enabling dual-stack via kubadm.
• [ ] (will happen after 1.21) Add apiserver-advertise-address support for dual-stack

Is there anything that I miss?

[neolit123]

that's it. my understanding is that apiserver-advertise-address is not going to happen in 1.21.

[pacoxu]

@neolit123 @aojea
Should we support multi node IPs in kubeadm init and join?

• auto-detect or with a flag --node-ips
• if dual-stack is true, detect both; if single-stack, prefer ipv4 or detect from pod-cidr first.

> --node-ip string IP address (or comma-separated dual-stack IP addresses) of the node. If unset, kubelet will use the node's default IPv4 address, if any, or its default IPv6 address if it has no IPv4 addresses. You can pass '::' to make it prefer the default IPv6 address rather than the default IPv4 address.

[aojea]

great catch, I think so, or your hostnetwork pods will be single-stack, that can be a problem ...

[neolit123]

well, given kubeadm already supports passing kubelet config / flags we should just document how users can do that (KubeletConfiguration == preferred).
i don't think we should expose this is a wrapper in the kubeadm config or kubeadm flags.

arguably the kubeadm flag --node-name should not exist for similar reasons, but it's already there.

[aojea]

documentation is fine, I didn't want to mean to create a new option

[pacoxu]

@neolit123 node-ips auto detect should be a kubelet feature in my mind.
Currently, we can add it to the docs for reminding users.

[aojea]

@neolit123 node-ips auto detect should be a kubelet feature in my mind.
Currently, we can add it to the docs for reminding users.

the user has to specify the IP addresses of each family, separated by comma, there is no autodetect for dual-stack