Enhancements: Kubeadm for Windows

[ksubrmnn]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubeadm support for Windows
• Kubernetes Enhancement Proposal: 20190424-kubeadm-for-windows
• Primary contact (assignee): @benmoss
• Responsible SIGs:
  /sig-windows
  /sig-cluster-lifecycle
• Enhancement target (which target equals to which milestone):
  
  
  
  • Alpha release target 1.16
  
  
  • Beta release target: 1.18
  
  
  • GA release target: TODO
  
  

[neolit123]

/sig windows
/sig cluster-lifecycle

[craiglpeters]

@ksubrmnn, Enhancement Freeze for Kubernetes 1.15 has passed and this did not meet the deadline. This is now being removed from the 1.15 milestone and the tracking sheet. If there is a need for this to be in 1.15, please file an Enhancement Exception. Thank you.

[craiglpeters]

/milestone clear

[craiglpeters]

Hi @ksubrmnn . Code Freeze is Thursday, May 30th 2019 @ EOD PST. All enhancements going into the release must be code-complete, including tests, and have docs PRs open.

Please list all current k/k PRs so they can be tracked going into freeze. If the PRs aren't merged by freeze, this feature will slip for the 1.15 release cycle. Only release-blocking issues and PRs will be allowed in the milestone.

If you know this will slip, please reply back and let us know. Thanks!

[michmike]

@craiglpeters according to the KEP for kubeadm for 1.15, we were not going to add any more tests beyond unit tests. are we good on this area? thanks!
cc: @ksubrmnn

[makoscafee]

Hey, @michmike @ksubrmnn 👋 I'm the v1.15 docs Lead.
Does this enhancement require any new docs (or modifications)?

Sorry for getting to you late, Code Freeze is Thursday, May 30th 2019 @ EOD PST It would be great to have a placeholder PR by then can be just empty PR with title and description. Let me know if you have any questions!

[craiglpeters]

@michmike and @ksubrmnn yes agreed that unit tests are good for alpha. I still need to know which k/k PRs to track for this issue. It's now being marked as At Risk in the 1.15 Enhancement Tracking Sheet. Is there high confidence these will be merged by EOD PST today? After this point, only release-blocking issues and PRs will be allowed in the milestone with an exception.

[ksubrmnn]

@craiglpeters See https://github.com/kubernetes/kubeadm/issues/1393

We are still on track for alpha. I will fix up https://github.com/kubernetes/kubernetes/pull/78324 and it will get merged today.

[ksubrmnn]

Docs PR open: https://github.com/kubernetes/website/pull/14644

[michmike]

@craiglpeters we will continue working on this. if we can get an exception, it will make it into 1.15 and if not it will come later. the code changes are good, but there is still lots of documentation work left.

[PatrickLang]

also related https://github.com/kubernetes/kubernetes/pull/78189

[craiglpeters]

Hi @ksubrmnn, tomorrow is code freeze (one day delay) for the 1.15 release cycle. The k/k PRs have not yet been merged. It's now being marked as At Risk in the 1.15 Enhancement Tracking Sheet. Is there high confidence these will be merged by EOD PST tomorrow? After this point, only release-blocking issues and PRs will be allowed in the milestone with an exception.

[kacole2]

/milestone clear

[kacole2]

Hi @PatrickLang @michmike , I'm the 1.16 Enhancement Lead. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[michmike]

hi @kacole2 , yes this is tracked for an alpha release with 1.16. thanks!

[michmike]

@ksubrmnn can link the PRs when we have them (likely will create some this week). the KEP is already approved for this work

[sethmccombs]

Hey @ksubrmnn @michmike

I'm one of the v1.16 docs shadows.
Does this enhancement (or the work planned for v1.16) require any new docs (or modifications to existing docs)? If not, can you please update the 1.16 Enhancement Tracker Sheet (or let me know and I’ll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday, August 23rd, it can just be a placeholder PR at this time. Let me know if you have any questions!

[ksubrmnn]

Hey @ksubrmnn @michmike
I'm one of the v1.16 docs shadows.
Does this enhancement (or the work planned for v1.16) require any new docs (or modifications to existing docs)? If not, can you please update the 1.16 Enhancement Tracker Sheet (or let me know and I’ll do so)
If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday, August 23rd, it can just be a placeholder PR at this time. Let me know if you have any questions!

@daschott

[daschott]

@sethmccombs we would like to modify the page on how to join Windows nodes in the Windows user guide but the exact content of that page have yet to be confirmed 100%. Is it OK if we raise a placeholder PR for now?

EDIT: Updated URL. We may also want to add a very short page titled something along the lines of "Joining Windows nodes" to the kubeadm section containing a pointer to the instructions in the Windows user guide.

[daschott]

@sethmccombs I raised https://github.com/kubernetes/website/pull/15750 as a placeholder PR for the doc updates.

[kacole2]

@daschott @michmike @PatrickLang @neolit123 code freeze for 1.16 is on Thursday 8/29. It looks like everything on https://github.com/kubernetes/kubeadm/issues/1393 has been merged. So if this is on track we are good to go! If there are any other outstanding k/k PRs please update the issue.

[michmike]

hi @kacole2 , we are on track for 1.16. thank you!

[mrbobbytables]

Hey there @ksubrmnn @michmike -- 1.17 Enhancements lead here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

/milestone clear

[michmike]

@ksubrmnn were you able to get some time commitments to push this to beta for 1.17? thanks

[ksubrmnn]

@neolit123

From the following link, it looks like kubeadm upgrade for nodes will just fetch the upgraded config map from the control plane. Then the scripts will have to update kubelet and kube proxy and restart them.

Is that correct? I'm trying to get a time estimate for this work, but it seems like it's possible that this may just work already.

[ksubrmnn]

@PatrickLang @adelina-t For Beta, we need to have testing that uses Kubeadm so that we can have a good signal. We should discuss the plan for this further

[neolit123]

@ksubrmnn i think it should work already on the kubeadm side, you can give it a try.
but yes, a wrapper script would need to execute kubeadm upgrade node then update the kube-proxy / kubelet binaries.

[mrbobbytables]

@PatrickLang @ksubrmnn -- just checking back in...did you want to track this for 1.17? Enhancements freeze is just around the corner 😬 (Tuesday, October 15th, 2019).

[PatrickLang]

@mrbobbytables - yes, we have testing coming online and hope to be able to meet the minor updates needed for 1.17. Can you add to tracking sheet?

[mrbobbytables]

@PatrickLang sliding in at the last minute. XD

If you could, the KEP and parent issue still all reference as targeting alpha in 1.16, that should be bumped for 1.17.

/milestone v1.17

[pgburt]

Hello @ksubrmnn @PatrickLang , I'm one of the v1.17 docs shadows.
Does this enhancement (or the work planned for v1.17) require any new docs (or modifications to existing docs)? If not, please update the 1.17 Enhancement Tracker Sheet (or let me know and I'll do so).

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.17) due by Friday, November 8th, it can just be a placeholder PR at this time. Let me know if you have any questions!

[pgburt]

Hello @ksubrmnn @PatrickLang . Just following up on the docs PR. We'd love to get the Enhancement Tracking sheet updated with a link to the placeholder docs PR before the upcoming due date.

Let me know if I can help with any of that.

[jeremyrickard]

Hello @ksubrmnn @PatrickLang,

Jeremy from the 1.17 enhancements team here 👋

We're closing in on the 1.17 Code Freeze on Nov 14th. Will this still landing in 1.17? We don't currently have any PRs associated with this in our tracking list, so if you could please share any PRs reflecting this work, I'll get them added!

[jeremyrickard]

Hey @PatrickLang any update on this? Looking for relevant PRs to track, as well as a draft docs PR before the end of the day today, otherwise this will need to bump from the 1.17 release and you'll need to obtain an exception. Please let us know!

[jeremyrickard]

/milestone clear

[jeremyrickard]

Hey @neolit123 @ksubrmnn @PatrickLang, the deadline for opening a draft docs PR has passed, so we'll unfortunately need to remove this from the 1.17 release. In order to have this added back to the release, you'll need to obtain an exception from the release lead, @guineveresaenger.

[PatrickLang]

We discussed yesterday, things are still in-the-works to be able to support kubeadm upgrade which was a beta criteria. I don't think it will be done this week.

[jeremyrickard]

Hey there @ksubrmnn @michmike @PatrickLang -- 1.18 Enhancements lead here. I wanted to check in and see if you think this Enhancement will be graduating to beta in 1.18?

The current release schedule is:

Monday, January 6th - Release Cycle Begins
Tuesday, January 28th EOD PST - Enhancements Freeze
Thursday, March 5th, EOD PST - Code Freeze
Monday, March 16th - Docs must be completed and reviewed
Tuesday, March 24th - Kubernetes 1.18.0 Released
If you do, please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

[michmike]

yes, this should make it to beta with 1.18. thanks!
cc: @benmoss and @neolit123

[jeremyrickard]

/milestone v1.18

[neolit123]

@jeremyrickard the KEP update is about to merge for beta graduation in 1.18.
https://github.com/kubernetes/enhancements/pull/1456

[savitharaghunathan]

Hi, @benmoss @ksubrmnn @michmike @PatrickLang :)

1.18 docs shadow here. Does this enhancement require any new docs (or modifications to existing docs)? If not, can you please update the 1.18 Enhancement Tracker Sheet (or let me know and I'll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.18) due by Friday, Feb 28th, it can just be a placeholder PR at this time. Let me know if you have any questions!

Thanks,
Savitha

[neolit123]

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.18) due by Friday, Feb 28th, it can just be a placeholder PR at this time. Let me know if you have any questions!

thanks for the heads up. docs updates will be required.

[jeremyrickard]

Hey @neolit123,

Just a friendly reminder that code freeze for 1.18 is March 05, 2020.

As we track toward code freeze, please list out/link to any PRs you are working on toward graduating this enhancement!

[neolit123]

ack
@gab-satchi @benmoss

[savitharaghunathan]

Hi @neolit123 @benmoss @gab-satchi - I'm one of v1.18 shadow, since we're approaching the deadline for docs placeholder PR, Can you create a placeholder PR and link it here.

Thanks!

[neolit123]

@savitharaghunathan

Can you create a placeholder PR and link it here.

this is the docs PR:
https://github.com/kubernetes/website/pull/19217

@jeremyrickard

As we track toward code freeze, please list out/link to any PRs you are working on toward graduating this enhancement!

code PRs are tracked here:
https://github.com/kubernetes/kubeadm/issues/1393

not all checkmarks need to be checked for the feature to graduate to beta.

[jeremyrickard]

@neolit123 thanks for the pointer to the tracking issue. Could you share or annotate which ones definitely need to be checked to graduate to beta?

[neolit123]

i do not think we have all items in there yet.
deferring your question to @benmoss and @gab-satchi

[benmoss]

I think that issue isn't 100% accurate anymore in terms of the unresolved issues there. From the revised KEP I think the remaining work is

• [ ] PR the remaining scripts to sig-windows-tools
• [ ] set up e2e tests
• [ ] finalize the documentation

[neolit123]

i'm going to add the above items to https://github.com/kubernetes/kubeadm/issues/1393
under a BETA section. thanks.

EDIT: done organized the work items better.

[jeremyrickard]

@neolit123 @benmoss @gab-satchi

We're only a few days from code freeze. It looks like you still have a few items open. Do you think that these will merge before the code freeze on March 5th or do you think this should defer to 1.19?

Thanks!

[neolit123]

@jeremyrickard

i will provide a summary:

the crucial PR merged:
https://github.com/kubernetes-sigs/sig-windows-tools/pull/34

this one is a nice-to-have:
https://github.com/kubernetes/kubernetes/pull/88287

docs are WIP:
https://github.com/kubernetes/website/pull/19217

e2e are TBD

i think we are still good for 1.18.

[neolit123]

status update:

• the main body of work is in place: https://github.com/kubernetes-sigs/sig-windows-tools/pull/34
• test signal is in place: https://k8s-testgrid.appspot.com/sig-windows#kubeadm-windows-gcp-k8s-stable
  we are a tracking some failures caused by infrastructure issues (bad test image)
• docs merged: https://github.com/kubernetes/website/pull/19217

this work covered the beta graduation requirements.

[michmike]

excellent. thank you @benmoss , @neolit123 and @gab-satchi for all the work on this!

[palnabarun]

/milestone clear

(removing this from the v1.18 milestone as that is complete)

[msedzins]

Hey there @benmoss , @neolit123 -- 1.19 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating in 1.19?

In order to have this part of the release:

1. The KEP PR must be merged in an implementable state
2. The KEP must have test plans
3. The KEP must have graduation criteria.

The current release schedule is:

• Monday, April 13: Week 1 - Release cycle begins
• Tuesday, May 19: Week 6 - Enhancements Freeze
• Thursday, June 25: Week 11 - Code Freeze
• Thursday, July 9: Week 14 - Docs must be completed and reviewed
• Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released

If you do, I'll add it to the 1.19 tracking sheet (http://bit.ly/k8s-1-19-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[benmoss]

No I don't think will be graduating in 1.19

[msedzins]

Thank you @benmoss for letting me know

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/remove-lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[neolit123]

/remove-lifecycle rotten

[kikisdeliveryservice]

Hi @benmoss

Enhancements Lead here. Any plans for this in 1.20?

Thanks!
Kirsten

[neolit123]

AFAIK, no. Once we get privileged container support we can revisit. Perhaps
in 1.21.

[marosset]

@neolit123 is correct, we'll revisit this after Windows gains support for privileged containers (which we hope will be in 1.20 for alpha)

[kikisdeliveryservice]

Thanks for the fast update @marosset & @neolit123 ! Much appreciated!!