Enhancements: Support Out-of-Tree Azure Cloud Provider

[andrewsykim]

Enhancement Description

• One-line enhancement description (can be used as a release note):
  Support Out-of-Tree Azure Cloud Provider by running the cloud-controller-manager
• Primary contact (assignee):
  @lachie83 @justaugustus @khenidak
• Responsible SIGs:
  SIG Azure, SIG Cloud Provider
• KEP: Supporting Out-of-Tree Azure Cloud Provider
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  @justaugustus @khenidak @andrewsykim
• Approver (likely from SIG/area to which enhancement belongs):
  @justaugustus @khenidak @andrewsykim
• Enhancement target (which target equals to which milestone):
  
  
  
  • Alpha release target: 1.15
  
  
  • Beta release target: 1.20
  
  
  • Stable release target: 1.21
  
  

Ref https://github.com/kubernetes/enhancements/issues/88

/sig azure cloud-provider

[justaugustus]

/assign @justaugustus @khenidak @dstrebel @feiskyer
/milestone 1.14
/stage alpha

[k8s-ci-robot]

@justaugustus: The provided milestone is not valid for this repository. Milestones in this repository: [v1.13, v1.14]

Use /milestone clear to clear the milestone.

In response to this:

/assign @justaugustus @khenidak @dstrebel @feiskyer
/milestone 1.14
/stage alpha

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[justaugustus]

(^^ what, is it my first day here?)
/milestone v1.14

[andrewsykim]

Forgot to ref https://github.com/kubernetes/kubernetes/issues/50752

[lachie83]

@andrewsykim @justaugustus can you please add me as an assignee as I'll run point on getting the KEP together

[justaugustus]

Done!
/assign @lachie83

[andrewsykim]

FYI put together the boiler plate for this KEP here https://github.com/kubernetes/enhancements/pull/735

[justaugustus]

Thanks @andrewsykim!
@lachie83 @feiskyer -- please use #735 as the template for the Azure KEP

[claurence]

@lachie83 @justaugustus @khenidak since the KEP for this issue hasn't been marked "implementable" yet we will be removing it from the 1.14 milestone. To have it added back in please file an exception - information on the exception process can be found here: https://github.com/kubernetes/sig-release/blob/master/releases/EXCEPTIONS.md

[kacole2]

I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. Please update the original post with the link to the KEP as well.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[feiskyer]

yep, this is on tracked.

/milestone v1.15

[kacole2]

Hi @feiskyer @andrewsykim we are going to start cleaning up some issues. If an enhancement doesn't touch any k/k code, then it no longer needs to be tracked within enhancements nor the release. I'm assuming the Azure cloud controller manager falls into this category. However, once the in-tree provider code is deprecated and needs to be removed, that will need to be tracked. Does this sound good to everyone?

@justaugustus @claurence @mrbobbytables @craiglpeters

[craiglpeters]

My thinking process is that because deprecation of some k/k depends upon this change, it makes sense to track it, but I'm open to being convinced otherwise. My concern is that if we don't track it, and the out of tree changes are released, it won't be easily discoverable to the Kubernetes community. Or is there some other place the out-of-tree changes will be broadcast?

[mrbobbytables]

My thinking process is that because deprecation of some k/k depends upon this change, it makes sense to track it, but I'm open to being convinced otherwise.

I am inclined to agree on tracking their removal, but I don't know if it falls within the general scope of alpha/beta/stable? If not as a KEP, there should be some large signaling that "Hey this thing is over here now'.

[justaugustus]

We should continue to track cloud provider extraction as it straddles the line and includes in-tree and out-of-tree code changes.

[kacole2]

@spiffxp had something to suggest on this as well. Perhaps we will bring it up again tomorrow in SIG-PM?

[feiskyer]

If an enhancement doesn't touch any k/k code, then it no longer needs to be tracked within enhancements nor the release.

Because AzureDisk/AzureFile and Azure credential provider are still not ready yet for out-of-tree, Azure cloud provider is still making changes in k/k repo.

[kacole2]

@@lachie83 @justaugustus @khenidak, Kubernetes 1.15 Enhancement Freeze is 4/30/2019. To be included in the Kubernetes 1.15 milestone, KEPs are required to be in an "Implementable" state with proper test plans and graduation criteria. Please submit any PRs needed to make this KEP adhere to inclusion criteria. If this will slip from the 1.15 milestone, please let us know so we can make appropriate tracking changes.

[kacole2]

@lachie83 @justaugustus @khenidak, Enhancement Freeze for Kubernetes 1.15 has passed and this did not meet the deadline. KEP is still in provisional state. This is now being removed from the 1.15 milestone and the tracking sheet. If there is a need for this to be in 1.15, please file an Enhancement Exception. Thank you

/milestone clear

[feiskyer]

The blocking issues are tracked now at https://github.com/kubernetes/cloud-provider-azure/issues/124.

[kacole2]

/milestone v.16

will be tracking the blocked issues to see if this will make it to code freeze.

[k8s-ci-robot]

@kacole2: The provided milestone is not valid for this repository. Milestones in this repository: [keps-beta, keps-ga, v1.15, v1.16]

Use /milestone clear to clear the milestone.

In response to this:

/milestone v.16

will be tracking the blocked issues to see if this will make it to code freeze.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[evillgenius75]

Hi @lachie83 @justaugustus @khenidak, I'm the 1.16 Enhancement Shadow. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

As a reminder, every enhancement requires a KEP in an implementable state with Graduation Criteria explaining each alpha/beta/stable stages requirements.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[kacole2]

@lachie83 @justaugustus @khenidak friendly reminder that this KEP is a Provisional state and needs to be Implementable by EOD tomorrow for Enhancement freeze of 1.16. Thanks

[feiskyer]

Pending on https://github.com/kubernetes/enhancements/pull/1158.

[kacole2]

@feiskyer @lachie83 @justaugustus @khenidak

Enhancement Freeze has passed for 1.16. The KEP at #1158 was never merged and now this is being removed from the 1.16 milestone. If this would like to be re-added, please file an exception and it will require approval from the release lead.

/milestone clear

[jeremyrickard]

Hey @feiskyer -- 1.17 Enhancements shadow here 👋 . I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, I'll add it to the 1.17 tracking sheet (https://bit.ly/k8s117-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍 For inclusion in 1.17, you'll need to have KEP https://github.com/kubernetes/enhancements/pull/1158 merged in an implementable state with both Graduation Criteria and a Test Plan

Thanks!

[feiskyer]

@jeremyrickard Thanks, we're planning to go Beta of this.

/milestone v1.17

[feiskyer]

@jeremyrickard The link https://bit.ly/k8s117-enhancements is 404, could you update the correct link here?

[jeremyrickard]

@feiskyer oops, correct link is: https://bit.ly/k8s117-enhancement-tracking

I'll update. Thanks! Just a reminder, the KEP needs to be updated to indicate that it is in an implementable state, it's marked provisional right now. Thanks again!

[jeremyrickard]

👋 Hey @feiskyer, just wanted to check in and remind you that the enhancements freeze will soon (October 15, EOD PST).

The KEP for this looks like it still needs to be merged. It's currently showing provisional vs implementable and is missing Graduation Criteria and a Test Plan. Those are requirements for inclusion in the 1.17 release.

[jeremyrickard]

Hey @feiskyer, Unfortunately the deadline for the 1.17 Enhancement freeze has passed and the KEP is still merged as implementable. For now this is being removed from the milestone and 1.17 tracking sheet. If there is a need to get this in, please file an enhancement exception.

[jeremyrickard]

/milestone clear

[kikisdeliveryservice]

Hey there @justaugustus @khenidak @dstrebel @feiskyer -- 1.18 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to [alpha|beta|stable] in 1.18?

The current release schedule is:
Monday, January 6th - Release Cycle Begins
Tuesday, January 28th EOD PST - Enhancements Freeze
Thursday, March 5th, EOD PST - Code Freeze
Monday, March 16th - Docs must be completed and reviewed
Tuesday, March 24th - Kubernetes 1.18.0 Released

To be included in the release, this enhancement must have a merged KEP in the implementable status. The KEP must also have graduation criteria and a Test Plan defined.
If you would like to include this enhancement, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍
We'll be tracking enhancements here: http://bit.ly/k8s-1-18-enhancements
Thanks!

[feiskyer]

@kikisdeliveryservice KEP https://github.com/kubernetes/enhancements/pull/1158 gets merged. we'd like to get the feature into beta in 1.18.0.

[kikisdeliveryservice]

thanks @feiskyer !

[kikisdeliveryservice]

/milestone v1.18

[kikisdeliveryservice]

Hi @feiskyer just as a reminder:

The attached KEP is in a provisional state and needs to be implementable by Enhancements freeze deadline.

[feiskyer]

Opened https://github.com/kubernetes/enhancements/pull/1466 to update the KEP to be implementable.

[kikisdeliveryservice]

great thanks, @feiskyer !!

[feiskyer]

Because the CSI drivers couldn't reach beta in 1.18 (see https://github.com/kubernetes/enhancements/issues/1490), we would move the beta to v1.19.

/milestone v1.19

cc @kikisdeliveryservice

[kikisdeliveryservice]

thanks for the update @feiskyer !

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[palnabarun]

Hi @feiskyer -- 1.19 Enhancements Lead here, I wanted to check in if you think this enhancement would graduate to Beta in 1.19?

---

The current release schedule is:

• Monday, April 13: Week 1 - Release cycle begins
• Tuesday, May 19: Week 6 - Enhancements Freeze
• Thursday, June 25: Week 11 - Code Freeze
• Thursday, July 9: Week 14 - Docs must be completed and reviewed
• Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released

[feiskyer]

@palnabarun Yep, it's on track for Beta in 1.19.

[palnabarun]

Thank you @feiskyer for the updates. I have updated the tracking sheet accordingly. :+1:

[annajung]

Hi @feiskyer 👋 1.19 docs shadow here! Does this enhancement work planned for 1.19 require new or modification to docs?

Friendly reminder that if new/modification to docs are required, a placeholder PR against k/website (branch dev-1.19) are needed by Friday, June 12.

[palnabarun]

@feiskyer -- Can you please link to all the implementation PR's here - k/k or otherwise?

---

The current release schedule is:

• ~Monday, April 13: Week 1 - Release cycle begins~
• ~Tuesday, May 19: Week 6 - Enhancements Freeze~
• Thursday, June 25: Week 11 - Code Freeze
• Thursday, July 9: Week 14 - Docs must be completed and reviewed
• Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released

[annajung]

Hi @feiskyer hope you're doing well, checking in again to see if docs are required for this or not. Could you confirm?

[feiskyer]

@annajung docs would be included in https://github.com/kubernetes-sigs/cloud-provider-azure repo, so we don't need any PRs to k/website repo.

[annajung]

Great, thank you! I'll update the tracking sheet accordingly

[palnabarun]

Can you please link to all the implementation PR's here - k/k or otherwise?

@feiskyer -- Pinging back as a friendly reminder for the above. :slightly_smiling_face:

[palnabarun]

Hi @feiskyer -- just wanted to check in about the progress of the enhancement. Can you please link all the implementation PR's here -- k/k or otherwise?

The release timeline has been revised recently, more details of which can be found here.

Please let me know if you have any questions. :slightly_smiling_face:

---

The revised release schedule is:

• Thursday, July 9th: Week 13 - Code Freeze
• Thursday, July 16th: Week 14 - Docs must be completed and reviewed
• Tuesday, August 25th: Week 20 - Kubernetes v1.19.0 released

[palnabarun]

Can you please link all the implementation PR's here -- k/k or otherwise?

Hi @feiskyer :wave:, pinging back as a reminder of the above. :slightly_smiling_face:

[feiskyer]

Hi, @palnabarun

Related PRs would be tracked by https://github.com/kubernetes-sigs/cloud-provider-azure/issues/342 and https://github.com/kubernetes-sigs/cloud-provider-azure/issues/344.

[palnabarun]

Hi @feiskyer :wave:, thank you for linking the issues here. I see that https://github.com/kubernetes/kubernetes/pull/91319 is yet to be merged and https://github.com/kubernetes-sigs/cloud-provider-azure/issues/342 has no corresponding PRs yet. Do you think they would be complete by the Code Freeze on Thursday, July 9th?

[palnabarun]

Hi @feiskyer :wave:, pinging back as a reminder of the above. :slightly_smiling_face:

[feiskyer]

@palnabarun I'm asking the original PR authors

[palnabarun]

@feiskyer -- Thank you for doing that. :slightly_smiling_face: Please note that tomorrow is Code Freeze.

If the enhancement misses the code freeze, you will need to file an Exception Request in order to get this back into 1.19.

---

Code Freeze begins on Thursday, July 9th EOD PST

[palnabarun]

@feiskyer @andrewsykim -- I see that the issue description here was changed to reflect v1.20 as Beta release target. Can you please confirm if we should drop the enhancement from v1.19 release cycle?

[palnabarun]

@feiskyer @andrewsykim -- We are into the Code Freeze now.

Since this enhancement still has pending work left (https://github.com/kubernetes/enhancements/issues/667#issuecomment-653919438) and I see that the issue description has been updated to set v1.20 as Beta release target, I am going ahead and removing this enhancement from the milestone.

You will need to file an Exception Request in order to get this back into 1.19.

---

Code Freeze begins on Thursday, July 9th EOD PST

[palnabarun]

/milestone clear

[feiskyer]

@palnabarun Thanks, the required dependencies are still not merged yet. let's move this to v1.20.
/milestone v1.20

[kikisdeliveryservice]

Hi @feiskyer !

Enhancements Lead here, do you still intend to target this for beta in 1.20?

Thanks!
Kirsten

[feiskyer]

@kikisdeliveryservice yep, of course.

[kikisdeliveryservice]

Hi @feiskyer

Enhancements freeze is on October 6th.
Looking at the KEP it seems to be missing some requirements for inclusion:

• alpha to beta graduation criteria, can you please clarify from what's listed here: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/azure/20190125-out-of-tree-azure.md#graduation-criteria
• testing plan: can you please clarify here what parts of the test plan are applicable to this stage/beta?

Also as a note, the KEP is using the old format that is missing several important parts such as the PRR, could you update using this template: https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template

Thanks,
Kirsten

[feiskyer]

@kikisdeliveryservice thanks for the TIP, would file a PR to address the issues.

Update: KEP updated by PR https://github.com/kubernetes/enhancements/pull/2028.

[kikisdeliveryservice]

Thanks @feiskyer the above PR merged looks good!

It is still using the old format, so feel free to update to the new template when you get a chance.

Moving this to tracked on the sheet.

Best,
Kirsten

[kikisdeliveryservice]

Hi @feiskyer

Since your Enhancement is scheduled to be in 1.20, please keep in mind the important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thanks!
Kirsten :smiley_cat:

[SomtochiAma]

Hello @lachie83 @justaugustus @khenidak, 1.20 Docs shadow here 👋🏽.
Does this enhancement work planned for 1.20 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.20 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Nov 6th

Also take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[feiskyer]

@SomtochiAma the docs would be updated at https://github.com/kubernetes-sigs/cloud-provider-azure

[SomtochiAma]

@feiskyer Does any modification need to be made to this website/docs? https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features

If not, then you are all good.

[feiskyer]

@feiskyer Does any modification need to be made to this website/docs? https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features

No required modification here.

[annajung]

Hi @feiskyer

Please keep in mind the important upcoming dates:

• ~Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline~ Docs Not Required
• Thursday, Nov 12th: Week 9 - Code Freeze
• ~Monday, Nov 23rd: Week 11 - Docs PR Ready for Review~ Docs Not Required

As a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

[kikisdeliveryservice]

Hi @feiskyer

Just a reminder that Code Freeze is coming up in 2 days on Thursday, November 12th. All PRs must be merged by that date, otherwise an Exception is required.

I don't see any PRs linked to this issue, can you confirm that your beta graduation criteria were met: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/azure/20190125-out-of-tree-azure.md#alpha---beta-graduation Was n e2e test added in testgrid, if so can you please link?

Thanks,
Kirsten

[feiskyer]

testgrid jobs have been added here: https://testgrid.k8s.io/provider-azure-cloud-provider-azure

[kikisdeliveryservice]

Perfect, thanks @feiskyer ! You're all set.