Enhancements: In-tree storage plugin to CSI Driver Migration

[davidz627]

Feature Description

• One-line feature description (can be used as a release note): Migrate the internals of the in-tree plugins to call out to CSI Plugins while maintaining the original API
• Primary contact (assignee): @davidz627
• Responsible SIGs: sig-storage
• Design proposal link (community repo): https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/csi-migration.md
• KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190129-csi-migration.md
• PRs:
  
  
  
  • [x] https://github.com/kubernetes/kubernetes/pull/69830
  
  
• Link to e2e and/or unit tests:
  
  
  
  • GCE: https://testgrid.k8s.io/sig-gcp-compute-persistent-disk-csi-driver#Migration%20Kubernetes%20Master%20Driver%20Latest
  
  
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @saad-ali @jsafrane
• Approver (likely from SIG/area to which feature belongs): @saad-ali
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release: 1.13
  
  
  • Beta release: 1.17
  
  
  • Stable release target (x.y): 1.19
  
  

[davidz627]

/sig storage
/cc @ddebroy

[kacole2]

@davidz627 this release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement for 1.13 if there is a high level of confidence it will meet the following deadlines:
Docs (open placeholder PRs): 11/8
Code Slush: 11/9
Code Freeze Begins: 11/15
Docs Complete and Reviewed: 11/27

Do you feel confident it will make these? If so we can include it for 1.13 tracking. Thanks!

[saad-ali]

@kacole2 This is multi quarter work with other components depending on it so we want to get it started ASAP (this quarter). That said as we approach end of the month @davidz627 and @ddebroy will make a call about if it is feasible to ship something this quarter or not. If not, we will remove it from the 1.13 milestone. For now let's track it.

[tfogo]

Hi @davidz627 and @ddebroy, do you think we will be able to land this in 1.13? Please keep in mind this release has an aggressive timeline and we should only include it if you're confident it will meet the deadlines @kacole2 outlined above. Thanks!

[tfogo]

Also, I'm assuming this enhancement won't need any documentation updates on k/website since it's an internal change. Please let me know if we will need documentation. Thanks!

[davidz627]

@tfogo no documentation required. It's looking like we wont hit our 1.13 target and will target 1.14 instead.
@saad-ali

[AishSundar]

@davidz627 and @saad-ali is it safe to move this out to 1.14? if so I can do that.

[AishSundar]

Synced with @davidz627 offline. This is now punted to 1.14. No revert / rollback needed in k/k because feature is being developed on a separate branch to enable atomic commit for the feature when it's ready

@kacole2 @tfogo

[claurence]

@davidz627 Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP - I don't see a KEP for this issue can you please drop a link for the KEP.

[davidz627]

@claurence Work on this started before the KEP process started so we have a design in the "old" style.
https://github.com/kubernetes/community/blob/b7daf2fdfa7c4984d9236336b4f1ccc9f048db98/contributors/design-proposals/storage/csi-migration.md
We have a significant amount of work planned for the 1.14 release including but not limited to:
https://github.com/kubernetes/kubernetes/pull/71901 <- PR will be broken up into smaller PR's soon
https://github.com/kubernetes/kubernetes/pull/70909
https://github.com/kubernetes/kubernetes/pull/72950

As well as some work that has already merged:
https://github.com/kubernetes/kubernetes/pull/72770

[claurence]

@davidz627 - apologies for getting back to this so late - thanks for linking the original design proposal doc - we do want all issues to have a KEP for 1.14 - even if it's just pointing to the design proposal doc. And that it spells out graduation criteria and testing plan.

[saad-ali]

KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/1234-20190129-csi-migration.md

[dims]

Looks like it got renamed Saad - https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190129-csi-migration.md

[spiffxp]

Adding tracked/yes label to reconcile against https://bit.ly/k8s114-enhancements
FYI @claurence @lachie83 @lledru @ameukam

[makoscafee]

Hi, @davidz627 👋 I'm the v1.14 docs shadow.

Does this enhancement require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.14) due by Friday, March 1. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[makoscafee]

@tfogo no documentation required. It's looking like we wont hit our 1.13 target and will target 1.14 instead.
@saad-ali

Sorry! From this i assume it wont need any documentation still, right ?

[thockin]

Does this need an API review?

[dims]

@thockin it's a simple translation and does not change API in any way.

@ddebroy please confirm

[liggitt]

I thought this was the item the CSIDriver/CSINodeInfo APIs were being added under. Are those objects added under https://github.com/kubernetes/enhancements/issues/557 instead?

[dims]

@liggitt the PR for "Make CSINodeInfo and CSIDriver Core APIs"? that would be https://github.com/kubernetes/kubernetes/pull/73883

which points to https://github.com/kubernetes/enhancements/issues/770 and https://github.com/kubernetes/enhancements/issues/603

[liggitt]

@liggitt the PR for "Make CSINodeInfo and CSIDriver Core APIs"? that would be kubernetes/kubernetes#73883

which points to #770 and #603

thanks, updated https://github.com/kubernetes-sigs/architecture-tracking/projects/3 with those and removed this

[msau42]

The API changes for #557 #603 #770 and this are all covered by kubernetes/kubernetes#73883

[claurence]

@davidz627 I see a few open PRs mentioned above ^^ are any of these blocking for the 1.14 release? Or are all PRs for 1.14 merged?

[ddebroy]

We do need kubernetes/kubernetes#74926 reviewed and merged. /cc @msau42

[davidz627]

also:
https://github.com/kubernetes/kubernetes/pull/74835
https://github.com/kubernetes/kubernetes/pull/74858
https://github.com/kubernetes/kubernetes/pull/73840

[kacole2]

Hello, I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

/milestone clear

[msau42]

The plan is to move to beta in 1.15.

[kacole2]

/milestone v1.15
/stage beta

[davidz627]

Our project board can be accessed here for beta milestone tracking: https://github.com/orgs/kubernetes-csi/projects/13

[justaugustus]

/assign @davidz627

[daminisatya]

Hey, @davidz627 I'm the v1.15 docs release shadow.

I see that you are targeting the beta version of this enhancement for the 1.15 release. Does this require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions! 😄

[ddebroy]

The current plan for the CSI Migration feature is to stay Alpha during 1.15 since the feature is depending on alpha level APIs bring introduced in 1.15.

[daminisatya]

Thank you @ddebroy So for now, there no requirement for documentation?

[daminisatya]

Ok, I just went through the history and it is mentioned that there is no requirement for docs since it is an internal change.

[ddebroy]

Correct.

[kacole2]

Hi @ddebroy @msau42. Code Freeze is Thursday, May 30th 2019 @ EOD PST. All enhancements going into the release must be code-complete, including tests, and have docs PRs open.

Assuming we are still on track and this board should have all issues moved to Done by Thursday:
https://github.com/orgs/kubernetes-csi/projects/13

If you know this will slip, please reply back and let us know. Thanks!

[davidz627]

Hi @kacole2, we are remaining in Alpha this release and that project board contains all the items we wanted to get done by Beta. Could you confirm what you would need from us to remain in Alpha this release?

[kacole2]

@davidz627 if you are remaining in Alpha and this will not graduate, then that's perfectly acceptable. I just need to know that for tracking purposes. We will also want to change the label to stage/alpha and remove 1.15 from the Beta target in the original post.

[davidz627]

@kacole2 I am confirming that we will remain in Alpha for k8s v1.15, thanks!

[kacole2]

Hi @davidz627 , I'm the 1.16 Enhancement Lead. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[davidz627]

@kacole2 This feature will be graduating to Beta in 1.16.
We are tracking progress here and will keep it updated going forward: https://github.com/orgs/kubernetes-csi/projects/13

[kacole2]

/milestone v1.16
/stage beta

[VineethReddy02]

Hi @davidz62, I'm the v1.16 docs release shadow.

Does this enhancement require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday,August 23rd. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

Thanks!

[davidz627]

These docs are still fairly accurate: https://github.com/kubernetes/website/pull/12935/files
however, it looks like we'll need to update the feature state to 'beta', I can do that.

[davidz627]

We're bringing Migration to Beta, as well as MigrationGCE to Beta.

I think the other possibility is MigrationAWS to Beta but we're not sure on that yet, so did not include it in the same docs PR. I will sync with @leakingtapan about AWS status.

https://github.com/kubernetes/website/pull/15629

[leakingtapan]

ack. Will post the decision here once I figure out.

[leakingtapan]

Confirming that we are bring MigrationAWS to beta as well.

[leakingtapan]

/cc @wongma7

[kacole2]

@davidz627 @leakingtapan @wongma7 I see that https://github.com/kubernetes/kubernetes/pull/78523 has been merged. Code freeze for 1.16 is on Thursday 8/29. Are there any outstanding k/k PRs that still need to be merged for this to go Beta?

[davidz627]

@kacole2 I believe we're actually blocked on CSI Resize Beta which I believe is not happening this quarter. We will remain in Alpha this quarter unless that dependency is resolved.

/cc @msau42

[msau42]

@davidz627 @kacole2 CSI resize beta is on track for 1.16

[davidz627]

ack @kacole2 we're going to Beta!
Remaining PRs:
Flag flip PR: https://github.com/kubernetes/kubernetes/pull/81536
Docs PR: https://github.com/kubernetes/website/pull/15629

[kacole2]

@davidz627 it looks like https://github.com/kubernetes/kubernetes/pull/81536 didn't merge and it's not ready for 1.17. I'll take this off the tracking sheet for 1.16

[unixkartik01]

Hey Flock,
Can any one let me know exactly in which page below mentioned function script is available in kubernets.io .

Thanks in advance....

[mrbobbytables]

Hey there @davidz627 -- 1.17 Enhancements lead here. I wanted to check in and see if you think this Enhancement will be graduating to beta in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

EDIT: removed msau42 -- sorry for the unneeded ping.

[davidz627]

@mrbobbytables yes the plan is to graduate to beta in 1.17
The PR that would need to merge is:
https://github.com/kubernetes/kubernetes/pull/83098#issuecomment-535270409
And we need to resolve known issue:
https://github.com/kubernetes/kubernetes/issues/79043
https://github.com/kubernetes/kubernetes/issues/75698

[mrbobbytables]

Awesome, thanks for the update @davidz627 :) I'll add it to the tracking sheet!

[VineethReddy02]

@davidz627

I'm one of the v1.17 docs shadows.
Does this enhancement (or the work planned for v1.17) require any new docs (or modifications to existing docs)? If not, can you please update the 1.17 Enhancement Tracker Sheet (or let me know and I’ll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.17) due by Friday, Nov 8th, it can just be a placeholder PR at this time. Let me know if you have any questions!

Thanks!

[VineethReddy02]

Hey, @davidz627 Just a friendly reminder we're looking for a PR against k/website (branch dev-1.17) due by Friday, Nov 8th. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!
We are 5 days away from the deadline. Thanks!

[jeremyrickard]

Hey @davidz627,

Jeremy from the 1.17 enhancements team here. We're quickly approaching code freeze on Nov 14th, so I wanted to reach out and see how things are progressing. Are you still looking to land this in 1.17?

Thanks in advance!

[davidz627]

Hi @VineethReddy02 @jeremyrickard thanks for the reminders! We're still targeting Beta for 1.17 and I will create a placeholder docs PR soon.
We're tracking resolution of:
https://github.com/kubernetes/kubernetes/issues/84668
https://github.com/kubernetes/kubernetes/issues/84736
https://github.com/kubernetes/kubernetes/pull/83098

[jeremyrickard]

👍

[VineethReddy02]

Here is the placeholder PR for doc related changes on this enhancement.
https://github.com/kubernetes/website/pull/17478
Thanks! @davidz627

[leakingtapan]

Also this one for EBS CSI migration: https://github.com/kubernetes/website/pull/16126

[davidz627]

This one for the CSIMigrationInTreeOff flag: https://github.com/kubernetes/website/pull/17533

[jeremyrickard]

Hi @davidz627 tomorrow, 14.11.2019, is code freeze for the 1.17 release cycle. It looks like the k/k PRs you mentioned above have not yet been merged. We're flagging as At Risk in the 1.17 Enhancement Tracking Sheet.
Do you think they will be merged by the EoD of the 14th (Thursday)? After that point, only release-blocking issues and PRs will be allowed in the milestone with an exception.

[davidz627]

We are working on it and aim to have everything merged by EOD today (Wednesday). Thanks for flagging. We are so close that we will file for an exception if necessary.

[davidz627]

PRs are in various states - they have all been approved + lgtm-ed but need to go in sequentially (probably some merge conflicts). The biggest one https://github.com/kubernetes/kubernetes/pull/83098 re-testing before going in merge queue

[jeremyrickard]

Hey @davidz627, thanks for all the work getting things across the line. I see the things you called out above were taken care of, including https://github.com/kubernetes/kubernetes/pull/83098...but was https://github.com/kubernetes/kubernetes/pull/85231 needed for this? It looks like that hasn't merged, I see it referenced above.

[jeremyrickard]

Hey @davidz627, Code freeze is now in effect for the 1.17 release. It doesn't look like the https://github.com/kubernetes/kubernetes/pull/85231 was added to a merge pool in time. I'm going to go ahead and remove this from the milestone, if you feel it is release blocking or urgent that it gets merged, please file an exception request

[jeremyrickard]

/milestone clear

[jeremyrickard]

If you'd like to file an exception, please see here

[davidz627]

Hi @jeremyeder, The PR https://github.com/kubernetes/kubernetes/pull/85231 was LGTM-ed and Approved and in the Milestone by 5:34PM PST 2019-11-14. However, it continued to get kicked out of merge pool because of merge conflicts and unrelated test issues (pull-kubernetes-node-e2e-containerd has been failing consistently since yesterday). Unfortunately, I had to sleep around 1am PST so couldn't fix the merge conflict that appeared sometime in the night. The PR is now back in a mergable state and can be added back to the milestone. Thanks @jeremyrickard!

[jeremyrickard]

Hey @davidz627, since the PR didn't land in the merge before the code freeze occured, we'll still need to get an exception on this one so if you could submit one, we can work on getting this in.

[jeremyrickard]

Per @guineveresaenger this has been granted an exception for code freeze. Thanks for submitting @davidz627

/milestone v1.17

[draveness]

Hi @davidz627, do we have the plan to graduate this feature to GA In 1.18? We could remove some predicates in the scheduler after graduation. https://github.com/kubernetes/kubernetes/issues/82375

[davidz627]

@draveness unfortunately we do not intend to graduate this feature to GA in 1.18 at this time. Our current target is going to be 1.19.

[jeremyrickard]

Thanks for the update on graduation @davidz627. I'll mark it defered to 1.19.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[msau42]

/remove-lifecycle stale

[palnabarun]

/milestone clear

(removing this issue from v1.17 milestone as the milestone is complete)

[johnbelamaric]

@msau42 @davidz627 Release 1.19 enhancements shadow here. Any plans for this in 1.19?

[msau42]

No plans to graduate in 1.19. I think next steps are to have each of the plugins do further testing and fixes, as well as moving towards deploying their csi driver along with the respective cloud controller manager, so that we can eventually flip the feature on by default.

[johnbelamaric]

Ok, so no major changes either then that need to be tracked? That is, are you expecting some of that work to move forward, and should be track the associated PRs to make sure they are on track for inclusion in 1.19? If so, let me know.

_Enhancements freeze is in one week, on May 19._

Note also that recently the KEP format has changed, and #1620 merged recently, adding production readiness review questions to the KEP template. Please take your soonest opportunity to reformat your KEP and also answer the questions add to the template in that PR.

Thanks!

[msau42]

There are no known issues that need to be tracked. As each driver does more testing and potentially find issues, then we will track them.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[kikisdeliveryservice]

Hi @msau42

Enhancements Lead here. Are there any plans to graduate this to stable in 1.20?

Thanks!
Kirsten

[msau42]

We plan to stay in beta for 1.20. There are 3 known outstanding issues we need to resolve before moving to GA:

• Implement dangling attach in CSI: https://github.com/kubernetes/kubernetes/issues/80488
• Implement fsgroup during mount: https://github.com/kubernetes/enhancements/issues/695
• Testing on Windows: https://github.com/kubernetes/kubernetes/issues/94778

@kubernetes/sig-storage-feature-requests

[msau42]

/remove lifecycle/stale

[msau42]

/remove-lifecycle stale

[kikisdeliveryservice]

Thanks for the update @msau42 ! I'll update the tracking sheet. :+1: