Enhancements: Snapshot / Restore Volume Support for Kubernetes (CRD + External Controller)

[jingxu97]

Feature Description

• One-line feature description (can be used as a release note):

Snapshot / restore functionality for Kubernetes and CSI. This provides standardized APIs design (CRDs) and adds PV snapshot/restore support for CSI volume drivers.

Old description:

Expose the ability in the Kubernetes API to create,list, delete, and restore snapshots from an arbitrary underlying storage systems that support it.

• Primary contact (assignee): @jingxu97, @xing-yang
• Responsible SIGs: storage-SIG
• Design proposal link (community repo):
  
  
  
  • Old: https://github.com/kubernetes/community/pull/2335
  
  
  • Old: https://docs.google.com/document/d/17WS4Wk4MXRH24i-BpMpIFo5F-SNoRkm_KtkBMZEEoAo/edit
  
  
• KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/177-volume-snapshot
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  
  
  
  • Luis Pabón (@lpabon)
  
  
  • Ritesh Shukla (@kerneltime)
  
  
  • Brad Childs (@childsb)
  
  
• Approver (likely from SIG/area to which feature belongs):
  
  
  
  • Saad Ali (@saad-ali)
  
  
• Feature target (which target equals to which milestone):
  
  
  
  • External repo (x.y): 1.8
  
  
  • Alpha release target (x.y): 1.12
  
  
  • Beta release target (x.y): 1.16
  
  

[davidopp]

kubernetes/kubernetes#39657
kubernetes/kubernetes#44172

[timothysc]

/cc @skriss

[mdelio]

@jingxu97 I think we're going to have something in alpha for 1.7, can we can we please set the milestone to 1.7?

[calebamiles]

@kubernetes/sig-storage-feature-requests could someone please update the issue description to the new template. Thanks!

[jingxu97]

I will work on it. Thanks!

Best,
Jing

On Thu, May 4, 2017 at 8:43 AM, caleb miles [e-mail]
wrote:

@kubernetes/sig-storage-feature-requests
https://github.com/orgs/kubernetes/teams/sig-storage-feature-requests
could someone please update the issue description to the new template.
Thanks!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/features/issues/177#issuecomment-299225105,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASSNxR4fUmmxnJ4QQc-Fe0K5CKT5tRO9ks5r2fIcgaJpZM4Lrsji
.

--

• Jing

[alkar]

Is this based on the proposal here: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/volume-snapshotting.md ?

[jingxu97]

No, the new proposal doc is here
https://docs.google.com/document/d/17WS4Wk4MXRH24i-BpMpIFo5F-SNoRkm_KtkBMZEEoAo
Please let me know if you cannot open it. Thanks!

Jing

On Tue, May 9, 2017 at 7:34 AM, Dimitrios Karagiannis <
[email protected]> wrote:

Is this based on the proposal here: https://github.com/kubernetes/
community/blob/master/contributors/design-proposals/volume-snapshotting.md
?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/features/issues/177#issuecomment-300184509,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASSNxVKLqOq9lsJ72MrdqkNN-2pbfmtIks5r4HmQgaJpZM4Lrsji
.

--

• Jing

[alkar]

@jingxu97 thanks! I can't access it but I just requested access.

[yanivlavi]

@jingxu97 I wanted to add an important note here regarding overall approach. I'm not this is the right place to put this, but I would be a happy to be guided to the right forum to bring this up.

Coming from a long background in oVirt and OSP I feel there is an important aspect that needs to be discuss which is ownership of the state of snapshots (and overall volume metadata) and discoverablility of the this metadata.

In cloud and on premise users (developers and admins) might prefer the native storage APIs over the Kube APIs. Kube is also not the one creating the snapshot, it is the cloud service or the storage itself, therefore it is not the owner of this metadata.

In OSP they have already made this mistake with Cinder that forces users that want to use snapshots to go through the Cinder API to have it be available in the OSP environment. Cinder doesn't know snapshot created on the storage and if you lose the Cinder you lose everything as the metadata that counts is in his stateful DB.

What I'm trying to say is that it is important that Kube doesn't try to be the owner of the volume metadata, which means things like periodically checking if a new snapshot was created directly via the storage service API or that the ID used for the snapshot is the storage service snapshot ID.

It is clear that Kube should expose snapshotting since it very needed to the container use case, but it is very important it doesn't become a storage abstraction service like Cinder. We do not want to chase the storage service features or limit user to use the Kube API, it should be a option. We also want to allow discovery of volumes no matter where they where created.

As we get to more complex features like QOS and oversubscription for example, we want to allow exposing and reusing the storage service capabilities, not replace them or block users from using them via the cloud service or storage management APIs.

[idvoretskyi]

@jingxu97 any progress on the feature description? @kubernetes/sig-storage-feature-requests

[idvoretskyi]

@mdelio @jingxu97 please, update the feature description with the new template - https://github.com/kubernetes/features/blob/master/ISSUE_TEMPLATE.md

[saad-ali]

@idvoretskyi I updated the original comment with the new template. This feature is not actually shipping any bits in the Kubernetes core for v1.7, so therefore I moved it to the next-milestone. This means that it will not need documentation, etc. for 1.7 in the Kubernetes core. I will remove the feature from the 1.7 tracking board as well.

[idvoretskyi]

@saad-ali any updates for 1.8? Is this feature still on track for the release?

[rootfs]

cc @tsmetana

[childsb]

@idvoretskyi this is still on track: https://github.com/kubernetes-incubator/external-storage/pull/331

[jdumars]

@jingxu97 @rootfs any update on missing docs for this? PR is due today.

[jingxu97]

@jdumars We put this into external, is there still a deadline for it?

[jdumars]

This is an independent feature outside of the specific 1.8 milestone.

[jdumars]

Docs can wait until you release separately @jingxu97 - thanks for letting us know

[saad-ali]

This did not make it in to 1.9. Punting to 1.10

[idvoretskyi]

@saad-ali @kubernetes/sig-storage-feature-requests any progress for 1.10?

[jingxu97]

We are working on a proposal to push it in tree. Will update the status soon.

[justaugustus]

@jingxu97 @saad-ali @kubernetes/sig-storage-feature-requests Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

cc @idvoretskyi

[dhawal55]

Any updates?? Any plans to target this for v1.12??

[justaugustus]

@saad-ali -- Thanks for the update! This has been added to the 1.12 Tracking sheet.

Just to clarify, https://github.com/kubernetes/features/issues/543 mentions the following timeline:

Alpha release target (x.y): 1.12
Beta release target (x.y): 1.13
Stable release target (x.y): 1.14

Can you confirm that?

[justaugustus]

/assign @xing-yang

[k8s-ci-robot]

@justaugustus: GitHub didn't allow me to assign the following users: xing-yang.

Note that only kubernetes members and repo collaborators can be assigned.
For more information please see the contributor guide

In response to this:

/assign @xing-yang

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[xing-yang]

@justaugustus "Alpha release target 1.12..." is correct. The prototype snapshot support in "External repo" was added in 1.8.

[zparnold]

Hey there! @jingxu97 I'm the wrangler for the Docs this release. Is there any chance I could have you open up a docs PR against the release-1.12 branch as a placeholder? That gives us more confidence in the feature shipping in this release and gives me something to work with when we start doing reviews/edits. Thanks! If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it?

[xing-yang]

@zparnold Added a placeholder doc PR https://github.com/kubernetes/website/pull/9948

[zparnold]

Thank you!

On Mon, Aug 20, 2018 at 9:36 PM Xing Yang [e-mail] wrote:

@zparnold https://github.com/zparnold Added a placeholder doc PR
kubernetes/website#9948 https://github.com/kubernetes/website/pull/9948

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/features/issues/177#issuecomment-414530549,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE81SM80cpm_P_RyA84IJ4eBqconFlY1ks5uS3JCgaJpZM4Lrsji
.

[justaugustus]

@jingxu97 @xing-yang --
Any update on docs status for this feature? Are we still planning to land it for 1.12?
At this point, code freeze is upon us, and docs are due on 9/7 (2 days).
If we don't here anything back regarding this feature ASAP, we'll need to remove it from the milestone.

cc: @zparnold @jimangel @tfogo

[xing-yang]

@justaugustus Code was merged. Doc PR is ready for review: https://github.com/kubernetes/website/pull/9948

[justaugustus]

Thanks for the update!

[claurence]

Kubernetes 1.13 is going to be a 'stable' release since the cycle is only 10 weeks. We encourage no big alpha features and only consider adding this feature if you have a high level of confidence it will make code slush by 11/09. Are there plans for this enhancement to graduate to alpha/beta/stable within the 1.13 release cycle? If not, can you please remove it from the 1.12 milestone or add it to 1.13?

We are also now encouraging that every new enhancement aligns with a KEP. If a KEP has been created, please link to it in the original post. Please take the opportunity to develop a KEP

[xing-yang]

Actually this feature is alpha in 1.12. We are not planning to go beta in 1.13. It should be changed as follows:

External repo (x.y): 1.8
Alpha release target (x.y): 1.12
Beta release target (x.y): 1.14

[kacole2]

Thanks @xing-yang
/milestone clear

[claurence]

@jingxu97 Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

[xing-yang]

Hi @claurence - we are working on "execution hook" and "PV/PVC taints and tolerations" for 1.14. Do we need KEP for those? If so, I'll submit a KEP for execution hook. Jing is working on a spec for taints and tolerations.

[claurence]

@xing-yang Thanks - all issues should have a KEP associated with them so please add one for this issue. Also do you have links to the PRs for the two items referenced above?

[claurence]

Also is alpha still the correct stage or is this feature graduating to beta?

[xing-yang]

@claurence Yes, it is still Alpha.
We are still working on the KEP so it is not submitted yet. What is the deadline for submitting a KEP?

[xing-yang]

@claurence here's the KEP on ExecutionHook: https://github.com/kubernetes/enhancements/pull/705

[claurence]

@xing-yang please submit the KEP before the enhancements freeze date for Jan 29th if there is expected work for 1.14 - if this is out of scope for 1.14 a KEP will be needed for work in 1.15

[kacole2]

Hello @xing-yang, I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet. This will also require a KEP to be included.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[xing-yang]

Hi @kacole2 , the snapshot feature tracked by this issue will remain as alpha. I'm going to create another enhancement issue to track the ExecutionHook feature separately.

[xing-yang]

Hi @kacole2 , enhancement issue is created for ExecutionHook here: https://github.com/kubernetes/enhancements/issues/962

[msau42]

We are targeting beta for 1.16

[kacole2]

/stage beta
/milestone v1.16

@msau42 this is still missing a KEP. I'll put it in the tracking sheet for now.

[msau42]

@xing-yang can you look into creating a kep for this? I'm not sure how up-to-date the original design doc was, but we can point to it from the kep

[xing-yang]

@msau42 @kacole2 Here is the original design doc that was already merged. https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/csi-snapshot.md

Do I need to create a KEP and point to this doc?

[msau42]

Yes, all enhancements should go through keps now. If the original design doc is mostly up-to-date, then you can just create a kep that points to it. Take a look at the template and see if the kep sections are already included in the design doc or not (namely graduation criteria, test plan, implementation history, etc). Here's an example of a storage kep that points to the original design doc and adds missing sections: https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20190124-csi-topology.md

[xing-yang]

KEP submitted: https://github.com/kubernetes/enhancements/pull/1133

[simplytunde]

Hey, @jingxu97 @xing-yang I'm the v1.16 docs release lead.

Does this enhancement (or the work planned for v1.16) require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday,August 23rd. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[xing-yang]

Hi @simplytunde, we need to modify a few places in the docs to change from v1alpha1 to v1beta1 and also indicates the feature gate change. Will submit a doc PR soon. Thanks.

[kacole2]

@xing-yang @jingxu97
Enhancement freeze is EOD 7/30. This issue still requires a merged KEP to be considered a part of the release. This is being labeled as At Risk and will be dropped from the milestone if it's not merged tomorrow. Thanks.

[xing-yang]

In-tree PR is submitted here: https://github.com/kubernetes/kubernetes/pull/80058.
Out-of-tree PR is submitted here: https://github.com/kubernetes-csi/external-snapshotter/pull/139

[xing-yang]

Doc PR submitted: https://github.com/kubernetes/website/pull/15942

[kacole2]

@xing-yang friendly reminder that Thursday 8/29 is k/k code freeze for getting https://github.com/kubernetes/kubernetes/pull/80058 merged

[xing-yang]

Thanks @kacole2 for the reminder! We are working on it.

[xing-yang]

@kacole2 We couldn't get code merged today. This will be delayed to 1.17.

[mrbobbytables]

Hey there @xing-yang @jingxu97 -- 1.17 Enhancements lead here. I wanted to check in and see if you think this Enhancement will be graduating to beta in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[xing-yang]

Hi @mrbobbytables Yes, we are planning to bring this feature to Beta in 1.17.

[mrbobbytables]

Awesome, thank you! I'll go ahead and add it to the tracking sheet :)

[irvifa]

Hello @xing-yang @jingxu97 I'm one of the v1.17 docs shadows.
Does this enhancement for (or the work planned for v1.17) require any new docs (or modifications to existing docs)? If not, can you please update the 1.17 Enhancement Tracker Sheet (or let me know and I'll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.17) due by Friday, November 8th, it can just be a placeholder PR at this time. Let me know if you have any questions!

[xing-yang]

Thanks @irvifa ! I'll submit a doc soon.

[xing-yang]

@irvifa Doc PR is submitted here: https://github.com/kubernetes/website/pull/17233

[jeremyrickard]

Hey there @xing-yang. I'm one of the enhancements shadows for the 1.17 release. Code Freeze is rapidly approaching on November 14th, so I wanted to touch base and see how this was going. Could you also please link to any k/k PRs that are being worked on as part of this issue? We don't have anything tracked in the enhancements spreadsheet.

Thanks so much!

[xing-yang]

Hi @jeremyrickard, here are the PRs:

• [Merged] Move snapshot APIs to Beta: https://github.com/kubernetes-csi/external-snapshotter/pull/139
• [Merged] Update external-provisioner to use the new snapshot beta API: https://github.com/kubernetes-csi/external-provisioner/pull/335
• Snapshot controller split PR: https://github.com/kubernetes-csi/external-snapshotter/pull/178
• Updated snapshot in-tree e2e tests using snapshot Beta APIs and enable feature gate: https://github.com/kubernetes/kubernetes/pull/80058
• Doc PR: https://github.com/kubernetes/website/pull/17233

[jeremyrickard]

Thanks so much @xing-yang! I'll get the spreadsheet updated.

[xing-yang]

Two more PRs got merged. Here is the new status:

• [Merged] Move snapshot APIs to Beta: kubernetes-csi/external-snapshotter#139
• [Merged] Update external-provisioner to use the new snapshot beta API: kubernetes-csi/external-provisioner#335
• [Merged] Snapshot controller split PR: kubernetes-csi/external-snapshotter#178
• [Merged] Updated snapshot in-tree e2e tests using snapshot Beta APIs and enable feature gate: kubernetes/kubernetes#80058
• [Merged] Doc PR: kubernetes/website#17233

So we've made the Code Freeze date. The only required PR outstanding is the Doc PR which is being reviewed.

[xing-yang]

Doc PR kubernetes/website#17233 is also merged.

[jeremyrickard]

Hey there @xing-yang @jingxu97 -- 1.18 Enhancements lead here. I wanted to check in and see if you think this Enhancement will be graduating to stable in 1.18?

The current release schedule is:

Monday, January 6th - Release Cycle Begins
Tuesday, January 28th EOD PST - Enhancements Freeze
Thursday, March 5th, EOD PST - Code Freeze
Monday, March 16th - Docs must be completed and reviewed
Tuesday, March 24th - Kubernetes 1.18.0 Released

If you do, once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[xing-yang]

Hi @jeremyrickard, this feature will stay at Beta in 1.18. Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[palnabarun]

/remove-lifecycle stale

[msedzins]

Hey there @jingxu97 -- 1.19 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating in 1.19?

In order to have this part of the release:

1. The KEP PR must be merged in an implementable state
2. The KEP must have test plans
3. The KEP must have graduation criteria.

The current release schedule is:

• Monday, April 13: Week 1 - Release cycle begins
• Tuesday, May 19: Week 6 - Enhancements Freeze
• Thursday, June 25: Week 11 - Code Freeze
• Thursday, July 9: Week 14 - Docs must be completed and reviewed
• Tuesday, August 4: Week 17 - Kubernetes v1.19.0 released
• Thursday, August 20: Week 19 - Release Retrospective

If you do, I'll add it to the 1.19 tracking sheet (http://bit.ly/k8s-1-19-enhancements). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[xing-yang]

Hi @msedzins, we plan to stay beta in 1.19. Thanks!

[msedzins]

Hi @xing-yang,

Thank you for the update!

[msedzins]

/milestone clear

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[xing-yang]

/remove-lifecycle stale

[xing-yang]

We are planning to bring this feature to GA in 1.20.

[xing-yang]

Hi @kikisdeliveryservice,

Can you please add this to the v1.20 tracking sheet? Thanks!

[kikisdeliveryservice]

Done :+1: Thanks for the update.

[kikisdeliveryservice]

Hi @xing-yang

Looks like your PR is complete and ready to merge: https://github.com/kubernetes/enhancements/pull/1994#issuecomment-701426142

You just need to remove the hold so it can merge by Freeze on October 6th

:smile:

[xing-yang]

Hi @kikisdeliveryservice,

I just need Michelle to review and LGTM:).

Thanks!

[kikisdeliveryservice]

Hey @xing-yang !

Since your Enhancement is scheduled to be in 1.20, please keep in mind the important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thanks! :smile_cat:
Kirsten

[xing-yang]

Hi @kikisdeliveryservice,

Thanks for the reminder!

Here is the k/k PR:
https://github.com/kubernetes/kubernetes/pull/95282

Xing

[eagleusb]

Hello @xing-yang @jingxu97 :wave:, 1.20 Docs shadow here.

Does this enhancement work planned for 1.20 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.20 branch in the k/website repo.

This PR can be just a placeholder at this time and must be created before Nov 6th
Also take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[xing-yang]

Thanks @eagleusb ! I'll submit a placeholder doc soon.

[eagleusb]

Hi @xing-yang :wave:

Thanks for your update. In the meantime, the docs placeholder deadline is almost here.

Please make sure to create a placeholder PR against the dev-1.20 branch in the k/website before the deadline.

Also, please keep in mind the important upcoming dates:

• Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
• Thursday, Nov 12th: Week 9 - Code Freeze
• Monday, Nov 23rd: Week 11 - Docs PR Ready for ReviewAs a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

[xing-yang]

Hi @eagleusb ,

Thanks for the reminder! Doc PR is submitted here: https://github.com/kubernetes/website/pull/24849

[kikisdeliveryservice]

Hi @xing-yang

Looks like kubernetes/kubernetes#95282 is still open but being actively worked on. Just a reminder that Code Freeze is coming up in 2 days on Thursday, November 12th. All PRs must be merged by that date, otherwise an Exception is required.

Best,
Kirsten

[xing-yang]

Hi @kikisdeliveryservice ,

Thanks for the reminder! We are trying to get reviewers to finish reviewing and approving the PR by the 11/12 deadline.

Xing

[xing-yang]

This PR that updates snapshot CRDs to v1 for cluster addon is merged: https://github.com/kubernetes/kubernetes/pull/96383
We are moving closer.

[kikisdeliveryservice]

Great! just waiting on kubernetes/kubernetes#95282

[xing-yang]

@kikisdeliveryservice ,

https://github.com/kubernetes/kubernetes/pull/95282 is approved. Just waiting for it to be merged:).

[kikisdeliveryservice]

Yay! It's merged! Updating tracking sheet.

Congrats! :fireworks:

[xing-yang]

Thanks @kikisdeliveryservice!