Enhancements: CSI Inline Volume Support

[saad-ali]

Feature Description

• One-line feature description (can be used as a release note): CSI volumes can only be referenced via PV/PVC today. This works well for remote persistent volumes. However, we would like CSI to be used to create local ephemeral volumes (e.g. volumes use to inject identity information in to a pod), and these types of volumes should be able to be referred to directly in pod spec (without PV/PVC).
• Primary contact (assignee): @jsafrane
• Responsible SIGs: @kubernetes/sig-storage-feature-requests
• Design proposal link (community repo): https://github.com/kubernetes/community/pull/2273
• KEP: 20190122-csi-inline-volumes.md
• Link to e2e and/or unit tests:
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @childsb
• Approver (likely from SIG/area to which feature belongs): @saad-ali
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target (x.y): 1.13
  
  
  • Beta release target (x.y): 1.16
  
  
  • Stable release target (x.y): 1.17?
  
  
• PRs
  
  
  
  • [x] https://github.com/kubernetes/kubernetes/pull/74086
  
  

[justaugustus]

Thanks for the update! This has been added to the 1.12 Tracking sheet.

/assign @jsafrane
/stage alpha

[zparnold]

Hey there! @saad-ali I'm the wrangler for the Docs this release. Is there any chance I could have you open up a docs PR against the release-1.12 branch as a placeholder? That gives us more confidence in the feature shipping in this release and gives me something to work with when we start doing reviews/edits. Thanks! If this feature does not require docs, could you please update the features tracking spreadsheet to reflect it?

[justaugustus]

@jsafrane @vladimirvivien --
Any update on docs status for this feature? Are we still planning to land it for 1.12?
At this point, code freeze is upon us, and docs are due on 9/7 (2 days).
If we don't here anything back regarding this feature ASAP, we'll need to remove it from the milestone.

cc: @zparnold @jimangel @tfogo

[saad-ali]

This PR (https://github.com/kubernetes/kubernetes/pull/67452) did not make it to 1.12. Moving to next milestone

[justaugustus]

Thanks for the update!

[vladimirvivien]

/assign vladimirvivien

[AishSundar]

@saad-ali @vladimirvivien how confident are we about this going Alpha in 1.13? Is there a list of pending PRs or issues we are tracking for this work?

[kacole2]

@vladimirvivien there hasn't been any communication on this issue but from the looks of https://github.com/kubernetes/kubernetes/pull/68232 there is activity happening so this is going to continue to be tracked for v1.13 milestone.

[tfogo]

Hi @vladimirvivien @jsafrane @saad-ali, I'm the docs wrangler for the 1.13 release. Could you please open a placeholder PR for the docs for this enhancement against the dev-1.13 branch of k/website and send me a link? If you already have a docs PR open, or if this doesn't require docs in k/website, please let me know.

The deadline for placeholder PRs for the 1.13 release is November 8. So it's important to make a docs PR as soon as possible.

If you have any questions about any of this, I'm happy to help. You can also message me on slack (I'm tfogo there too).

Thanks!

[claurence]

Hi @vladimirvivien I'm an enhancements shadow checking in on how this issue is tracking - looks like lots of activity in the K/K PR. Code slush is on 11/9 and code freeze is coming up on 11/15 are you confident this issue will make those dates?

[vladimirvivien]

Doc - https://github.com/kubernetes/website/pull/10934

[AishSundar]

@tfogo for docs PR

[AishSundar]

@vladimirvivien @saad-ali could you plz provide the latest update on this enhancement and status for 1.13 please. Is kubernetes/kubernetes#68232 is the only pending PR (code snd test).

[kacole2]

/milestone 1.14

[k8s-ci-robot]

@kacole2: The provided milestone is not valid for this repository. Milestones in this repository: [v1.13, v1.14]

Use /milestone clear to clear the milestone.

In response to this:

/milestone 1.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kacole2]

/milestone v1.14

[claurence]

@saad-ali @jsafrane Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

[kfox1111]

Yes, this is expected to be alpha in 1.14.

[claurence]

@kfox1111 @jsafrane I'll add it to the 1.14 tracking sheet. I don't see a KEP for this issue and we want to have KEPs for all issues in 1.14 - can you drop a link to the KEP?

[kfox1111]

Yeah. it was started preKEP. We can try and get one in soon.

[vladimirvivien]

@claurence I will convert this into a KEP. Than you.

[vladimirvivien]

@claurence KEP started here - https://github.com/kubernetes/enhancements/pull/716

[claurence]

Thanks @vladimirvivien - any open PRs we should track for 1.14?

[makoscafee]

Hey @vladimirvivien @jsafrane👋 I'm the v1.14 docs shadow.

Does this enhancement require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.14) due by Friday, March 1. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[kfox1111]

@claurence https://github.com/kubernetes/kubernetes/pull/68232

@MAKOSCAFEE I think there was a docs pr opened when this almost merged in 1.13. I don't have the link handy though. @vladimirvivien do you?

[makoscafee]

@kfox1111 i think this is the PR you were talking about kubernetes/website#10934. If so can this be opened against dev-1.14 branch and looks like it has a lot of conflicts to resolve.

[kfox1111]

That's the one I was thinking of. Thanks for finding it. We should update that, yeah.

[vladimirvivien]

@MAKOSCAFEE @kfox1111 there will definitely be updates to the doc, not sure if a new one will be created.

[vladimirvivien]

* PRs to follow *

• https://github.com/kubernetes/kubernetes/pull/74086

[makoscafee]

@vladimirvivien thanks for the information. docs PR(or a placeholder PR) should be against dev-1.14 branch before March 1 which is the deadline. the sooner the better

[saad-ali]

I am interested in shadowing the API review for this.

CC @liggitt

[vladimirvivien]

@MAKOSCAFEE Thanks for the thorough info. I will get that started soon.

[liggitt]

/assign

[thockin]

I can help with API, too if you need Jordan.

@vishh shadow

[krmayankk]

Is this getting to beta in 1.14 ? We are interested in this feature as a user and can also help with implementation if you need a hand

[vladimirvivien]

@krmayankk this feature will be alpha in 1.14

[claurence]

@jsafrane looking over the KEP for this enhancement I don't see any testing plans - can someone help PR in testing plans for this enhancement? This information is helpful for knowing readiness of this feature for the release and is specifically useful for CI Signal.

If we don't have testing plans this enhancement will be at risk for being included in the 1.14 release

[msau42]

@vladimirvivien has a followup PR to add more detailed testing information: https://github.com/kubernetes/enhancements/pull/790

[makoscafee]

Hi @vladimirvivien just to check if you managed to open docs PR(or a placeholder PR) against dev-1.14 as the deadline for doing so(1 March) is tomorrow.
Thanks

[vladimirvivien]

@MAKOSCAFEE here is the place holder doc PR for this feature - https://github.com/kubernetes/website/pull/10934

[vladimirvivien]

@MAKOSCAFEE can someone take a look at doc PR please - PTAL - https://github.com/kubernetes/website/pull/10934

[makoscafee]

Hi @vladimirvivien sure let me ping the right person to take a look. Thanks

[kacole2]

I'm the Enhancement Lead for 1.15. Is this feature going to be graduating alpha/beta/stable stages in 1.15? Please let me know so it can be tracked properly and added to the spreadsheet.

Once coding begins, please list all relevant k/k PRs in this issue so they can be tracked properly.

[msau42]

This feature is going to remain alpha in 1.15.

[vladimirvivien]

@kacole2 Updated doc PR for this - https://github.com/kubernetes/website/pull/14704

[rbitia]

Hey @jsafrane, I'm the 1.16 Enhancement Shadow. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If it's not graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thanks!

[msau42]

Hi we're planning to target beta in 1.16.

[pohly]

/assign

[kacole2]

/stage beta
/milestone v1.16

[VineethReddy02]

Hey, @jsafrane I'm the v1.16 docs release shadow.

Does this enhancement require any new docs (or modifications)?

Just a friendly reminder we're looking for a PR against k/website (branch dev-1.16) due by Friday,August 23rd. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

Thanks!

[pohly]

Doc change for 1.16 here: https://github.com/kubernetes/website/pull/15705

[kacole2]

@pohly @jsafrane i see that https://github.com/kubernetes/kubernetes/pull/74086 is already merged. Are there any other k/k PRs we need to track so this is ready to go Beta for 1.16? code freeze for 1.16 is on Thursday 8/29.

[pohly]

We want to merge more tests: https://github.com/kubernetes/kubernetes/pull/81960

Once that is in, we need to turn the feature on by default: https://github.com/kubernetes/kubernetes/pull/82004

That should be all for 1.16.

[mrbobbytables]

Hey there @pohly @jsafrane -- 1.17 Enhancements lead here. I wanted to check in and see if you think this Enhancement will be graduating to alpha/beta/stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

/milestone clear

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[pohly]

Nothing changed in 1.17, still beta as in 1.16.

Personally I think we cannot consider this done until we at least have solved capacity tracking (https://github.com/kubernetes/enhancements/pull/1353) because some kinds of ephemeral inline volumes (those which consume significant amount of storage) are very hard to use reliably otherwise.

/remove-lifecycle stale

[kfox1111]

We may need to solve the service account to driver as well. Not sure. It could be considered out of scope though.

[helayoty]

Hey @pohly @jsafrane -- 1.18 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to stable in 1.18 or having a major change in its current level?

The current release schedule is:

Tuesday, January 28th EOD PST - Enhancements Freeze
Thursday, March 5th, EOD PST - Code Freeze
Monday, March 16th - Docs must be completed and reviewed
Tuesday, March 24th - Kubernetes 1.18.0 Released

If you would like to include this enhancement, please list all relevant k/k PRs in this issue so they can be tracked properly.

Thanks!

[pohly]

This will stay in beta in 1.18.

[helayoty]

@pohly Thank you for the update. Any planning on which release this feature might be graduating to stable?

[pohly]

Any planning on which release this feature might be graduating to stable?

Nothing specific yet.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[pohly]

/remove-lifecycle stale

[johnbelamaric]

@pohly Release 1.19 enhancements shadow here. Any plans for this in 1.19?

[johnbelamaric]

@pohly @saad-ali Any plans for this in 1.19? We need to track it in the release team if so - please let me know.

_Enhancements freeze is in one week, on May 19._

Note that recently the KEP format has changed. Additionally, #1620 merged recently, adding production readiness review questions to the KEP template. Please take this opportunity to reformat your KEP and also answer the questions add to the template in that PR.

Thanks!

[pohly]

No changes planned in 1.19. We keep this in beta because we are working on a related feature (see #1698) which may or may not be merged with this one here.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[pohly]

/remove-lifecycle stale
/lifecycle frozen

[fejta-bot]

Enhancement issues opened in kubernetes/enhancements should never be marked as frozen.
Enhancement Owners can ensure that enhancements stay fresh by consistently updating their states across release cycles.

/remove-lifecycle frozen

[kikisdeliveryservice]

Hi @pohly

Enhancements Lead here. Any plans to graduate this in 1.20?

Thanks,
Kirsten

[kikisdeliveryservice]

Hi @pohly

Following up: 1.20 Enhancements Freeze is October 6th. Could you let us know if you have any plans in 1.20? To be included in the milestone:
The KEP must be merged in an implementable state
The KEP must have test plans
The KEP must have graduation criteria (missing)

Also the linked KEP is using an older format please see for ref and update https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template

Thanks,
Kirsten

[pohly]

No plans for graduation in 1.20. https://github.com/orgs/kubernetes-csi/projects/33 tracks graduation work items. Updating the KEP is tracked there.

[pohly]

We discussed the current API (https://docs.google.com/document/d/1yAe3SPPosgC_QgmnY7oJTmZYWrqLrii1oA4de67DEcw/edit) and agreed
that we want to change the API before moving to GA.

There was consensus that having "CSI" in "CSI ephemeral volumes" is
confusing, primarily because not all CSI drivers support it. The Google
doc has various alternatives that were considered. Eventually we agreed
to rename the API:

• VolumeSource.CSI -> VolumeSource.Ephemeral.Custom
• CSI.VolumeAttributes -> Custom.Parameters

The existing "CSI" variant will become deprecated. But because it is
part of the v1 VolumeSource API, it cannot be removed in practice from the
current API, i.e. the in-tree code will have to support both ways of
specifying it. Each VolumeSource instance may have either CSI or
Ephemeral.Custom, but not both.

This enhancement needs an owner who has the time to focus on that
change and can move it forward. Because of the API change, introducing
Ephemeral.Custom as beta (?) is probably the next step instead of going
straight to GA with a new API.