Enhancements: Provide environment variables expansion in sub path mount

[kevtaylor]

Feature Description

• One-line feature description (can be used as a release note): Provide environment variables expansion in sub path mount
• Primary contact (assignee): @kevtaylor
• Responsible SIGs: kubernetes/sig-storage
• Design proposal link (community repo):
  kubernetes/kubernetes#48677
• KEP: https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20181029-volume-subpath-env-expansion.md
• Link to e2e and/or unit tests:
  https://github.com/kubernetes/kubernetes/pull/49388
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: @msau42 @jsafrane
• Approver (likely from SIG/area to which feature belongs): @msau42
• Feature target (which target equals to which milestone):
  
  
  
  • Alpha release target (1.11)
  
  
  • Beta release target (1.15)
  
  
  • Stable release target (1.17)
  
  

[msau42]

Assignee should be @kevtaylor ;-)

[kevtaylor]

Related to this issue https://github.com/kubernetes/kubernetes/issues/48677 not the one above

[jsafrane]

"Design proposal link" leads to "taint node 'shutdown' condition"

[kevtaylor]

@jsafrane Sorry, I corrected it in the comment above - but I didn't know how to edit the original

[kevtaylor]

@jsafrane Actually - forget that - I just did it

[kevtaylor]

@jsafrane I also update the assignee as per @msau42 comments

[justaugustus]

@kevtaylor
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

• Description
• Milestone
• Assignee(s)
• Labels:
  
  
  
  • stage/{alpha,beta,stable}
  
  
  • sig/*
  
  
  • kind/feature
  
  

/assign @kevtaylor

cc @idvoretskyi

[kevtaylor]

/sig storage

[kevtaylor]

/kind feature

[justaugustus]

Thanks for the update, @kevtaylor!

[justaugustus]

@kevtaylor --
We're doing one more sweep of the 1.11 Features tracking spreadsheet.
Would you mind filling in any incomplete / blank fields for this feature's line item?

[kevtaylor]

@justaugustus I didn't have edit rights - I have requested them

[kevtaylor]

@justaugustus Should be up to date now.

[justaugustus]

@kevtaylor yup, just saw it. Thanks for the update!

[kevtaylor]

@justaugustus Hi. I wanted to update the test status of the feature in the spreadsheet but it no longer seems to allow me to edit it. Is that intentional?

[justaugustus]

@kevtaylor definitely intentional. We locked up permissions on the sheet, but feel free to drop comments in the relevant cells, tag me, and I'll get it updated.

[kevtaylor]

@justaugustus Thanks, I saw that you can add a comment, so I have added one.

[kacole2]

@kevtaylor This feature was worked on in the previous milestone, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.12 as mentioned in your original post.

If there are any updates, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.

---

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

• Docs deadline (open placeholder PRs): 8/21
• Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

[kevtaylor]

@kacole2 The intention was to do a simple migration to beta, but this has been blocked by @liggitt and @thockin who want to revamp the api instead so I don't know what the future plans are. Discussion is here https://github.com/kubernetes/kubernetes/pull/65769

[msau42]

Hi @kevtaylor sorry I missed the discussion earlier. The points that were brought up make sense from an API backwards compatibility view, sorry I didn't think about it when reviewing the alpha design :( Given that, then the first approach outlined in #48677 would make more sense.

[kevtaylor]

@msau42 Okay. Fine. I will leave it in your hands then to schedule an API implementation change as you see fit. Thanks

[msau42]

@kevtaylor there's no need to schedule. If anyone is interested in driving this then we can track for this release.

[justaugustus]

Sounds like this might not make it in. Let me know if that changes.
Pinging @kubernetes/sig-storage-misc to see if anyone is interested in taking this on.

/help

[kevtaylor]

@justaugustus Yes, that is likely as the proposed change is a large scale change involving the api and that is beyond my expertise in the product.

[cbluth]

I have enabled the feature gate VolumeSubpathEnvExpansion, are there usage examples?
@kevtaylor

[kevtaylor]

https://kubernetes.io/docs/concepts/storage/volumes/#using-subpath-with-expanded-environment-variables

[bungder]

I have already run a cluster using kubeadm v1.11 and I don't want to reset and re-init it. How can I enable VolumeSubpathEnvExpansion without stopping my cluster? I searched kubernetes documentations and found no way to do it. It would be great if you know how to do so. Thanks a lot.
@kevtaylor @cbluth

[kevtaylor]

@bungder It is a feature gate that needs to be enabled --feature-gates as it is only alpha - it is also being reshaped for future releases, so is subject to change - you would need to restart the server components to enable any feature-gate

[bungder]

@kevtaylor I found 2 commands that can be followed by --feature-gates:

• kubeadm init --feature-gates
• kubeadm config upload from-flags --feature-gates

The 1st one is used to init cluster so it doesn't fit this situation.
The 2nd one doesn't support this feature, it returns unrecognized feature-gate key: VolumeSubpathEnvExpansion

So is there any command that could enable VolumeSubpathEnvExpansion feature without stopping cluster?

[bungder]

@kevtaylor I did not see "you would need to restart the server components to enable any feature-gate" just now. Rolling update is acceptable. But where should I place this feature config?

[kevtaylor]

@bungder I don't use kubeadm so unfortunately, I don't know how feature gates get into your cluster using that tool - probably better to go down into one of the slack channels and ask

[bungder]

@kevtaylor Still, thanks a lot

[kobozo]

@bungder Did you found a solution to this? I have the same issue

[bungder]

@kobozo Sorry, I didn't. I gave up using this feature. If you find any way to solve this, please add a comment to this issue.

[kevtaylor]

/remove-help

[kevtaylor]

KEP Approved proposal: https://github.com/kubernetes/community/pull/2871

[claurence]

@kevtaylor Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP

[kevtaylor]

@claurence This enhancement does have a kep and is in review at https://github.com/kubernetes/kubernetes/pull/71351
/cc @msau42

[msau42]

We plan to make some changes to the alpha feature in 1.14. It will still stay in alpha, but there will be a new API, so we'll still need new docs.

[claurence]

@msau42 @kevtaylor thanks! Do either of you have a link to any PRs for the changes being made in 1.14? Thanks

[kevtaylor]

@claurence The code PR kubernetes/kubernetes#71351
The docs PR: https://github.com/kubernetes/website/pull/11843
And the kep: https://github.com/kubernetes/community/pull/2871

[zqzzq]

Hello, I want to know how to enable the VolumeSubpathEnvExpansion feature gate, I added a "- --feature-gates=VolumeSubpathEnvExpansion=true" command in the kube-apiserver.manifest configuration file, but it seems not work, there is no subPath corresponding directory in the container,and the error is "sh: getcwd: No such file or directory",so I need some help. Thanks a lot.

[kevtaylor]

@zqzzq I think you'll have to enable the feature gate on kubelet/controller manager as well - if you post me the yaml for the volume mount, I can take a look, but this is probably the wrong forum to discuss support issues - I am in the kubernetes slack room, if you want to ping me there

[liggitt]

/assign

I'll take API review for this

[thockin]

I feel like I've already done this one?

[liggitt]

It got moved to a separate field. PR was open at https://github.com/kubernetes/kubernetes/pull/71351 and had lgtm for API, was pending a couple questions on tests

[kevtaylor]

@liggitt I ironed out the breaking/flaking test and discovered the problem, so the alpha suite passes consistently each time now. So as far as I am concerned all test bases have been covered as requested by yourself and Michelle

[kevtaylor]

@liggitt Also, the docs are all in line for the new api and approved and just on hold https://github.com/kubernetes/website/pull/11843

[claurence]

@kevtaylor looking over the KEP for this enhancement I don't see any testing plans - can someone help PR in testing plans for this enhancement? This information is helpful for knowing readiness of this feature for the release and is specifically useful for CI Signal.

If we don't have testing plans this enhancement will be at risk for being included in the 1.14 release

[msau42]

We have alpha tests running here

The new testing plan and graduation criteria sections were added to the KEP template after this cycle's KEP deadline. We will need to go back and fill them in.

[claurence]

Thanks @msau42 - if we can just add that link to the KEP that would be helpful!

[lachie83]

Hello @kevtaylor, 1.14 enhancement shadow here. Code Freeze is March 7th and all PRs must be merged by then to your issue to make the 1.14 release. What open K/K PRs do you still have that need to merge? Thanks

[kevtaylor]

@lachie83 The PRs for both the code and the documentation are merged

[lachie83]

Thanks for the update @kevtaylor @msau42. Can you please submit a PR against the KEP with the testing plan and graduation criteria?

This information is helpful for knowing readiness of this feature for the release and is specifically useful for CI Signal.

If we don't have testing plans this enhancement will be at risk for being included in the 1.14 release

[kevtaylor]

Can you please submit a PR against the KEP with the testing plan and graduation criteria?
@msau42 What does this mean?

[claurence]

@kevtaylor if you have E2E tests and graduation criteria for this enhancement can you add them to the KEP? @msau42 indicated there was alpha tests running so adding a link to them in the KEP would be helpful for CI Signal

[kevtaylor]

@claurence please can you clarify what you mean by graduation criteria? And where does the link go in the kep? I couldnt see any obvious place in it. Do we add a new section to the original doc? An example would be helpful. Thanks

[claurence]

@kevtaylor For graduation criteria is there criteria for when we know this enhancement is ready to move from alpha --> beta --> stable

Making a new section in the KEP is fine to add the link to test plans. https://github.com/kubernetes/enhancements/blob/master/keps/YYYYMMDD-kep-template.md#graduation-criteria that KEP template should explain more

[kevtaylor]

@claurence great thanks. Ill sort that out

[kacole2]

@kevtaylor did this graduate to stable in 1.14? If so, please close the issue. If not, is there any work planned for 1.15?

/milestone clear

[msau42]

This was still alpha in 1.14.

For 1.15, we can consider moving to beta.

[kacole2]

@msau42 is there a KEP with associated graduation criteria? Apologies if I missed it somehow.

[kevtaylor]

https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/20181029-volume-subpath-env-expansion.md

[kacole2]

/milestone v1.15
/stage beta

[kevtaylor]

Updated kep with the beta links https://github.com/kubernetes/enhancements/pull/1058

[simplytunde]

@kevtaylor Just a friendly reminder we're looking for a PR against k/website (branch dev-1.15) due by Thursday, May 30. It would be great if it's the start of the full documentation, but even a placeholder PR is acceptable. Let me know if you have any questions!

[kevtaylor]

@simplytunde It is all detailed on this feature above: https://github.com/kubernetes/website/pull/13846 and it is on the updated kep as well

[kevtaylor]

@msau42 Do I need to get someone to approve the website and kep changes?

[simplytunde]

Thanks @kevtaylor

[kacole2]

Hi @kevtaylor @msau42. Code Freeze is Thursday, May 30th 2019 @ EOD PST. All enhancements going into the release must be code-complete, including tests, and have docs PRs open.

Please list all current k/k PRs so they can be tracked going into freeze. If the PRs aren't merged by freeze, this feature will slip for the 1.15 release cycle. Only release-blocking issues and PRs will be allowed in the milestone.

If you know this will slip, please reply back and let us know. Thanks!

[kevtaylor]

@kacole2 The merged PR is https://github.com/kubernetes/kubernetes/pull/76546
and the docs are lgtm at https://github.com/kubernetes/website/pull/13846

[kevtaylor]

@kacole2 The tests are all documented in the kep and operational on the test grids

[kacole2]

Hi @kevtaylor , I'm the 1.16 Enhancement Lead. Is this feature going to be graduating alpha/beta/stable stages in 1.16? Please let me know so it can be added to the 1.16 Tracking Spreadsheet. If not's graduating, I will remove it from the milestone and change the tracked label.

Once coding begins or if it already has, please list all relevant k/k PRs in this issue so they can be tracked properly.

Milestone dates are Enhancement Freeze 7/30 and Code Freeze 8/29.

Thank you.

[kevtaylor]

@kacole2 This feature graduated to Beta in 1.15 so my understanding is we skip a version before moving to GA in 1.17. Perhaps @msau42 could confirm that.

[kacole2]

@kevtaylor I would recommend a release cycle of burn-in before immediately promoting. But some enhancements are high priority and are moving at higher pace. Whatever you all are comfortable with. For now, I'll clear the milestone and we can revisit if you have any changes.

/milestone clear

[justaugustus]

AFAIK, the cycle of burn-in is supposed to be required.

[liggitt]

AFAIK, the cycle of burn-in is supposed to be required.

I'm not aware of a requirement for a dead release between beta and stable

[kevtaylor]

@liggitt I seem to remember reading it in the api doc but couldnt point to it. If were happy to make this GA in 1.16, then Ill prepare the necessary PRs

[liggitt]

I'll defer to @kubernetes/sig-storage-feature-requests for guidance, but as far as API and skew compatibility is concerned, it could be promoted in 1.16

[msau42]

I think it would be good to get some more soak time for one more release. Realistically, adopting and upgrading to a new release in production is slower than our quarter release cycle.

[kevtaylor]

/stage stable

[palnabarun]

Hey there @kevtaylor , 1.17 Enhancements shadow here. I wanted to check in and see if you think this Enhancement will be graduating to stable in 1.17?

The current release schedule is:

• Monday, September 23 - Release Cycle Begins
• Tuesday, October 15, EOD PST - Enhancements Freeze
• Thursday, November 14, EOD PST - Code Freeze
• Tuesday, November 19 - Docs must be completed and reviewed
• Monday, December 9 - Kubernetes 1.17.0 Released

If you do, I'll add it to the 1.17 tracking sheet (https://bit.ly/k8s117-enhancement-tracking). Once coding begins please list all relevant k/k PRs in this issue so they can be tracked properly. 👍

Thanks!

[kevtaylor]

@palnabarun yes this is graduating in 1.17

[kevtaylor]

@palnabarun
KEP Graduation update: https://github.com/kubernetes/enhancements/pull/1255
Code PR: https://github.com/kubernetes/kubernetes/pull/82578
Docs PR: https://github.com/kubernetes/website/pull/16547

[palnabarun]

@kevtaylor Awesome. :tada: Thank you for the updates. I have updated the tracking sheet with the details.

[palnabarun]

/milestone v1.17

[irvifa]

Hello @kevtaylor I'm one of the v1.17 docs shadows.
Does this enhancement for (or the work planned for v1.17) require any new docs (or modifications to existing docs)? If not, can you please update the 1.17 Enhancement Tracker Sheet (or let me know and I'll do so)

If so, just a friendly reminder we're looking for a PR against k/website (branch dev-1.17) due by Friday, November 8th, it can just be a placeholder PR at this time. Let me know if you have any questions!

[kevtaylor]

@irvifa Docs PR: kubernetes/website#16547

[irvifa]

Awesome... Thanks a lot @kevtaylor !

[palnabarun]

Hi @kevtaylor

I am one of the Enhancements Shadow for the 1.17 Release Team. We are very near to the Code Freeze (Nov 14th) for this release cycle. Just checking in about the progress of this enhancement. I see that https://github.com/kubernetes/kubernetes/pull/82578 was filed in relation to this. Is there any other PR related to this enhancement? If yes, can you please link them here?

Thank you in advance :smile:

[kevtaylor]

@palnabarun No further PRs other than those already mentioned here

[palnabarun]

@kevtaylor Thank you for the updates! :)

[palnabarun]

Hi @kevtaylor , tomorrow is code freeze for the 1.17 release cycle. It looks like the k/k PRs have not yet been merged. We're flagging as the enhancement as At Risk in the 1.17 Enhancement Tracking Sheet.

Do you think they will be merged by the EoD of the 14th (Thursday)? After that point, only release-blocking issues and PRs will be allowed in the milestone with an exception.

[kevtaylor]

@palnabarun the issue that was added is not an essential enhancement but merely to switch the tests to conformance. The original PR is not blocked by this and is merged. If @msau42 approves today, this new test suite can also go in

[palnabarun]

@kevtaylor

Thank you for the updates. In that case, shall I remove https://github.com/kubernetes/kubernetes/pull/85128 from 1.17 tracking due to its non-blocking nature and flip back the status to Tracked?

[kevtaylor]

@palnabarun I would do that in case 85128 doesnt get approved in time. Sorry to confuse everything! If it gets merged today we can put it in the release if that's not going to cause pain.

[palnabarun]

@kevtaylor Now worries! That sounds good to me.

Do reach out in case you have any queries. :smiley:

[kevtaylor]

@palnabarun it appears that the ga feature has been removed from the milestone in the spreadsheet tracker.

[guineveresaenger]

We're putting it back in, sorry for all the confusion! :shipit:

[jeremyrickard]

Hey @kevtaylor , Happy New Year! 1.18 Enhancements lead here 👋 Thanks for getting this across the line in 1.17!!

I'm going though and doing some cleanup for the milestone and checking on things that graduated in the last release. Since this graduated to GA in 1.17, I'd like to close this issue out but the KEP is still marked as implementable. Could you submit a PR to update the KEP to implemented and then we can close this issue out?

Thanks so much!

[kevtaylor]

@jeremyrickard Hi. Thanks for that. I'll sort the KEP out and ping back when I've submitted it.

[justaugustus]

@kevtaylor -- You can feel free to come back and /close this once the KEP has been flipped to implemented.