External-storage: Dynamic provisioner for Hostpath

[tamalsaha]

Hi,
I am interested in a hostpath provisioner that can work with StatefulSet with replica > 1. I had a conversation in slack api-machinery room. They pointed me to this project. Here is my conversation: https://github.com/appscode/k8s-addons/issues/14#issuecomment-283732665

I am new to writing my own PV provisioner. Do you think what I am asking for is possible? How do I take the demo hostpath provisioner in this repo and make that a proper one.

Thanks.

[wongma7]

Yes, I think it's possible. Don't know how easy it will be though.

First consider that if you want the hostpath provisioner to handle pre-creating + deleting directories (MkdirAll & RemoveAll) on the host when corresponding PVs are created + deleted (with reclaim policy Delete), an instance of it will need to run on every node and mount every hostPath. And the logic in Delete will have to be improved.

If you don't care about that, if you are willing to handle cleaning up PVs' data yourself (effectively reclaim policy Retain), then you can throw away the assumption that each hostPath provisioner needs to actually mount its node's hostPath. So you can run just one hostpath provisioner for the whole cluster that just creates PVs and ignores creating/deleting directories. You will need to remove the RemoveAll and MkdirAll and such from the code.

Now, the hard part, the scheduling. tbh I don't know much about how to achieve this but I believe you have to extend the scheduler to be aware of those node hostname labels on the PV clayton is talking about. If you go with the first option I said where you have to run an instance of hostPath provisioner on every node, then obviously each provisioner labels its PVs with the node it's running on. If you go with the second option, then I guess you can just round-robin.

[wongma7]

Also pvDir is hardcoded,you'll want to change it https://github.com/kubernetes-incubator/external-storage/blob/master/docs/demo/hostpath-provisioner/hostpath-provisioner.go#L59

Plus the identity is not very useful as-is, if you understand the principle of it then you can remove/improve it if you want.

[tamalsaha]

@wongma7 , thanks for the detailed explanation. I have read the code mostly. I think I understand your comments regarding retention policy, I think those can be made parameterized. Users can use them based on parameters passed to StorageClass. For one more they can use Deployment for a singel pod. In the other mode, they can run DaemonSet and manage host paths.

Regarding Scheduling, I am also new to this level of details. I read the articles:

• https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler.md
• https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler_algorithm.md

From these articles I don't see any mention of default scheduler picks nodes based on label of PV. It is possible to write custom scheduler. But that raises the level of complexity. I will as in sig-scheduler google groups.

[wongma7]

Yes you can do whatever you want re:retention policy, making it an SC parameter sounds good. Note that the controller will always pass to Provision 'options.ReclaimPolicy=delete" but you can ignore it, it's only because upstream hasn't decided the "official" way to do retention policy yet. But again yeah if you want to set it to Retain and do the 'Deployment' style thing then sure why not.

RE: scheduler, please also check out https://kubernetes.io/docs/admin/multiple-schedulers/ . I believe you need write a predicate that looks at a pod->PVC->PV node label to determine which node the pod can be scheduled to. It doesn't look too complex but yeah, hostPath PV support is nonexistent in kube so you may have to write a scheduler and deploy it alongside the default scheduler. (Or patch the existing scheduler but I believe that's way less desirable.)

[tamalsaha]

It doesn't look too complex but yeah, hostPath PV support is nonexistent in kube so you may have to write a scheduler and deploy it alongside the default scheduler. (Or patch the existing scheduler but I believe that's way less desirable.)

The existing scheduler will work fine as-is as long SS are not scaled up after scaling down. Say I have a SS with replica =3 (also uses PV). Now, later I reduce # of replicas to 2. This deleted the pod-2. Then if I increase the replicas to 3, I see that the original PVC, if exists, will be reused. If this is a required behavior, this is not possible without extending the default scheduler for hostpath provisioner. Do you know who can give me a definitive answer?

[tamalsaha]

@wongma7 , regardless of the scheduler issue do you think the hostpath provisioner can be moved from demo to a supported provisioner? I will be willing to send prs if that is necessary.

[tamalsaha]

I found a new proposal for local storage management:

• https://github.com/kubernetes/community/pull/306
• https://github.com/kubernetes/features/issues/121
• https://github.com/kubernetes/kubernetes/issues/9802

[wongma7]

Yes, it's coming though it will be a while (another release or two?) before it is a reality.

In the meantime I'm reluctant to make the hostpath provisioner any more official. It will have to come with a big disclaimer plus the scheduling caveat. Prs are always welcome but the scheduling issue unless solved will make it hard to use in typical clusters.

[tamalsaha]

Yeah. At least another 6 months to wait for official solution. I think we want something sooner. I have read the scheduler code and I think the actual change necessary is fairly simple. Currently default scheduler set PodSpec.Nodename once scheduling decision is made. Custom scheduler also need to apply the scheduler.beta.example.com/nodename:<nodename> on PVs that can bound (directly or via PVC). During the filtering process, I need to check this in addition to checking PodSpec.NodeName.

There is an edge case. Say SS was reduced from replica = 3 to replica=2. Then the node where replica 3 was scheduled originally was removed from cluster. The correct action in this scenario will be left to the user via scheduler.beta.example.com/reassign-missing-node:true (default false) annotation on PodSpec. Scheduler can either fail to schedule the pod or reassign to a new node. If scheduler fails to assign the pod, then user can manually fix the underlying issue, remove the nodename annotation from PV and then reschedeule this pod.

Obviously I have to figure out how to build and distribute the patched scheduler.

[ghost]

So if we implement this, we can use hostpaths across cluster as a PV(I mean PV that is local to pod's node) but without scheduling pod guarantees, isolation etc. of "local persistent volumes" at kubernetes/community#306 ?

My only priority on coming local PV is

• defining PVC with my yamls as "local" without knowing how underlying PV achieve that (works in bare metal cluster, minikube...)
• support for match labels of PVC to try to bind used PV before
• and use node's local ssd or hdds
• having durability (retain)
• without installing anything to host OS, install over just kubernetes
• when we get volume through PVC, we shouldn't deal with permissions like host path wants user to be root or needs some hack

I am not in hurry for isolation, resource limitations, security, ephermal storage aspects of coming proposal.

I would like to contribute once our goals here is clear because we don't want to wait for local PVs streamlined

[tamalsaha]

@thenamli , I have started here from v1.5.4 tag: https://github.com/appscode/kubernetes/commits/hostpath

So far I have figured out how to build and push the binary. My docker image : https://hub.docker.com/r/appscode/hostpath-scheduler/tags/

I am following: https://kubernetes.io/docs/admin/multiple-schedulers/

[tamalsaha]

/cc: @sadlil, @aerokite

[eparis]

reuse of a PVC/PV after a SS is scaled down and up is expected behavior as I recall.

[tamalsaha]

@thenamli, I have got a bare bones implementation in place. Code compiles. I need to do actual testing. If you want to do a code review that will be handy.

Diff: https://github.com/kubernetes/kubernetes/compare/release-1.5...appscode:hostpath

[wongma7]

So apparently there may be a way to co-opt the zone predicate of the scheduler to get hostPath PVs working...disclaimer this is extremely hacky, the zones annotation are still in beta to begin with, there's no guarantee that if this works now it will continue to work later, and you try this at your own risk.
1) label each node with its own "failure-domain.beta.kubernetes.io/zone"
2) label each PV your provisioner creates with a "failure-domain.beta.kubernetes.io/zone"
3) when your statefulset creates PVCs asking for your provisioner's PVs the scheduler respects the label and schedules the statefulset pods to their respective PVs' "zones" aka nodes.

So again, this is extremely hacky and my official recommendation is still to write & use your own scheduler (which is already laxer than upstream's official recommendation to wait patitently for them to implement local storage). But thought it was interesting and worth sharing. :)

[tamalsaha]

Thanks @wongma7 . I think writing scheduler might be the best option. Unfortunately, the new official solution for local storage will not solve my problem :( https://github.com/kubernetes/community/pull/306#issuecomment-286571242

The new local storage will add the PV label based scheduling to scheduler. So, I might not have to keep my forked scheduler for long term. But for my case, where my nodes have a single primary partition, I have to use my custom hostpath provisioner for durable local storage.

[ghost]

@TamalSaha I read your messages there in local PV. Unfortunately, every usage scenario outside of gce and aws seems hacky at best.

First I thought I can contribute k8s to get better k8s support on bare metal. Neither contributing to mainstream k8s nor having own scheduler doesn't give clear and complete solution in the near future. At least it will take 1 year for k8s to mature on persistent storage area.

I am still looking forward to contribute k8s in this area.

So I ended up thinking using network storage service of our bare metal provider and writing a provisioner for it soon.

---

As a metaphor, you are on the business of selling shoes. Then you decide to build your own factory. Then you realize your factory needs more electricity.

Then you realize dam nearby is not enough and people are building a new nuclear reactor for this near the lake. Then you found out that nuclear reactor have some problems for your needs.

At the end, you find yourself at the situation where you are producing coolant systems, control rods, etc. for nuclear reactor when you just want to sell shoes at first :)

[tamalsaha]

So I ended up thinking using network storage service of our bare metal provider and writing a provisioner for it soon.

If you would share what you come up with that will be great!

Btw, I smiled at your metaphor out of pain. This is most definitely how we became contributor to many open source stuff. 😭

[wongma7]

folks, https://docs.google.com/document/d/1so67pZPtBwv3uBg9d3pk4VLzfn9qtuZrbauv1DnNDSk/edit?ts=58dd9c72#heading=h.syg5p31g5672 may interest you as well.

It won't have dynamic provisioning in the first phase (1.7), but if you want to steer it in the right direction I encourage you to comment!

Seems there is a lot of movement around local storage :)

[tamalsaha]

I don't have read access on this file.

[wongma7]

@TamalSaha join/subscribe to the sig-storage mailing list, the original post is here https://groups.google.com/forum/#!topic/kubernetes-sig-storage/mYZ7hTmOO7U

[nailgun]

While dynamic provisioning is not in upstream I continued @TamalSaha work and modified example hostPath provisioner to support multiple storage classes: https://github.com/nailgun/k8s-hostpath-provisioner
There is also custom scheduler based on @TamalSaha work but more simple because it depends on this custom provisioner

[tamalsaha]

Awesome @nailgun ! I am going to give it a try.

[tamalsaha]

Base don my conversation with @msau42 , it seems that the new Local PV will eventually address this use-case. I am looking forward to that. Closing this one.