Community: Discussion and idea on improving multi-cloud abstraction

[embano1]

/sig architecture
@mhausenblas and I were discussing the current challenges when developing portable multi-cloud applications on Kubernetes. We acknowledge that at the core this is a controversial discussion, but more and more multi-cloud frameworks, e.g. recently knative, leverage the power of cloud abstraction by Kubernetes to build even more sophisticated frameworks.

Thanks to the Kubernetes cloud provider interface there is a level of standardization which improves decoupling from the underlying cloud provider. However, there are implementation details which leak through. For example, some lower level K8s primitives (e.g. "Volume" or "Ingress") allow provider-specific customizations which make applications/frameworks less portable or at least their deployment and maintenance more complex and thus error-prone. This is also the reason why a good installation guide has deployment files and manuals for every cloud provider targeted. Also, the out-of-tree cloud provider migration might lead to a situation of even less compatibility between clouds as every cloud provider can move faster/ at is own pace and is free to implement any capability due to the improved decoupling from K8s core.

On the other hand, we see more and more cloud-native SDKs, such as Go Cloud, Ballerina and Pulumi which try to abstract common building blocks from the major public cloud vendors, e.g. object storage, queues, etc. This supports the assumption that cross-cloud development has a fellowship.

@mhausenblas and I were wondering, whether there are attempts or discussions how to incorporate functionalities likes those in cloud SDKs natively into Kubernetes. knative is one approach, but as far as I understand it is designed solely with FaaS in mind. We think that any modern workload developed to run on Kubernetes could benefit from integrating common public cloud services, such as object storage, queues, etc. For example, instead of dealing with PersistentVolumeClaims, one would reference a kind: ObjectStore in the deployment spec which, depending on the cluster configuration, could leverage policies and configurations to be enabled for specific public cloud, e.g. AWS and GCP as an example. The application would use base CRUD operations against the storage object.

We understand that there is a need to reduce the number of API objects in K8s core and most of the extensions should be handled by CRDs or any other external extension mechanism. So if there is already a discussion or even development going on, it would be great to get some pointers.

Thanks for the feedback, ideas, and hopefully any pointers to work in that space.

[thockin]

@mhausenblas and I were wondering, whether there are attempts or discussions
how to incorporate functionalities likes those in cloud SDKs natively into Kubernetes.
knative is one approach, but as far as I understand it is designed solely with FaaS in
mind. We think that any modern workload developed to run on Kubernetes could
benefit from integrating common public cloud services, such as object storage,
queues, etc. For example, instead of dealing with PersistentVolumeClaims, one would
reference a kind: ObjectStore in the deployment spec which, depending on the cluster

Object Stores and Volumes are two VERY different things. PV/PVC is
already an abstraction over clouds.

WRT Object Stores, we really REALLY do not want to be in the data
path. Client-side abstractions are fine - wrap a library function
around a switch statement to figure out which cloud to use. I
strongly advise you not to plan to have your own API in the middle of
the client and the server.

configuration, could leverage policies and configurations to be enabled for specific public
cloud, e.g. AWS and GCP as an example. The application would use base CRUD
operations against the storage object.

Solve auth pass-thru. Solve HA. Solve error handling and retries.
Especially around storage, it is deceptively easy to build the first
75% of a working solution, but without that last 25%, it's not really
that useful. And that part is orders of magnitude harder.

We understand that there is a need to reduce the number of API objects in K8s core and
most of the extensions should be handled by CRDs or any other external extension
mechanism. So if there is already a discussion or even development going on, it would be
great to get some pointers.

It comes up every year or so.

[technosophos]

Much of this seems to overlap with the Service Catalog goals.
https://kubernetes.io/docs/concepts/extend-kubernetes/service-catalog/

I would advise opening some conversations with those folks. They're well-steeped in this problem space, and even if you don't see a connection immediately, I think they will have some invaluable insights.

[embano1]

Thank you for your feedback and the pointers. Actually I think Service Catalog could be a great resource to leverage and brainstorm.

[pierreozoux]

There is a KEP opened about this topic, if you are interested in the discussion: https://github.com/kubernetes/enhancements/issues/706