Pulumi-kubernetes: Allow a model similar to `kubectl apply -f`

[hausdorff]

In Kubernetes it is somewhat common to use {kubectl apply -f, kubectl patch, _etc_.} to update system resources after they have been created. For example, to update default ServiceAccount roles in an EKS cluster after it's been created. (_cf_., this community issue.)

There isn't really anything like this in Pulumi. We offer .get to retrieve the resources and track them in the statefile, but we offer no ability to change them.

In the medium term we can expose creation hooks, which would allow us to shell out to kubectl to update such resources on EKS creation (_cf_., https://github.com/pulumi/pulumi/issues/1691), but in the longer term it is likely that we'll want to solve the general problem of patching system resources that you didn't create.

One idea is to use a system of _overlays_:

• Expose something like core.apps.v1.Deployment.get(...).update([... deployment literal ...])
• The semantics of this are as follows:
  
  
  
  • If this is the first time we're get'ing the resource (_i.e._, we can think of this as a "create", but with an existing resource), then we'll take the live object, and do the normal update operation. What gets stored in the checkpoint file is (1) blank previous inputs, (2) the original live object, and (3) the "current inputs". Under pulumi destroy, we'll probably need to "undo" any updates, so we should hang on to the live object. More on this below.
  
  
  • If this is and subsequent update, we'll simply apply the update operation between (1) the last update args, (2) the current live object, and (3) the new update args.
  
  
  • If this is a pulumi destroy, we'll want to probably "undo" all of this entirely, and put the object back in the state it was before we touched it. In particular we do _not_ want to delete the whole object, or at least, not by default. In the case of system resources this is potentially catastrophic.
  
  

Thoughts? cc @pgavlin @lukehoban @joeduffy

[hausdorff]

Another idea is to call it .patch instead of .update and keep track of the current patch instead of the whole object.

Also worth noting that we probably don't want to expose update on any old CustomResource, but really just the ones we're returning from .get.

[gitfool]

I need support for read-modify-write / patch of existing resources. My use cases so far:

• Patch KubeProxy & CoreDNS version (https://github.com/pulumi/pulumi-eks/issues/253):

kubectl set image -n kube-system daemonset.apps/kube-proxy kube-proxy=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.14.6
kubectl set image -n kube-system deployment.apps/coredns coredns=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.3.1

• Patch KubeProxy metrics bind address for Prometheus (prometheus-operator#kubeproxy):

kubectl -n kube-system edit cm kube-proxy
# metricsBindAddress: 0.0.0.0:10249

These sorts of things need to be captured for automatic rollout.

[lukehoban]

We added a kubectlReplace endpoint in https://github.com/pulumi/pulumi-kubernetes/pull/579. I'm not sure that's quite what's needed here - but it's similar - and we may want to use a similar approach.

[lblackstone]

I actually removed the replace functionality awhile back because it didn't turn out to be very helpful. The main problem was that it was asynchronous, and didn't do any readiness checking.

It is possible to use Pulumi's import functionality to edit existing resources, but the process is a bit heavy:

1. Import existing resource
2. pulumi up to import
3. Make desired changes to imported resource
4. pulumi up to apply changes

Ideally, we should support simple edits in a single step to make this easy to accomplish in CI.
I like the original overlay suggestion as long as it's a synchronous method that includes readiness checks.

This might look something like:

k8s.apps.v1.Deployment.get("kube-system/coredns").patch({
    image: "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.3.1"
})

[gkonc001]

I'm currently trying to find something similar to kubectl patch deployment and I was wondering if a feature like that is/was implemented.

[lblackstone]

@gkonc001 The feature I described hasn't been implemented yet. Can you tell me more about your use case?

[gkonc001]

@lblackstone Yes, I currently have a spring cloud data flow chart that uses grafana monitoring and I'm trying to do something similar to kubectl patch deployment grafana --patch "$(cat ./kubernetes/grafana-deployment.yaml)" to the helm chart using the grafana-deployment.yaml file.

[davzucky]

@lblackstone I have a requirement as well for that feature.
I'm looking at this feature to be able to use the digital ocean registry for any image. From the doc you need to update the default profile
https://www.digitalocean.com/docs/images/container-registry/quickstart/
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "registry-<my-registry>"}]}'

[Sodki]

My use case is to update the configuration of CoreDNS, which comes pre-installed on EKS. I have two options:

1. Either use kubectl patch to update the configuration
2. Use kubectl apply with a YAML file provided by AWS, which can also be used to upgrade the version of CoreDNS

So, reading through other people's messages, it seems we need both an apply and a patch.

Personally I would like for an apply to be able to override existing objects, but the result would be resources managed by Pulumi.

[jwenz723]

Without this feature being available, would the current suggested way of using pulumi to do something like patching the kube-proxy container image be to use Dynamic Providers to wrab the standard kubernetes libraries?

[lblackstone]

Without this feature being available, would the current suggested way of using pulumi to do something like patching the kube-proxy container image be to use Dynamic Providers to wrab the standard kubernetes libraries?

Yes, using a dynamic provider around a native k8s client is a good way to handle this.

[gitfool]

@lblackstone I'd love to see a specific example, if there isn't one already.

Also, more generally, is it looking like there will be a way to write dynamic providers in other languages like .NET/C#? What's the status of the current experiments? Sorry to keep pressing this issue but it's become clear it's really important for lots of little things.

[Slaiyer]

Without this feature being available, would the current suggested way of using pulumi to do something like patching the kube-proxy container image be to use Dynamic Providers to wrab the standard kubernetes libraries?

Yes, using a dynamic provider around a native k8s client is a good way to handle this.

Can we have sample code for this?
I'm looking to patch the default serviceaccount of a given namespace with image pull secrets.

[clstokes]

Here's an example of doing kubectl apply -f from a DynamicProvider in TypeScript. Note, that provider also does some kops commands so it would need to be adjusted for this specific case.

[farvour]

I would like to see a feature like this. When EKS creates clusters, it's common for someone to want to edit/patch/change something about the default installed kube-proxy or coredns without having to full-blown manage it in Pulumi. This allow allows for a bit of idempotent when a new cluster is spun up without having to cover for all the semantics of changes on the object I don't care about (the image tag, for example, changes from EKS version to EKS version for kube-proxy).

[gitfool]

@farvour I'm hoping https://github.com/aws/containers-roadmap/issues/1159 will help in this specific case, although it looks a little rough around the edges.

Meanwhile, I'm still holding out that .NET/C# dynamic providers will be available soon.