Terraform-provider-kubernetes: Add generic kubernetes resource

[tmaier]

it would be great if there would be a generic kubernetes resource, which does "$ kubectl apply" on whatever json or yaml file given.

This would greatly simplify the support for not officially supported kubernetes objects and it would allow to keep the yaml format of kubernetes whenever it makes sense (e.g. the object to create is something taken from a readme, something not worth to be converted to the terraform format)

[shaneramey]

FYI there's a alternative Terraform provider for Kubernetes that allows for configuration via raw manifests.

[joshperry]

We need something like this as well since we have started codifying our operations domain knowledge in automation. However, what I think would be more useful is to have a resource definition format that would allow this provider to support any resource and properties with the full change-management fidelity that Terraform allows.

You seem to lose a large number of capabilities with the manifest-driven provider.

[tmaier]

Related issue: #3

More pro arguments:

• This allows to support alpha and beta resources of kubernetes
• Less maintenance necessary at kubernetes provider

[dominik-lekse]

While working with the Terraform Kubernetes Provider to manage a cluster, I also thought about having a generic resource. In particular, I was surprised that one of the most important resource such as the Deployment is not yet supported.

The alternative terraform-provider-k8s mentioned by @shaneramey has the disadvantage that it relies on kubectl and thereby on the target cluster configuration of kubectl.

A generic resource e.g. kubernetes_manifest must fully integrate into the configuration of the underlying Terraform provider and not rely on any external executable such as kubectl. In particular, the functionality of kubectl apply could be reproduced in a generic resource while taking the target cluster configuration, endpoints, credentials etc. of the Kubernetes provider. The resource must also be capable of detected changes between the actual and the desired state on a basis of the manifest.

A problem of a generic resource would be that it would not be possible to define the fields natively in HCL. To my understanding, Terraform requires all fields of a resource to be defined explicitly in the implementation. For this reason, the manifest must be provided in either YAML or JSON format inside a field of the resource. Terraform would be able to detect changes based on the content of this manifest field.

As a proposal, a generic kubernetes_manifest could be modelled as in the following example. The usage would be similar to the terraform-provider-k8s, but without the disadvantages mentioned above.

locals {
    nginx_name = "nginx"
    nginx_replica_count = 3
    nginx_image = "nginx"
    nginx_version = "1.7.9"
}

resource "kubernetes_manifest" "example_deployment" {
    manifest = <<EOT
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: ${local.nginx_name}-deployment
  labels:
    app: ${local.nginx_name}
spec:
  replicas: ${local.nginx_replica_count}
  selector:
    matchLabels:
      app: ${local.nginx_name}
  template:
    metadata:
      labels:
        app: ${local.nginx_name}
    spec:
      containers:
      - name: nginx
        image: ${local.nginx_image}:${local.nginx_version}
        ports:
        - containerPort: 80 
EOT
}

As an alternative, the manifest property could be populated from an external file.

resource "kubernetes_manifest" "example_deployment" {
    manifest = "${file("manifests/nginx-deployment.yaml")}"
}

To extend the list of pro arguments:

• Easier transition from management of Kubernetes resources via kubectl to terraform
• Allows to manage Custom Resources and thereby avoid the effort to implement support for custom resources explicitly

[tmaier]

For an MVP, I could also live with a small dependency to kubectl to be available on the system, as long as I can provide the cluster configuration via the provider.

[radeksimko]

Hi folks 👋
thanks for raising this question.

Supporting YAML/JSON was considered before (the very first proposal was exactly that) and during the initial implementation of this provider and we decided _not_ to do it. For that reason I'm going to close this issue as we have no intentions of implementing it nor accepting PRs which accept raw configs in JSON/YAML.

Feel free to use @ericchiang's provider or kubectl if you wish.

Full discussion: https://github.com/hashicorp/terraform/pull/3132

TL;DR version (summary): https://github.com/hashicorp/terraform/pull/3453#issuecomment-189733008

This allows to support alpha and beta resources of kubernetes

We plan to support alpha/beta resources eventually, but those are likely going to be implemented the same way as all existing resources for the reasons mentioned under the above links.

Less maintenance necessary at kubernetes provider

We plan to address that on a slightly different level, we might generate some or most of the code that is currently maintained manually. This is however a _long_-term goal, not something we're currently looking into - just to set expectations.

Thanks for understanding and sorry for (potentially) bad news.

[dominik-lekse]

The idea of code generation sounds promising especially in cases of well defined/specified APIs such as the K8s API. Is there a more general approach Hashicorp is currently working on such as resource/provider code generation from Swagger API definitions etc? From such an approach I am sure more providers could benefit.

Also in the light of the counter arguments, in my opinion it is valuable for the provider to support a generic resource as fallback mechanism. The documentation can state clearly which of the usual Terraform features to expect and which are not supported.

[madmod]

Now that Kubernetes it being extended with Custom Resource Definitions extensively by common projects (Istio, Helm 3.0, and others) can this issue be revisited?

I don't think that it will be feasible for this provider to implement every CRD which people may want to use. Also the Kubernetes APIs change relatively quickly and updating this provider to match the latest features is impractical in my opinion.

This is a very significant limitation for me as my use case is to use Helm for deploying services and to manage the k8s resources which configure those services in Terraform.

[bukzor]

In GKE, custom resources are used for configuring the details of load balancing (see BackendConfig), and as a Kubernetes application developer, I plan to use the controller/custom resource pattern throughout my app.

Without any support for custom resources, this provider is unusable, for me (and at least twelve others, judging from the reactions above). The answer can't be to bake in knowledge of all custom resources because that's not a enumerable set -- many of those custom resources are application-specific.

Please don't construe this as an argument for heredocs piped to kubectl apply; I agree that's worse albeit more usable. I think we can find a happy medium wherein custom resources look very much like the "fully implemented" resources, and we use a very similar deployment mechanism for custom resources as everything else, with a very minimal amount of baked-in knowlege.

[yves-vogl]

This is also an issue with EKS to apply the k8s CNI update. One of my colleagues (@thommaa) also encountered problem with Helm Chart and certmanager.

[laghao]

Up! issue needs to be fixed for multiple deployment use cases.

[maltefiala]

@ericchiang's provider is archived by now.

[romankuzmik]

any reason this issue was closed? was it addressed?
tons of helm's charts nowadays require you to apply CRDs. we need a proper support for those in terraform.

[bukzor]

For those following this issue, these are the _open_ issues representing the same issue:

• #183 "Ability to deploy CRD's"
• #215 "Feature Request: Custom Resource"

---

The answer is here: #215 (comment)

@alexsomesan commented on Nov 14, 2018
We do want to support custom resources (and generic resources in general). We've been discussing various ways to approach this for a while. The main challenge with the current state of things in Terraform is achieving a diff-ing behaviour that is consistent with current Terraform resource diffs. The plan is to wait for Terraform 0.12 to land first and try to make use of some of its upcoming enhancements in implementing such a resource.

[sagikazarmark]

@maltefiala we (Banzai Cloud) will continue supporting @ericchiang's terraform-provider-k8s

We also plan to improve the provider (for example by implement a better diff check). Follow the issue tracker for more information!

[kim0]

Terraform already supports azurerm_template_deployment, which provider an escape hatch allowing me to use any yet unsupported Azurerm resources. Why can't the same be done for k8s. Plus the eco-system of k8s is growing rapidly, and not every yaml file is going to be rewritten to .tf

[theomessin]

Any update on this?

[kim0]

@theomessin .. I am currently using (with success) https://github.com/banzaicloud/terraform-provider-k8s

[sagikazarmark]

As far as I can tell the most compelling reason not to support any manifest in the official provider is that Kubernetes can change the objects themselves after sending them to the API server, either by the API server itself or through mutating webhooks which would mean that terraform wouldn't work optimally, there would also be a diff with the remote state.

On the other hand sometimes it's just unavoidable to use Terraform, therefore we support and use the forked version.

[ralf-berger]

As far as I can tell the most compelling reason not to support any manifest in the official provider is that Kubernetes can change the objects themselves after sending them to the API server

ignore_changes – “In some rare cases, settings of a remote object are modified by processes outside of Terraform, which Terraform would then attempt to "fix" on the next run. In order to make Terraform share management responsibilities of a single object with a separate process, the ignore_changes meta-argument specifies resource attributes that Terraform should ignore when planning updates to the associated remote object.”

[hbensalem]

Any chance to reopen this issue ?

[hermanbanken]

@sagikazarmark: As far as I can tell the most compelling reason not to support any manifest in the official provider is that Kubernetes can change the objects themselves after sending them to the API server, either by the API server itself or through mutating webhooks which would mean that terraform wouldn't work optimally, there would also be a diff with the remote state.

That is why kubectl apply uses the last-applied-configuration. It allows you to diff against what you applied last time. Preferably Terraform would be compatible with kubectl apply -f, or even better kubectl apply -k. Combined with the Helm provider and a kubectl apply provider that would be my CloudOps heaven 😍 .