Terraform-provider-kubernetes: Feature Request: Custom Resource

[timthesinner]

Enabling the K8S provider to apply and manage the lifecycle for custom resources has a number of advantages:

• Allow terraform to bring an entire K8S cluster under management, without waiting for formal resource models.
• Allow terraform to manage resources for custom APIs.

Terraform Configuration Files

Inline Configuration

resource "kubernetes_custom" "cluster-issuer" {
  apiVersion = "certmanager.k8s.io/v1alpha1"
  kind       = "ClusterIssuer"

  metadata {
    name = "lets-encrypt-prod-issuer"
  }

  set {
    name = "spec"

    value = {
      acme = {
        email  = "[email protected]"
        server = "https://acme-v02.api.letsencrypt.org/directory"

        privateKeySecretRef {
          name = "lets-encrypt"
        }

        http01 {}
      }
    }
  }
}

Inline configuration

locals {
  wildcard-cert = <<WILDCARD_CERT
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
spec:
 secretName: ingress-wildcard-tls
 issuerRef:
   name: '${kubernetes_custom.cluster-issuer.metadata.name}'
   kind: ClusterIssuer
 dnsNames:
   - '*.example.com'
 acme:
   config:
     - http01:
       domains:
         - '*.example.com'
WILDCARD_CERT
}

resource "kubernetes_custom" "wildcard-ingress-certificate" {
  metadata {
    name      = "ingress-tls-wildcard"
    namespace = "kube-system"
  }

  values = "${local.wildcard-cert}"
}

File based configuration

resource "kubernetes_custom" "default-ingress-certificate" {
  metadata {
    name      = "default-ingress-tls-wildcard"
    namespace = "kube-system"
  }

  values = [
    "${file("default-certificate.yaml")}",
  ]
}

Expected Behavior

Define and manage the lifecycle of any K8S resources using the kubernetes provider.

Actual Behavior

This resource type is not supported

References

Open feature requests for formal models

• #14
• #212

[alexsomesan]

Thanks for documenting this request.

We do want to support custom resources (and generic resources in general). We've been discussing various ways to approach this for a while. The main challenge with the current state of things in Terraform is achieving a diff-ing behaviour that is consistent with current Terraform resource diffs. The plan is to wait for Terraform 0.12 to land first and try to make use of some of its upcoming enhancements in implementing such a resource.

[timthesinner]

Sounds great, thanks for the quick response.

[bryanlarsen]

The most obvious workaround is https://github.com/ericchiang/terraform-provider-k8s

But what I'm doing is creating a simple helm chart with my custom resources and using the helm provider, providing a local path as the chart name. That makes sense for us because we're using the helm provider to install a bunch of other stuff too.

[nfisher]

Hi @alexsomesan given HCL is JSON compatible what are the big obstacles you see to adopting it prior to 0.12 landing? Or is it more a matter of prioritisation and bandwidth?

[pdecat]

This was also discussed in #195

[benishak]

we at mobfox a year ago wrote a provider that allows the user to create Custom Resources and even despites the needs to create Custom Provider.

Now we are open sourcing it so Check it out here:
https://github.com/mobfox/terraform-provider-multiverse

you can even use AWS Lambda or execute a function locally in any language you like to manage your resources, it also keep state of your resource, so you can delete, read, update them too. It creates a resource, so it is not like External Data and behaves exactly like Custom Resources in AWS CloudFormation

[awfm9]

This is going to be needed to use the upcoming v2 release of traefik as ingress, as they are using custom resources to improve configuration: https://docs.traefik.io/v2.0/providers/kubernetes-crd/

[aelbarkani]

@alexsomesan now that 0.12 has landed, are there any plans to implement this feature ? it seems CRDs is the way to go according to many Kubernetes experts, and Redhat is investing a lot on it through Openshift operators. so it seems to me that it will become very soon a must have feature for Terraform I think

[kalebwalton]

Agree. Operators for 'day 2 operations' are becoming a big thing in IBM / RedHat. We're using the 'helm_resource' right now, but soon we're going to want to use an Operator CRD to manage our applications. It'll be a shame if we have to patch it into our orchestration outside of Terraform.

[davisford]

I wish the official Terraform k8s provider supported things like CRD, Jobs, and other smaller tasks that currently are unsupported, but my guess is it won't come any time _real_ soon. I've had some luck using the nice community provider that deals with raw yaml.

The original author indicated he does not have time to push it forward any farther, so I forked it and updated it for Terraform 0.12. It is working very nicely for us. We use the Couch infrastructure which does use an operator that pushes its own CRD. So far so good. The current state of the provider needs some work with respect to diffs and patches, but it does what we need it to do.

Fork is here that updates for Go modules and [email protected]

[gabzim]

Is there an estimate ETA for this feature? As more deployments require CRDs the number of things we can deploy to k8 with terraform grows smaller and smaller (I'm looking at you cert-manager)

[alexsomesan]

We are committed to having complete coverage of the Kubernetes API in this
provider. Jobs and CronJobs are going to be supported soon (PRs already
open). Other smaller missing parts will follow soon after.

There is no set ETA for CRDs as of now.
The main impediment is fundamental limitations in the Provider SDK (the
framework that connects the provider to Terraform core). These have been
mentioned as blockers by the authors of most raw manifest / CRD resource
attempts so far.

The good news is that we are experimenting various alternatives to work
around these limitations and working with the developers of Terraform core
to make sure however we end up implementing CRDs does provide the same UX
in regard to diffs/patching and planning behavior that users rely on with
all the other resources.

Finally, we are not looking to merge any “MVP” type of change that only
allows for creation but not manipulation / patching of CRDs resources. Also
“planning” needs to behave as expected and show the diff of changes within
the CRD manifest, not just the entire “blob”.

Hope this clarifies thing a bit.

On Sat 15. Jun 2019 at 21:27, Gabriel Zimmermann [e-mail]
wrote:

Is there an estimate ETA for this feature? As more deployments require
CRDs the number of things we can deploy to k8 with terraform grows smaller
and smaller (I'm looking at you cert-manager
https://github.com/jetstack/cert-manager)

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/terraform-providers/terraform-provider-kubernetes/issues/215?email_source=notifications&email_token=AAIL5G2FXNYAFFCKXMJE7RDP2U7CHA5CNFSM4GDZFLLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXY6S5Q#issuecomment-502393206,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAIL5GZOVC6UEEWUJ5AS65TP2U7CHANCNFSM4GDZFLLA
.

>

— Sent from my phone.

[curtbushko]

Adding to use cases for a simple CRD option. Basically, we only care if it just passes everything onto the kubernetes API and saves the state.

We are using the istio ingressgateway and to do so you need the VirtualService kind.

I would really like to use the kubernetes provider for setting up our services instead of helm.

[arnonki]

This is a must feature it prevents us from using terraform in production

[benishak]

@arnonki
check this, it might be helpful
https://github.com/benishak/terraform-provider-multiverse

[pescetto]

@alexsomesan I know it's only been a couple months since your last update, but any chance this can get added this year? I feel like CDR's are the biggest missing feature from the kubernetes terraform provider. Thanks again and I'm really looking forward to CDR's in the provider!

[zdevi]

Hi, any news on this ? Did they abandon implementing CRD's ??? This is a vital feature for me, CRD's are used to configure backends on GCP... Please let us know what is happening, and if there is an ETA, i see it has been more than a year people are asking for that feature.

[alexsomesan]

We have not abandoned CRD support.
However, it's a radical change that requires intervention into multiple layers of the provider SDKs (and overriding some of them). It going to be released as a separate experimental provider in Alpha state when we have it working enough for testing. We hope to be able to demo it after the end of Oct.

[zdevi]

So we are talking about Years just for this feature ? I can see it has already been discussed in early 2018. It is becoming hard to continue using kubernetes provider in production as it is lacking a lot, no custom configuration possible without CRD's.... I am gona be forced to find another solution, else than the official provider, that's sad.

[tombh]

I hope this isn't off topic, but what do people consider as alternative options? I'm thinking more of the declarative approaches like Cloudformation and maybe Pulumi?

I haven't really researched much recently but it still seems to me that Terraform is the best we have, even without CRD support. Is there something inherent about Terraform's design, or declarative Infrastructure As Code frameworks in general that makes it inextricably difficult to maintain feature parity with Kubernetes?

I believe it when I hear that implementing CRD support is a significant task. But I'm happy to wait and very grateful for the efforts being made because Terraform still seems like the best approach out there. Sure I have a few YAML files here and there that I need to manually manage but it's tiny compared to the swathes of HCL. It boggles my mind every time I see Kubernetes tutorials referencing YAML snippets, how can people be productive like that!?

[benabineri]

I believe (although I may be wrong) that Pulumi uses the Terraform providers underneath so you'd probably have the same issue there.

[stuart-c]

We use Helm charts via the Helm Terraform provider, with small pieces handled directly through this provider where it doesn't make sense for a Helm chart.

[curtbushko]

To get around the lack of CRDs, we use this provider to apply raw yaml to kubernetes.

https://github.com/lawrencegripper/terraform-provider-kubernetes-yaml

[pschulten]

Thanks @curtbushko I guess this is better than my local-exec kubectl apply one

[jemc]

Fun times wading through a sea of various deprecated third-party kubernetes providers, all of which have their own quirks and issues.

I ended up settling on a different approach for my temporary workaround while we wait for the Terraform folks to bring us a shiny official solution:

I am using this third-party "shell" provider plugin, which provides more flexibility than the local-exec approach, and lets us actually use kubectl diff to detect changes and kubectl apply for both create and update-in-place.

Here's a working snippet from my code. Just find/replace the string ingress_nginx with the name of your own YAML file you want to apply.

locals {
  ingress_nginx_yaml = file("ingress_nginx.yaml")
}

# This provider is from scottwinkler/terraform-provider-shell
# You can download and install the "shell" plugin binary from:
# - https://github.com/scottwinkler/terraform-provider-shell/releases/tag/v0.1.
# Refer to these Terraform docs for info about installing plugins:
# - https://www.terraform.io/docs/plugins/basics.html#installing-plugins
#
# We use it to execute local commands because it provides features
# that we need that Terraform's official `local-exec` doesn't have.
#
# Specifically, it allows us to run a script to detect when a resource
# needs to be updated, and it allows us to update-in-place rather than
# always running create-then-destroy to do an update.
# This lets us leave the idempotence to kubernetes and kubectl.
#
# We need to go to these lengths only because Terraform's official
# Kubernetes provider doesn't support arbitrary manifests.
# When it does eventually support them, we can use it instead of this.
# Subscribe to this issue ticket if you want to stay in the loop:
# - https://github.com/terraform-providers/terraform-provider-kubernetes/issues/215

provider "shell" {
  version = "0.1.1"
  alias = "ingress_nginx_yaml"
}
resource "shell_script" "kubernetes_ingress_nginx_yaml" {
  provider = shell.ingress_nginx_yaml
  working_directory = path.module
  environment = {
    KUBECTL_MANIFEST = local.ingress_nginx_yaml
  }
  lifecycle_commands {
    # Update the "taint_trigger" in the outputs of the local state
    # when `kubectl diff` shows that the remote resources differ.
    read = <<SCRIPT
      if echo "$KUBECTL_MANIFEST" | kubectl diff -f -
      then cat >&3 # keep the same state as received from stdin
      else echo '{"taint_trigger": "'$RANDOM'"}' >&3 # state change
      fi
    SCRIPT

    # Use idempotent `kubectl apply` for both create and update.
    create = <<SCRIPT
      echo "$KUBECTL_MANIFEST" | kubectl apply -f -
    SCRIPT
    update = <<SCRIPT
      echo "$KUBECTL_MANIFEST" | kubectl apply -f -
    SCRIPT

    # Use `kubectl delete` for delete.
    delete = <<SCRIPT
      echo "$KUBECTL_MANIFEST" | kubectl delete -f -
    SCRIPT
  }
}

[asce0705]

We have not abandoned CRD support.
However, it's a radical change that requires intervention into multiple layers of the provider SDKs (and overriding some of them). It going to be released as a separate experimental provider in Alpha state when we have it working enough for testing. We hope to be able to demo it after the end of Oct.

@alexsomesan I would like to ask the current progress of this issue. Thank you! I think many people are looking forward to it too.

[nikolay]

One year with no solution - not even an official workaround! Wow!

[JulienBreux]

Hi @alexsomesan!

Do you have some info about the progression of discussions? How can we help you?

🙏 🙏 🙏

[nikolay]

I cannot believe that a provider like terraform-provider-shell is not even a standard one! Keep downvoting, but the frustration that you can't do any modern Kubernetes setup with Terraform today without using third-party providers speaks for itself!

[nikolay]

Why downvote again? It is simply not acceptable to have local-exec not being idempotent. Yes, you can do CRD with local-exec, but then you need to manually delete (and update) the resource definitions.

[tombh]

It's hard to understand why you feel it's ok to put such pressure on the developers here? Yes of course Hashicorp is a for-profit company, but you didn't pay for the code in this repo. Indeed it is perfectly possible for you to learn Golang and Kubernetes' internals so that you are the one to complete this CRD feature. Why haven't you? How would you feel if I told you that it was unacceptable that you hadn't found the time to do that?

It comes across as if you have little to no experience with open source culture. Speaking on behalf of others, so I might not represent everyone, but to me your downvotes are because your attitude appears to come across as insensitive and entitled. You seem not to empathise with the very real human beings behind this project who are generously writing complex software for us for free. The only place your attitude might be considered appropriate is if this was the support forum for a paid product.

Indeed I might even suspect that your comments slow down the timeline for the shipping of this feature.

[chrw]

What @tombh said! How do your comments improve the situation in any way?

[JulienBreux]

What @tombh said! How do your comments improve the situation in any way?

/cc @tombh

Dear community friends!
IMHO, my comment isn't to apply a strong pressure to anybody. 🤗
It's just to provide/propose my hands, fingers and brain to help. 😊 I do swear.
Because I know that it's really hard to have the same velocity that the Kubernetes Core API.

[awfm9]

More active communication on progress and plans would be nice.

[mbarrien]

As someone who has put my fingers in the code and even put out a PR for some partial CRD support (https://github.com/terraform-providers/terraform-provider-kubernetes/pull/428), I will say this: I agree with both sentiment of both @nikolay and @awishformore . I would love to help, but after having gone in myself, it's obvious LARGE architectural changes are needed in underlying Terraform core in multiple ways to properly support this (dynamically generating resources at runtime to enforce types either during provider initialization or after the provider is initialized, supporting recursively defined schemas, ability to have a provider call an external service to perform a diff). It can't solely be done at the provider level because Kubernetes concepts just mismatch the very core concepts inside of Terraform core.

However, such large changes are something that really can only be done under Hashicorp's purview. With this understanding, the lack of communication is frustrating.

As for those who may criticize this comment and say "how is this comment improving the situation?", this is my call to ask for Hashicorp to make their roadmap more public. Yes, we're not paying for support, but other open source projects have a more visible development plan or working group or chat area to be able to talk about issues, or to hash out these architectural issues in a committee (since this kind of thing really can't be done by a single person). The current process is really blackboxed, and we have no idea whether it's even prioritized, or being worked on, or if we should be moving on to another tool. We have no idea whether our feedback is affecting this roadmap, and it become harder to believe it is when publicly stated deadlines pass by (e.g. "the demo by October" above) without any feedback as to what happened to the proposed work. On the flipside, being more open allows us to make our own development decisions, provides a forum to help us let Hashicorp know what features the community wants worked on next (which can help get more paying customers), and lets the open source community know the best way they can contribute whether through code or advocacy.

I get it, Hashicorp is a private company so they don't owe us any of this and may instead opt to not even convey any timelines at all in the future, but in that case it also can't hide behind the "we're open source" either.

[nikolay]

The same code base is being used in Terraform Enterprise so you can't deploy any operator with Terraform without nasty workarounds even if you're a dearly paying customer. So paying or not, you're still without luck and you need to use different deployment mechanisms and cannot solely rely on just Terraform, which defeats the purpose of using Terraform and not, let's say, CloudFormation or ARM templates! I've been using a weird mixture of Kustomize + kubectl, Terraform, and shell scripts given the limitations. We're almost 2020, and we can't have one tool to deploy hybrid infrastructure?! This is sad with or without the coat of sugar! I've tried Pulumi, which for certain things uses Terraform under the hood, and it seems the thing closes to the infra-as-code nirvana. HashiCorp spent so much time working on HCL 2.0 and it's still no match for even the worst programming language! I'm not sure why they didn't go with Skylark or Cue, honestly! Even the old Jsonnet would have been a better choice, but, no, they had to come up with something totally different!

[erlacherl-city]

For anyone just looking for a quick-and-dirty bodge, here's what i just implemented. Note that this is not robust, production grade, or fit for purpose for anything, furthermore it leaks your credentials all over stdout. But it might get you started.

# You already have a kubernetes provider set up with credentials - in this example it comes from aks
provider "kubernetes" {
  host                   = data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.host
  username               = data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.username
  password               = data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.password
  client_certificate     = base64decode(data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.client_certificate)
  client_key             = base64decode(data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.client_key)
  cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.cluster_ca_certificate)
}

# and you have a .yaml file you want to apply to that cluster
data "local_file" "aad_deployment_file" {
  filename = "${path.module}/aad-deployment-test.yaml"
}

# here's how
resource "null_resource" "aad_deployment" {
  triggers = {
    manifest_sha1 = sha1(data.local_file.aad_deployment_file.content)
  }

  provisioner "local-exec" {

    environment = {
      KUBE_CLIENT_CRT = data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.client_certificate
      KUBE_CLIENT_KEY = data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.client_key
      KUBE_CLIENT_PASSWORD = data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.password
    }

    command = <<EOC
echo ">>> Starting Prep"

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
chmod +x ./kubectl

echo $KUBE_CLIENT_CRT | base64 -d >>kube_client.crt
echo $KUBE_CLIENT_KEY | base64 -d >>kube_client.key
echo ${data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.cluster_ca_certificate} | base64 -d >>kube_ca.crt

echo ">>> Prep done, running kubectl"

./kubectl \
--server ${data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.host} \
--certificate-authority kube_ca.crt \
--client-certificate kube_client.crt \
--client-key kube_client.key \
--username ${data.azurerm_kubernetes_cluster.my-cluster.kube_config.0.username} \
--password $KUBE_CLIENT_PASSWORD \
apply --dry-run -f -<<EOF
${data.local_file.aad_deployment_file.content}
EOF

echo ">>> kubectl done, cleaning up"

rm kube_client.crt
rm kube_client.key
EOC
  }
}

[tjhiggins]

@erlacherl-city take a look at the shell provider - first ref here:
https://github.com/terraform-providers/terraform-provider-kubernetes/issues/215#issuecomment-542831683

I recently successfully tested using it to deploy an istio gateway/virtual service. Also not recommended for production, but it seems to get the job done. You also pretty easily make this a go module that takes a kubernetes manifest.

main.tf

# https://github.com/terraform-providers/terraform-provider-kubernetes/issues/215
provider "shell" {}

variable "namespace" {}

data "template_file" "istio" {
  template = "${file("${path.module}/istio.tpl.yaml")}"
  vars = {
    namespace = "${var.namespace}"
  }
}

resource "shell_script" "kubernetes_istio_yaml" {
  environment = {
    KUBECTL_MANIFEST = "${data.template_file.istio.rendered}"
  }
  lifecycle_commands {
    # Update the "taint_trigger" in the outputs of the local state
    # when `kubectl diff` shows that the remote resources differ.
    read = <<SCRIPT
      if echo "$KUBECTL_MANIFEST" | kubectl diff -f -
      then cat >&3 # keep the same state as received from stdin
      else echo '{"taint_trigger": "'$RANDOM'"}' >&3 # state change
      fi
    SCRIPT

    # Use idempotent `kubectl apply` for both create and update.
    create = <<SCRIPT
      echo "$KUBECTL_MANIFEST" | kubectl apply -f -
    SCRIPT
    update = <<SCRIPT
      echo "$KUBECTL_MANIFEST" | kubectl apply -f -
    SCRIPT

    # Use `kubectl delete` for delete.
    delete = <<SCRIPT
      echo "$KUBECTL_MANIFEST" | kubectl delete -f -
    SCRIPT
  }
}

istio.tpl.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
  namespace: ${namespace}
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: virtual-service
  namespace: ${namespace}
spec:
  hosts:
  - "*"
  gateways:
  - gateway
  http:
  - match:
    route:
    - destination:
        host: <SERVICE_NAME>
        port:
          number: 80

[ralimi]

I've started using https://github.com/gavinbunney/terraform-provider-kubectl to handle custom resources.

[lukasmrtvy]

Did anyone consider kustomize ? Seems that currently no 3rd party k8s manifest provider does not support it. Its worth to have support for kustomize, because its native and its implemented in kubectl already.

[hypery2k]

any news about this?

[aambert]

Bump BUMP, all the deployement use custom ressource traefik 2.0 aso....

[LuanAraldi]

@alexsomesan can you give us a report on how things are going for support on custom resource definitions?

[lukasmrtvy]

I had success with external data source that compares checksum of local annotation ( kubectl.kubernetes.io/last-applied-configuration - generated with kubectl create -f - ) with server annotation ( generated with kubectl get -f - ). Is possible to use jmespath and jq ( create proper array, sort and make checksum ) to compare local/server version -> datasource ouput, that triggers null_resource which has create_before_destroy flag that will prevent recreating resource if is tainted...
Its working great. ( I am using another external datasource to build my kustomization output )

Addtional info:

• minus: kubeconfig must be presented as file ( another resource -> local_file ,if its generated with TF )
• minus: its not possible to pass sensitive content to null_resource, only as local file path
• plus: is possible to use templatefile() inside null_resource
• plus: is possible to use ignore_changes = [ triggers["myvar"] ] for storing vars without triggering null_resource
• wet dream: null_resource should have some sensitve input field
• wet dream: something like templatecontent() interpolation should exists ( I am aware of template_file datasource )

Notable providers:

• https://github.com/gavinbunney/terraform-provider-kubectl ( diffs are visible in plan/apply/destroy, CRDs are not destroyed, provider is not waiting untill resource is destroyed, same applies for apply, does not do validation)
• https://github.com/banzaicloud/terraform-provider-k8s ( diffs are visible in plan/apply/destroy, bad error handling, seems that cert-manager is not even possible to deploy -> deployment requires validation=false flag, but it not even visible with TF_LOG=debug log level :D )
• https://github.com/kbst/terraform-provider-kustomize ( not finished, kubeconfig must be present prior to apply )

Usually I am fine with secrets -> I am using remote state file backend, but exposed sensitive values in plan/apply/destroy are nogo for me especially if You are using tools like atlantis, or You need to ship logs "somewhere" for auditing purposes...

[awfm9]

I have stopped using or promoting terraform. I appreciate the effort required to implement new features, but I'm sorely disappointed by the complete lack of communication, especially considering that other people were willing to contribute to getting this done. This is not how you run an open source community/project.

[nikolay]

@awishformore Same here! The true infra-as-code project today is https://github.com/pulumi/pulumi, which uses Terraform under the hood where there's no alternative, but has no limitation of not tolerating best practices from years ago. Terraform is infra-as-config anyway as HCL configs are not real code anyway.

[alexsomesan]

We have not abandoned CRD support.
However, it's a radical change that requires intervention into multiple layers of the provider SDKs (and overriding some of them). It going to be released as a separate experimental provider in Alpha state when we have it working enough for testing. We hope to be able to demo it after the end of Oct.

This statement still holds true.

Support for CRD is being worked on as a separate provider. It will not only support CRD resource, but we aim for a completely generic provider that out-of-the box will support the entire range of resources and attributes of the K8s API going forward.
Because this work side-steps the SDK to allow for dynamic resource schemas, it has several times run into uncharted territory when it comes to interactions with Terraform core. This is what is causing the delays.

I am ironing out the issues to get to an alpha version which will be published as soon as it works predictably end-to-end. I'm estimating a matter of weeks.

Oh, it also accepts raw YAML manifests :)

[hypery2k]

awesome !

[curtbushko]

@alexsomesan Alex, thank you for the response. This is great news.

Your design sounds similar to what I thought of the way to do it. I am guessing that you hold the CRD yaml file in state so that you can diff it just in case the CRD changed too?

[alexsomesan]

Actually no YAML goes into the state file.
Most important invariant of this endeavor is that we can present a per attribute diff of all resources, including CRDs. That isn't possible if we treat the resource as a monolithic YAML blob. Instead, it gets parsed into Terraform specific data types and stored as native state.
I'm making use of features specific to Terraform 0.12 that have not yet been exposed in the provider SDK. That in turn means the overall code structure of the provider is very different than what we are used to in the current providers.

[curtbushko]

Even in an alpha state, I'd be glad to pull it into our system and test it.

I am currently interacting with 3 Istio CRDs and we are using the k8sraw provider to just apply raw yaml. It fails once in a while so anything would be a step up.

[alexsomesan]

I'll put it up for preview / review as soon as all CRUD actions work correctly.
I'd really like to get user feedback as early as possible too.

[Nainterceptor]

Our workaround to prevent using "less maintained providers" is to wrap yaml into an helm chart, and use the helm provider to deploy the chart. Not beautiful, not short way, but working.

[Fryuni]

@Nainterceptor 🤯 I haven't even considered that. It's clever and can be automated. Good as an intermediary step while we migrate from Terraform.

[charles-d-burton]

I'm down to test this as well. This is the one thing we really need as a huge portion of our stack uses CRD/Operators.

[nikolay]

Years pass, and we still have to struggle using Terraform with the most basic scenarios - all kinds of secrets pop up here and there within the state files (and not .terraform/ as one would hope!) and basic Kubernetes resource types are not supported.

[nikolay]

Sorry, forgot to add to my venting how nontrivial is to install custom providers! This coming from the fathers of Vagrant, which allowed you to install plugins via the CLI. Even if I use any of the suggested ugly workarounds mentioned above, none of them are going be forward-compatible and they all seem to be just that - workarounds, and it will be a bumpy ride, and a wasted effort.

[pkrishnath]

any update on this?

[gabzim]

Please refrain from adding further venting comments, I think by now it's clear this is an issue that needs resolution and adding anything other than objective progress just pollutes the issue (and we get notifications on it that are really pointless). Just thumbs up the issue if you need to voice support for this feature.

[hazcod]

How would this relate to the helm provider, or e.g. istio moving to an operator model?

[cppforlife]

for those still wanting to have a generic k8s terraform provider, ive created https://github.com/k14s/terraform-provider-k14s. it uses get-kapp.io -- k8s specific deployment tool -- that's really good at calculating pending k8s changes, applying them, and waiting for them to succeed.

here is an guestbook app example deploy output that includes a diff for resources that are going to be created: https://github.com/k14s/terraform-provider-k14s/tree/master/examples/guestbook

feel free to drop by #k14s slack channel in kubernetes slack to chat.

[Jefski14]

I'll put it up for preview / review as soon as all CRUD actions work correctly.
I'd really like to get user feedback as early as possible too.

Any news or an estimation on how long it's going to take for a preview/alpha version? :)

[ccgagnon]

I'll put it up for preview / review as soon as all CRUD actions work correctly.
I'd really like to get user feedback as early as possible too.

Any news or an estimation on how long it's going to take for a preview/alpha version? :)

@alexsomesan that would be very nice to have some details. You mentioned you are working on a separate provider. May we have more details please? ^^

[Sytten]

What are the next steps since its been in the research phase for a while now? Anything the community can do to speed up the process? Even a basic implementation that would replace the resource everytime would be good enough to start with.

[drebes]

https://www.hashicorp.com/blog/deploy-any-resource-with-the-new-kubernetes-provider-for-hashicorp-terraform/

[BarnumD]

@drebes That looks great. We're using terraform cloud for our deploys. Is there a way to pull in a provider like that in the cloud that's not from the public repository?

[Sytten]

@BarnumD You have to commit it in the repo in .terraform.d/plugins. See https://www.terraform.io/docs/plugins/basics.html.

[BarnumD]

@Sytten Thanks. I had to put it in terraform.d/plugins (without the dot). I also had to run git update-index --chmod=+x .\terraform.d\plugins\linux_amd64\terraform-provider-kubernetes-alpha otherwise terraform cloud would complain about permissions. Didn't see that in the link you provided.

[alexsomesan]

Support for CRD is being rolled out in the new alpha provider. You can try out the alpha provider here: https://github.com/hashicorp/terraform-provider-kubernetes-alpha

Once it's been graduated out of alpha state, it's codebase will be merged into this provider and the features will become available here as additional resources.

[Sytten]

@alexsomesan Is there a timeline/roadmap for that to happen?

[nadworny]

Do you happen to know more about the roadmap?

[0dragosh]

Also interested in a roadmap so I know what provider to choose. HashiCorp is really lagging behind on this.

[mcfedr]

I've found this provider to do a really great job with any kubernetes resources this one cannot handle, https://registry.terraform.io/providers/banzaicloud/k8s/latest/docs/resources/manifest - and now its really easy to use with the registry.

[nadworny]

thanks @mcfedr , I'll check it out

[jurgenweber]

if you weren't aware, I think they are deprecating this provider in favour of: https://github.com/hashicorp/terraform-provider-kubernetes-alpha

[aareet]

Hi folks! We're currently evaluating the feasibility of merging the manifest resource into our official provider. We are also working on some SDK improvements that are required to support the alpha provider. We'll keep our roadmap up to date with timelines and changes as soon we have them.

[pawelprazak]

Any news on the roadmap?

CRDs are GA since Kubernetes 1.16 and almost everyone needs them now that every cloud provider and project use some kind of operator.

Btw. the plugin is available in the registry, I was able to install it on tf 0.13

terraform {
  required_providers {
    kubernetes-alpha = {
      source = "hashicorp/kubernetes-alpha"
      version = "0.2.1"
    }
  }
}

provider "kubernetes-alpha" {
  # Configuration options
}

https://registry.terraform.io/providers/hashicorp/kubernetes-alpha/latest/docs/resources/kubernetes_manifest

[viceice]

Also a workaround for deploying simple yaml manifests: https://registry.terraform.io/providers/gavinbunney/kubectl/latest

[vdahmane]

Hello, do you have ETA for this useful feature?

[jurgenweber]

yeah, in the end we abstracted away from TF deploying directly....

We have TF deploying ArgoCD apps via helm charts....

ArgoCD does the rest.

[yashbhutwala]

An update on the plans on the roadmap for this feature would help assess if/whether we should use kubernetes-alpha provider or not.

Info and links from @aareet's comment from last year has not been kept up to date, i.e.: https://github.com/hashicorp/terraform-provider-kubernetes/blob/master/_about/ROADMAP.md

Hi folks! We're currently evaluating the feasibility of merging the manifest resource into our official provider. We are also working on some SDK improvements that are required to support the alpha provider. We'll keep our roadmap up to date with timelines and changes as soon we have them.

[aareet]

Hi, the SDK improvements have now shipped and we have been working on integrating them into the alpha provider. As before, we continue to work to find a way to bring the alpha provider to a production-capable state.

[alexsomesan]

Quick update here: the recently released v0.3.1 of the Kubernetes Alpha provider has quite usable support for CRDs. I would encourage everyone to give that a try a provide us with feedback. That would help us to graduate that provider to GA as soon as possible.

[nomius]

Hi folks, any updates on moving this functionality to this (stable channel) provider?

Thanks!

[nikolay]

@alexsomesan "Usable" and "alpha" can't be used in one sentence!

[asaphe]

Hi! I love the Kubernetes Alpha provider and I'm using it here https://github.com/asaphe/terraform-keda to deploy a KEDA crd.

Would love to be able to use dynamic blocks for manifest.

When will the Kubernetes Alpha provider be merged with the Kubernetes provider? I would love to already start using it and plan ahead.