Terraform-provider-kubernetes: [Feature Request] Datasource equivilent of `kubectl get no`

[iridian-ks]

Terraform Version

0.12.8

Affected Resource(s)

(Theoretical)

• kubernetes.node(s)

Terraform Configuration Files

(Theoretical)

data "kubernetes_service" "ingress" {
  metadata {
    name           = "ingress"
    namespace = "ingress-system"
  }
}

data "kubernetes_nodes" "workers" {
  metadata {
    roles = [ "worker" ]
   }
}

resource "f5_blah" "loadbalancer" {
  pool = [
    for worker in data.kubernetes.workers :
    "${worker.ip}:${data.kubernetes_service.ingress.node_port}"
  ]
}

[dak1n1]

I did some investigation and ran this issue by @jrhouston . Offhand, we're concerned that querying Terraform for an IP address and adding it to a load balancer is a brittle configuration. It might be better to try using Kubernetes' Service LoadBalancer to automatically create and use load balancers of your preferred kind.

For example, a Service LoadBalancer works on AWS by automatically creating an Elastic Load Balancer associated with the Service. https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer

Each cloud provider will have its own type of load balancer available for Kubernetes to use in this way. There is even a way to use an F5 BIG-IP instance as this load balancer.

I found this article about the differences between using the F5 load balancer and AWS load balancers, in case that helps you decide. https://www.f5.com/services/resources/white-papers/load-balancing-on-aws-know-your-options

And it's possible to set up F5's own ingress controller to facilitate using the F5 load balancer.
https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/
https://clouddocs.f5.com/containers/v2/kubernetes/kctlr-ingress.html

I hope that helps. Please let me know if I've misunderstood the use case and we can evaluate it further.

[iridian-ks]

Hi @dak1n1

Thanks for the time you put into this.

Can you go deeper into why it might be brittle to query for node ip addresses? Generally, our ip addresses don't change unless new nodes are getting added or removed, which isn't often. In the case of a removal then f5 would take them out on a health check. Adding new nodes can be delayed until the next Terraform run without an issue. Right now, we are simply hard-coding ip addresses, but we could also shell exec a kubectl get no. A kubenetes_node data source could replace hard coded ip addresses or hacks.

As far as the f5 integrations go, they aren't great. For example, we use Gloo as our preferred ingress controller. The f5 integration _doesn't_ support Kubernete's Service LoadBalancer. Also, we don't want ClusterIP's polluting our f5 or even colliding since we have multiple clusters. This makes the f5 controller a non-starter.

I understand that provisioning external resources with kubernetes resources is preferred, but I still feel like this could be useful in the cases that a kubernetes integration doesn't exist but does in Terraform. Or, rather, better/more convenient in Terraform.

[jrhouston]

Thanks for opening this @iridian-ks.

In general I would consider a terraform configuration that you know is likely to drift from the state created when you run apply to be brittle. You have a config that you have to run periodically (with no changes) to keep things in sync, which isn't great. So I think what's being proposed here seems more like a way to work around the limitations of f5's integration with Kubernetes, rather than a pattern we would want people to follow.

If your last resort here is to statically assign IPs to the load balancer, is it possible for you to do this at the cloud provider level rather than via Kubernetes' nodes API? e.g. In AWS you could query the instances by tag and then get the IP addresses from there.

[iridian-ks]

Hi @jrhouston thank you so much for expanding on why this would be brittle and that definitively makes sense. We are not in a cloud environment at all. The nodes are brought up manually and are generally static, but could change with manual intervention (new server blades). Again, not cloud.

I was looking for a solution that could automatically sync node changes to the load balancer. From your explanation it sounds like Terraform isn't the right tool for that. We do use Ansible and I think we can get it done there and use it like a glue.

Thanks so much for the explaining this, it's much appreciated.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!