Terraform-provider-kubernetes: Support for running port-forward while Terraform is operating

[dpkirchner]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

It would be helpful to have a way to temporarily enable port-forwarding, perhaps as some sort of data resource, that would allow us to temporarily expose services in Kubernetes so that Terraform may send requests over that local port.

For example, let's say you're running Spinnaker in your Kubernetes cluster (as is typical). In order to use the Spinnaker provider, you need to be able to connect to Spinnaker's API gateway on port 8084. You could leave that port open 24/7, but it would likely be safer to only access it over an authenticated tunnel ala kubectl port-forward svc/spin-gate 8084, and only when necessary.

I'm sure this would be generally useful for other providers. SQL, for example.

Currently, I run the port-forward command in a loop in the background while terraform runs. Sometimes the tunnel is successfully creaetd before Terraform gets to the resources that require the tunnel, but most of the time it doesn't, and then I have to run terraform again to finish the entire plan.

Potential Terraform Configuration

data "kubernetes_service" "svc" { # or resource
  metadata {
    name      = "svc"
    namespace = "ns"
  }
}

data "kubernetes_port_forward" "svc" {
  namespace    = kubernetes_service.svc.metadata[0].namespace
  service_name = kubernetes_service.svc.metadata[0].name
  service_port = 8084 # or maybe strings for named ports
  local_port   = 9000 # optional
}

The data resource would need to be able to export the local port. If local_port isn't set perhaps the resource could allocate a random port.

The data resource would need to block until the service is reporting ready/referring to live pods (or some timeout is hit).

References

This is distinct from the existing provisioner + connection tunneling as it's for resources, not running specific commands on a remote server.

Example of port-forward use

Port forward API

Example of someone programmatically enabling port forwarding

[jrhouston]

This feature request has been open for a while and seems to have attracted a significant number of 👍 so I'd like to do two things:

1. Find out how people are working around this at the moment. Has anyone made something like this work by running kubectl port-forward inside a null resource, for example?

2. Collect any additional proposals for how this feature should work. I would propose that perhaps this should be implemented as a resource rather than a data source, as Terraform would be creating a tunnel not just fetching some data.

edit: We also have to figure out what problems we are going to see when setting up a tunnel, blocking until it's ready, and then passing along the details to the provider block of the downstream resources.

[lawliet89]

From a user POV, I think it would be best if the port forwarding is active from whenever it is possible to create the port-forwarding to the end of a Terraform operation.

The problem I can foresee with its implementation is that this does not really fit into the lifecycle of a Terraform resource. Not sure how it can be implemented with the SDK tools we have.

[hrvolapeter]

Hello everyone,
I was pointed here from slack where I was proposing a bit different solution

Use case I want to apply kubernetes resources to private clusters that are available over proxy currently best solution is probably to split terraform definition to stages first setting up bastion and then executing second stage with e.g HTTPS_PROXY=localhost:8888 terraform plan however this overly complicates whole terraform project structure.
I was thinking if the community would be willing to accept a new option for provider e.g

provider "kubernetes" {
  version          = ">= 1.11.0"
  load_config_file = false
  host                   = "https://${module.cluster.cluster_endpoint}"
  token                  = data.google_client_config.default.access_token
  cluster_ca_certificate = base64decode(module.cluster.cluster_ca_certificate)
  bastion_host           = localhost
  bastion_port           = 8888
}

Which would essentially allow to execute particular provider with proxy without affecting the other providers.
This can be then combined with a null_resource creating proxy connection gcloud compute ssh cluster-bastion --project cluster-x --zone europe-west1-c -- -L 8888:localhost:8888. Something similar is done for provisioners

This alternative would be relatively easy to implement - setting env variable for kubernetes provider if these options are passed to it.

@dpkirchner I'm not sure if I like the idea of defining proxy as a resource where it's not really a resource but just some sort of hack. Do you have examples where something similar was done in other providers?

[dpkirchner]

@hrvolapeter I did see one in another provider but unfortunately I can't find it now. It wasn't exactly the same but IIRC it created a connection that persisted throughout execution. I agree it's not a resource, and definitely a hack, but ultimately it's necessary that whatever it is can be depended on by other resources or modules so we don't try to create plans or apply changes without the tunnel.

By the way, in my specific use case, most of my resources don't speak HTTP/S and use community providers, so for it to work for me personally it'd need to support plain-ol TCP/TLS.

This might be best implemented as a third type of runtime-only "resource" that can be depended on, but I dunno.

[hrvolapeter]

I've implemented this example combining external provider with python script to set-up ssh bastion and combining it with bastion_host setting for which I create PR to kubernetes and helm provider. The same pattern can be used also for TCP proxies

https://github.com/jenkins-x/terraform-google-jx/commit/647decad45fb192d70295bdcc5bd3289938f0aae#diff-dc46acf24afd63ef8c556b77c126ccc6e578bc87e3aa09a931f33d9bf2532fbbR66