Terraform-provider-kubernetes: terraform refresh attempts to dial localhost (reopening with workaround)

[konryd]

This is a re-opening of #546

Occasionally, the kubernetes provider will start dialing localhost instead of the configured kubeconfig context.

Error: Get http://localhost/api/v1/namespaces/prometheus: dial tcp 127.0.0.1:80: connect: connection refused
Error: Get http://localhost/api/v1/namespaces/debug: dial tcp 127.0.0.1:80: connect: connection refused

In the instance of this problem that I ran into, the reason was: multiple terraform threads opening and writing the kubeconfig file without synchronization, which resulted in a messed-up kubeconfig file. This might have been related to the fact that my terraform config included multiple clusters (using this approach)

Workaround

I was able to make this go away by setting: -parallelism=1

[thirdeyenick]

Thanks for the provided workaround. We are also hitting this bug from time to time. I tried the parallelism approach and did not see the 'localhost issue' again. However, we went into a different issue with this.

I would love to know the reason why this bug happens at all (and why it can be mitigated by reducing the terraform threads). We are creating a kubeconfig file before we run terraform apply and are passing the path to it as a variable in our terraform modules. The kubernetes provider then just uses this path via var.kubeconfig. Still, from time to time it happens that the k8s provider wants to connect to localhost, although our file exists and the content is valid.

Here is our providers.tf:

provider "kubernetes" {
  config_path = var.kubeconfig
}

provider "helm" {
  kubernetes {
    config_path = var.kubeconfig
  }
  version = ">= 1.2.1"
}

[igoooor]

it happens to us too :(

[igoooor]

to be specific, in my case it happens during Refreshing state and my provider looks like this

provider "kubernetes" {
  load_config_file = false

  host                   = "https://${data.google_container_cluster.this.0.endpoint}"
  client_certificate     = data.google_container_cluster.this.0.master_auth.0.client_certificate
  client_key             = data.google_container_cluster.this.0.master_auth.0.client_key
  cluster_ca_certificate = data.google_container_cluster.this.0.master_auth.0.cluster_ca_certificate
}

If I run the same command (apply or destroy) with -refresh=false then it works fine

-parallelism=1 is not helping for me, the error is happening constantly.

[thirdeyenick]

Interesting @igoooor , does it also try to connect to localhost in your case? We have a similiar issue, like the one you describe, but in those cases we just get a 'permission denied' message (no indication that it tries to connect to localhost). If we use -refresh=false then everyhing works. I have the feeling that terraform uses an old client certificate which is not valid anymore (maybe cached in the state?).

[igoooor]

in my case I get the localhost error yes, when refreshing only

Error: Get http://localhost/api/v1/namespaces/xxx: dial tcp 127.0.0.1:80: connect: connection refused

If I replace my provider config and use variables (for host, client_certificate, etc...) instead of data.google_container_cluster... then it also works at refresh time.
It seems like when refreshing the state, it does not load values from data.google_container_cluster....

[thirdeyenick]

Yeah, this might be the case. In most of our cases we are not using terraform data sources to fill in the access credentials, but we are still experiencing this bug. I am currently checking if I get the same issue when not using a created kubeconfig file, but passing the client_certificate, client_key, etc instead directly via variables to the provider.

[igoooor]

This only happens to me since I update to terraform 13 today.
I stayed on terraform 12 until now because of some other stuff, and I never had this problem. Only now with the latest version

[alexsomesan]

I'm unable to reproduce this scenario. To me import seems to work as expected.

@igoooor Is the cluster referred to by data.google_container_cluster.this in you case already present or are you also creating the cluster in that same apply operation?

Also, everyone else, please post the versions of Terraform and provider you used.

[igoooor]

It is already present, before starting the terraform command.

[alexsomesan]

Alright, thanks for clarifying that.
Have you tried to A-B test with providing the credentials from that same cluster via a kubeconfig file?

[igoooor]

it works via kubeconfig and via parameters set for host, client_certificate, etc..
but it does not work when host, client_certificate, etc.. are set from the data.google_container_cluster.this

And again, it only fails during refresh, if I apply -refresh=false then it works.

[igoooor]

it also happens when I'm using a resource instead of a data.
Of course not the first time I apply when it creates the cluster, but afterwards if I apply again, it will try to refresh, and there it will fail as well with the same error

[sereinity]

For information, both workarounds doesn't work when using the remote backend:

Error: Custom parallelism values are currently not supported

The "remote" backend does not support setting a custom parallelism value at
this time.

Error: Planning without refresh is currently not supported

Currently the "remote" backend will always do an in-memory refresh of the
Terraform state prior to generating the plan.

[sodre]

I am experiencing the same issue @igoooor. The only difference is that I am using digitalocean instead of google.

[holleyism]

I ran into this issue as well. What appears to have happened in my case is that I had originally created a kubernetes_secret resource in a modules main.tf. Things changed and that was removed since it was no longer needed. When the refresh happened, I guess because the original resource didn't exist, it ignores any configuration for the kubernetes provider and always tries to use localhost. Without looking at the code, I'd say if there's a secret (or maybe other k8s resource) in the state file (we use remote state), but the definition for that resource is removed, then this will happen (but that's just a guess)

Our fix is to simply remove that resource from the state manually, and then manually clean up the resource.

[pduchnovsky]

Same problem here, any workarounds available ?
Any of the above is irrelevant for terraform cloud unfortunately.

[jacobwgillespie]

@pduchnovsky on Terraform Cloud, you should still be able to use the above workarounds.

1. To set -parallelism=1, you would add an environment variable named TFE_PARALLELISM and set it to 1. (see https://www.terraform.io/docs/cloud/workspaces/variables.html#special-environment-variables)
2. All of the terraform state subcommands still work with Terraform Cloud, so if you wanted to manually delete a resource from the state for manual cleanup, you can run terraform state rm path.to.kubernetes_resource.name locally and it will update Cloud. Similarly terraform state pull and terraform state push also work, if you needed to pull the entire state file down from Terraform Cloud.

[pduchnovsky]

@pduchnovsky on Terraform Cloud, you should still be able to use the above workarounds.

1. To set -parallelism=1, you would add an environment variable named TFE_PARALLELISM and set it to 1. (see https://www.terraform.io/docs/cloud/workspaces/variables.html#special-environment-variables)
2. All of the terraform state subcommands still work with Terraform Cloud, so if you wanted to manually delete a resource from the state for manual cleanup, you can run terraform state rm path.to.kubernetes_resource.name locally and it will update Cloud. Similarly terraform state pull and terraform state push also work, if you needed to pull the entire state file down from Terraform Cloud.

To be honest, this workaround is not really acceptable, e.g. I am creating single GKE cluster with two non-default node pools of which one is GPU enabled.. then I deploy around 10 kubernetes_deployment(s) of which one is created in average of 8 minutes (big images) and it would take AGES to deploy/update those if I set parallelism to 1.
I 'could' use the older version of this provider but it doesn't work with taint "nvidia.com/gpu" since it has a dot and a slash in the name..

So for the time being I made a workaround that after cluster is created I extract it's IP and cert to variables and then use those as a reference.. of course now I cannot change the cluster itself, but that's not something we do often.

Looking forward for when the PR #1078 is merged.

[RichiCoder1]

Also ran into this. Was able to work around it KUBECONFIG, but that has the exact issue stated above where I can no longer re-create my cluster as it depends on hard-coded/pre-existing variables instead of runtime variables.

[ilya-git]

I am not sure if this issue is related, but I have figured out that if a cluster needs recreation, then the output of host/ca certificate/etc. would be empty. This would result in the empty values being passed to the provider which results in connection to localhost error that hides the original fact that cluster needs recreation.

So you see the "localhost" error in the output, while the original problem that is hidden is that cluster will be re-created (meaning there is no host to connect to obviously before it is created).

See this issue for more details.

[alex-shafer-ceshop]

I am also experiencing the same issue as @ilya-git where the refresh/plan will fail if a cluster that is referenced in a dynamic provider configuration needs to be recreated.

resource "aws_eks_cluster" "main" {
  # contains some update that requires a destroy/create rather than a modify
}

provider "kubernetes" {
  host = aws_eks_cluster.main.endpoint
  ...
}

This reliably produces an error message similar to the one in the initial comment on this issue, on the first attempt to refresh a resource using the kubernetes provider.

Targeting the cluster in a "first pass" and then proceeding with the rest appears to be a viable workaround; i.e. https://github.com/hashicorp/terraform/issues/4149 would appear to be a viable fix.