Terraform-aws-eks: Error: Delete "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 127.0.0.1:80: connect: connection refused

[ankur-gupta-guavus]

As soon as I try to delete EKS cluster, it fails at k8s config map (aws-auth) deletion:

Error: Delete "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 127.0.0.1:80: connect: connection refused

Provider Config:

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  load_config_file       = false
  version                = "~> 1.11"
  alias                  = "giq"

  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    args        = ["token", "--cluster-id", data.aws_eks_cluster.cluster.endpoint]
    command     = "aws-iam-authenticator"
  }

}

data "aws_availability_zones" "available" {
}

module "eks" {

  providers = {
    aws = aws.giq
    kubernetes = kubernetes.giq
  }

  source          = "./eks"

Although, I have given load_config as false in kubernetes provider, I thought this can be related to kubeconfig (one created by using write_kubeconfig as true) being deleted before this config map, so i added this:

resource "kubernetes_config_map" "aws_auth" {
  count      = var.create_eks && var.manage_aws_auth ? 1 : 0
  depends_on = [
    null_resource.wait_for_cluster[0],
    local_file.kubeconfig
  ]

But, even now i get the same error, i can confirm my kubeconfig file exists, but post deletion of node_groups i get this error. Any help will be highly appreciated. Many thanks in advance.

[dpiddockcmp]

This is an error that sometimes comes up but is hard to reproduce.

The kubernetes provider, as configured, does not know anything about the kubeconfig file generated by the module. There is no relationship between them. You don't even need to write the kubeconfig file. The provider is supposed to get all of its configuration from the data sources that you are passing in.

I'm guessing you are doing a straight terraform destroy?

The easiest solution is to drop the kubernetes_config_map resource from the terraform state and then continue with the destroy.

terraform state rm module.eks.kubernetes_config_map.aws_auth
terraform destroy

[Vermyndax]

This happens every time I try to delete the EKS cluster. Doing the manual remove from the state file resolved it, but makes it painful for CI/CD automation.

[barryib]

When you got this kind of error, generally it's because your kubernetes provider is miss-configured (due to a bug or human error).

Can you please try with the latest version of the kubernetes provider and also remove exec from the provider. You don't need that, because you're already using token for authentication.

[Puneeth-n]

This is an error that sometimes comes up but is hard to reproduce.

The kubernetes provider, as configured, does not know anything about the kubeconfig file generated by the module. There is no relationship between them. You don't even need to write the kubeconfig file. The provider is supposed to get all of its configuration from the data sources that you are passing in.

I'm guessing you are doing a straight terraform destroy?

The easiest solution is to drop the kubernetes_config_map resource from the terraform state and then continue with the destroy.

terraform state rm module.eks.kubernetes_config_map.aws_auth
terraform destroy

Thanks this works

[barryib]

Closing this since, you resolved your issue. Feel free to reopen it if needed.

[jspawar]

@barryib We are running into pretty much the same exact issue: the kubeconfig resource is being deleted before the aws_auth ConfigMap resource is, so we would like to reopen this issue please. We're following the guidance of the following guide from HashiCorp: https://github.com/hashicorp/learn-terraform-provision-eks-cluster.

Can you all provide some guidance on how to actually mitigate this? That is, per this comment, what could we have misconfigured in the kubernetes provider:

When you got this kind of error, generally it's because your kubernetes provider is miss-configured (due to a bug or human error).

We are configuring our kubernetes provider exactly like this: https://github.com/hashicorp/learn-terraform-provision-eks-cluster/blob/master/kubernetes.tf

The suggestion to run terraform state rm is strongly not preferable for the same reasons originally provided (automation), so would love to know if there is some workaround, and if not, if we should provide some changes to accommodate?

[vvchik]

This bug hit me in the latest terraform, 0.14 version.