Terraform-aws-eks: Error: Post "http://localhost/api/v1/namespaces/kube-system/configmaps": dial tcp 127.0.0.1:80: connect: connection refused

[vrathore18]

I am started getting this issue:

Error: Post "http://localhost/api/v1/namespaces/kube-system/configmaps": dial tcp 127.0.0.1:80: connect: connection refused

  on .terraform/modules/eks/terraform-aws-eks-11.1.0/aws_auth.tf line 62, in resource "kubernetes_config_map" "aws_auth":
  62: resource "kubernetes_config_map" "aws_auth" {

All my code were working fine but as I upgraded my terraform versions, providers version. I started getting above issue.

version on which everything was working:
provider:-
aws: 2.49
kubernetes: 1.10.0
helm: 0.10.4
eks: 4.0.2

others:-
terraform:0.11.13
kubectl: 1.11.7
aws-iam-authenticator:0.4.0-alpha.1

Now my versions
terraform:0.12.26
kubectl: 1.16.8
aws-iam-authenticator:0.5.0

eks.yaml

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "12.1.0"

  cluster_name    = var.name
  subnets         = module.vpc.private_subnets
  vpc_id          = module.vpc.vpc_id
  cluster_version = var.cluster_version
  manage_aws_auth = "true"

  kubeconfig_aws_authenticator_additional_args = ["-r", "arn:aws:iam::${var.target_account_id}:role/terraform"]

  worker_groups = [
    {
      instance_type        = var.eks_instance_type
      asg_desired_capacity = var.eks_asg_desired_capacity
      asg_max_size         = var.eks_asg_max_size
      key_name             = var.key_name
    }
  ]

  map_accounts = [var.target_account_id]

  map_roles = [
    {
      rolearn = format("arn:aws:iam::%s:role/admin", var.target_account_id)
      username = format("%s-admin", var.name)
      groups    = ["system:masters"]
    }
  ]

  # don't write local configs, as we do it anyway
  write_kubeconfig      = "false"
}

resource "local_file" "kubeconfig" {
  content  = module.eks.kubeconfig
  filename = "./.kube_config.yaml"
}

In the above code write_kubeconfig = "false" and creating a local file kubeconfig. I am using this file in helm and kubernetes provider.

provider.yaml

`provider "aws" {
region = var.region
version = "~> 2.65.0"

assume_role {
role_arn = "arn:aws:iam::${var.target_account_id}:role/terraform"
}
}

provider "kubernetes" {
config_path = "./.kube_config.yaml"
version = "~> 1.11.3"
}

provider "helm" {
version = "~> 1.2.2"

kubernetes {
config_path = "./.kube_config.yaml"
}
}`

On terraform apply, script is not able to create module.eks.kubernetes_config_map.aws_auth[0]:

I tried some of the suggestion mentioned here but didn't worked for me
https://github.com/terraform-aws-modules/terraform-aws-eks/issues/817

[dpiddockcmp]

If you have manage_aws_auth = true then you need to configure the kubernetes provider as per the documentation in the README.

[arielvinas]

I think this is a problem with the k8s provider it self... I have it configured correctly and randomly fails to connect:

https://github.com/hashicorp/terraform/issues/4149
https://github.com/hashicorp/terraform/issues/4149

I found a comment in the provider Golang code that explains the problem:
https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/kubernetes/provider.go#L244

[jurgenweber]

I get this as well, but when I tried to disable the cluster after creation (so destroy it) the plan fails.

Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 127.0.0.1:80: connect: connection refused

[xsqian]

I encountered the similar issue:
Error: Post "https://0ED4D7D93F983B4B6F3664DA6B0262D0.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps": dial tcp: lookup 0ED4D7D93F983B4B6F3664DA6B0262D0.gr7.us-east-2.eks.amazonaws.com on 192.168.86.1:53: no such host
Any help would be appreciated!

[deanshelton913]

@dpiddockcmp It would help if you were just a tad bit more specific.

What precisely in the README's example configuration solves this problem? Is it the version number which is called out explicitly in the readme the important part? Meaning we cant take the latest version for some reason? The concat functions in use? I tried a copy paste of that readme and i get:

No provider "kubernetes" plugins meet the constraint "1.10,>= 1.11.1".

[deanshelton913]

As a workaround... I was able to use the AWS CLI to write my config, after my first deployment was only partially successful...

aws eks update-kubeconfig --name myApp --region $AWS_DEFAULT_REGION --alias myApp

Putting this ^ before the apply step (obviously only works after a successful creation of the cluster when the update to the aws-auth CM simply failed) worked for me... But if we ever burn the infra to the ground, we need to do this multi step process again.

[kunickiaj]

I get this as well, but when I tried to disable the cluster after creation (so destroy it) the plan fails.

Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp 127.0.0.1:80: connect: connection refused

Ran into this as well... given the relative instability around the lifecycle of EKS using this module I'm probably going to consider separating it from other infra in the vpc.

[dpiddockcmp]

You need to copy the two data sources and the kubernetes provider block from the usage example. Assuming your module definition was called eks:

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  load_config_file       = false
  version                = "~> 1.12"
}

[kunickiaj]

@dpiddockcmp yep I get that, the problem is encountered if you set the create_eks flag to false to destroy the cluster and then set it back to true.

I think I hit some other funkiness where the state file even showed the correct host and CA cert but the provider was using the local host and missing CA entirely.

Will see if I can get a more specific set of repro steps.

[cidesaasoptics]

I had the same problem. you have to delete the cluster manually because the terraform just says that it already exists instead of deleting then recreating. Once you delete the cluster you get this error.

I resolved the problem by running terraform state rm module.saasoptics.module.eks.kubernetes_config_map.aws_auth[0]

[kunickiaj]

Yeah, looked into this a bit. It's because terraform tries to refresh the config map resource before deleting it -- however the cluster's already been destroyed.

This module essentially needs to ensure that destruction of the config map happens before cluster destruction if that's possible. Otherwise the manual removal of the configmap from the state seems like the best solution here.

An alternative workaround to cleanly remove the cluster (you must not have gotten yourself into the state where you have the localhost error for this to work):

1. Use target mode to destroy only the EKS cluster: terraform destroy -target module.eks
2. Subsequently, set the create_eks flag to false _after_ the first step
3. Run an apply to clean up the old cluster configuration. terraform apply

[eightlimbed]

this fixed it for me

terraform state rm module.eks.kubernetes_config_map.aws_auth

thanks, @cidesaasoptics

[schollii]

I got the same error after the following sequence:

• created cluster but the instances would not join the node group, I deleted the node group
• upon recreating the cluster, got the auth map error as terraform is trying to create the map. So terraform state rm was not an option for me since the map was no longer in the state but was still \ already in the cluster (still unclear how this happened).

I was able to complete the creation by setting manage_aws_auth=false, and later deleting the map with kubectl. Then I was able to set the flag back to true.

[fliphess]

Same issue here, in my pipeline the kubeconfig is not present during apply as it's a new ci run in a fresh ci container.
This results in the provisioner connecting to localhost.

EDIT: The workaround mentioned worked for me too: First apply the pipeline with manage_aws_auth=false. After that you can safely remove the cluster without errors and start over.

[andrewalexander]

if you are deleting and know the config map is gone, creating a listener and giving the terraform client a 204 response also seemed to work to make a stuck terraform destroy proceed happily.

I don't see any real difference in the net effect between this and the terraform state rm module.eks.kubernetes_config_map.aws_auth that also worked (as did running the terraform apply manage_aws_auth=false when creating the resources in the first place)

nc -l 80
DELETE /api/v1/namespaces/kube-system/configmaps/aws-auth HTTP/1.1
Host: localhost
User-Agent: HashiCorp/1.0 Terraform/0.14.2
Content-Length: 43
Accept: application/json, */*
Content-Type: application/json
Accept-Encoding: gzip

{"kind":"DeleteOptions","apiVersion":"v1"}
HTTP/1.1 204 OK

module.eks-cluster.module.eks.kubernetes_config_map.aws_auth[0]: Destruction complete after 2m30s

Destroy complete! Resources: 1 destroyed.

[luizmiguelsl]

I had the same problem. you have to delete the cluster manually because the terraform just says that it already exists instead of deleting then recreating. Once you delete the cluster you get this error.

I resolved the problem by running terraform state rm module.saasoptics.module.eks.kubernetes_config_map.aws_auth[0]

That's it. In my case first I ran terraform state list to see what was stored in my state and got module.your-module-name.kubernetes_config_map.aws_auth[0]. After this i just ran terraform state rm module.your-module-name.kubernetes_config_map.aws_auth[0] and now I'm able to run plans and applies again. Thanks