Terraform-aws-eks: When re-creating a cluster from scratch, aws_auth.tf somehow tries the previous-run's EKS API endpoint

[christopher-demarco]

I have issues

I'm submitting a...

• [X] bug report

What is the current behavior?

Create a cluster, it fails:

Error: Post https://ENDPOINT-REDACTED-1.yl4.us-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps: dial tcp: lookup ENDPOINT-REDACTE-1.yl4.us-west-2.eks.amazonaws.com on 8.8.8.8:53: no such host

  on .terraform/modules/cluster/terraform-aws-eks-11.0.0/aws_auth.tf line 61, in resource "kubernetes_config_map" "aws_auth":
  61: resource "kubernetes_config_map" "aws_auth" {

That endpoint is the endpoint of the previous (n-1th) run.

After destroying all resources, tfstate is clean as expected. No residual state in the .terraform/module directory—where is that stale reference coming from?!

Re-running will succeed, so it's clearly an issue with resource-ordering or cache?

Issue is in both v8.2.0 and v11.0.0 .

Environment details

• Affected module version: 8.2.0, 11.0.0
• OS: macOS
• Terraform version: 0.12.23

Any other relevant info

[christopher-demarco]

We fixed this, but it might bear updating documentation: Adding a kubernetes provider using the module-built EKS endpoint fixes this behavior. Apparently the various kubernetes_* resources here were using the local $KUBECONFIG, which wasn't yet updated.

This situation is potentially bad: if $KUBECONFIG points to an accessible cluster, then the aws-auth configmap could be unexpectedly updated.

[barryib]

I don't get your point. Do you mean that the example and/or readme are not working ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[naveenb30]

I'm also experiencing same issue, any solution for this ?

[mitchelladam]

I'm having this issue as well, explicitly when i'm trying to create 2 eks clusters in the same vpc.
this happens in from-scratch as soon as it reaches the aws_auth kubernetes_config_map

because i'm creating two eks clusters, i have two aliased kubernetes providers. using them works fine and the config for them is created correctly,.
However the kubernetes config map in both modules doesn't use either of the configurations that i've just created and instead uses the default local kubernetes and whatever context i've currently got it set to. there doesn't seem to be any way of getting it to use the correct configuration either.

i've attached an example where this occurs.

main.tf.txt

[dpiddockcmp]

@mitchelladam You also need to pass the aliased kubernetes provider to the module. Otherwise Terraform will use a zero-config provider which defaults to whatever $KUBECONFIG or ~/.kube/config points to.

module "eks" {
  # ....
  providers = {
    kubernetes = kubernetes.clusterA
  }
}

More details in the docs here

[shankar96]

@dpiddockcmp what is clusterA ? can u show some examples. i am facing this frequent issue while recreation of cluster after destroying.

[mitchelladam]

@shankar96 it's from my example main.tf in my comment
@dpiddockcmp that did work, thanks for the quick response

[shankar96]

@mitchelladam can u share working main.tf
I am getting follwing errors

quotes surrounding this reference to silence this warning.

Error: Unsupported block type

  on eks.tf line 116, in module "eks":
 116:   providers {

Blocks of type "providers" are not expected here.

[mitchelladam]

@shankar96
just missing the = sign
providers = {

[barryib]

Closing, since this sounds like you fixed you issue. Feel free to re-open it if you don't.