Terraform-aws-eks: Error: Post https://xxxx.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps: dial tcp xxx:443: i/o timeout
On every example on first run:
Error: Post https://C39B1571479D46ED13A85BC0105510E2.gr7.us-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps: dial tcp 44.229.104.205:443: i/o timeout
on ../../aws_auth.tf line 45, in resource "kubernetes_config_map" "aws_auth":
45: resource "kubernetes_config_map" "aws_auth" {
This is because theres a small between when the cluster endpoint is returned after creation but is not available.
If you rerun terraform apply the configmap is created fine.
Quite annoying 馃槄
max-rocket-internet
All 22 comments
Is this an EKS bug that's worth reporting to AWS container team, or an interplay between the aws and kubernetes providers?
dpiddockcmp
on 10 Dec 2019
I keep on having the same issue. IMO, the kubernetes provider should wait / retry few times before coming back with an error.
xposix
on 10 Dec 2019
max-rocket-internet
on 11 Dec 2019
Is this an EKS bug that's worth reporting to AWS container team, or an interplay between the aws and kubernetes providers?
I think a socket error should be retried by the provider.
In the mean time, could we make a work around somehow? Like a null resource that sleeps for 10 seconds after cluster creation?
max-rocket-internet
on 11 Dec 2019
I'm using kubectl hack for this but it requires kubectl and a lot of depends_on and in general it ugly
resource "null_resource" "wait-eks-api" {
provisioner "local-exec" {
command = "until kubectl --kubeconfig ${path.root}/kubeconfig_${var.cluster_id} -n kube-system get pods >/dev/null 2>&1;do echo 'Waiting for EKS API';sleep 5;done"
}
}
in the meantime I'm going to fork eks module with kubernetes provider commits reverted because it's not usable for our usecases.
ivanich
on 13 Dec 2019
in the meantime I'm going to fork eks module with kubernetes provider commits reverted because it's not usable for our usecases.
You can just use release v7.0.1, no?
max-rocket-internet
on 13 Dec 2019
You can just use release v7.0.1, no?
7.0.1 has faulty kubernetes provider commit, so staying with 7.0.0 for now, and if it's will be an issue for a while fork is the way to go at least we can have new features provided by upstream.
ivanich
on 13 Dec 2019
7.0.1 has faulty kubernetes provider commit
No it doesn't 馃檪 I made it specifically the commit before: https://github.com/terraform-aws-modules/terraform-aws-eks/pull/630
https://github.com/terraform-aws-modules/terraform-aws-eks/releases/tag/v7.0.1
max-rocket-internet
on 13 Dec 2019
No it doesn't 馃檪 I made it specifically the commit before: #630
Nice, thanks for the tip, haven't checked commits since kubernetes provider merge. 馃憤
ivanich
on 13 Dec 2019
This is definitely a "feature" of the AWS EKS API. It returns creation complete before the cluster is actually ready to be used 馃槥
This requires all tools creating clusters to have a retry on their first access. e.g.:
- eksctl's
WaitForControlPlane - logic in the "quickstart" example
run_command - v7 of this module
local_exec
We need to go back to having a retry loop hack. Which kind of defeats the point of moving to the kubernetes provider 馃槥
I was comparing with GCP's kubernetes offering. Their service only returns ready once the cluster is actually usable. You can easily chain GKE creation with the kubernetes terraform provider.
Test code here
Raised an issue in the containers roadmap here . See what they say
dpiddockcmp
on 15 Dec 2019
FYI, I opened a PR in the AWS provider to wait for kubernetes endpoint https://github.com/terraform-providers/terraform-provider-aws/pull/11426. I don't know if it the right quick win before https://github.com/aws/containers-roadmap/issues/654 get solved. Feedbacks are welcome.
barryib
on 27 Dec 2019
Hit another issue with this damn provider when fiddling about with node_groups:
EKS Node Groups modify, and if necessary, create the aws-auth ConfigMap. This causes the kubernetes provider to always fail if you create the cluster and node groups from scratch in one run. The k8s provider needs resources to not exist or to be imported. There's no force create option.
Can we go back to kubectl and forget this whole exercise?
Edit: Raised an issue with the k8s provider. See what they say. https://github.com/terraform-providers/terraform-provider-kubernetes/issues/719
dpiddockcmp
on 29 Dec 2019
Can we go back to kubectl and forget this whole exercise?
I'm leaning towards this also.
max-rocket-internet
on 2 Jan 2020
What should we do? Revert the k8s provider change? Or implement something like in this PR? https://github.com/terraform-aws-modules/terraform-aws-eks/pull/639
max-rocket-internet
on 2 Jan 2020
Hit another issue with this damn provider when fiddling about with node_groups:
EKS Node Groups modify, and if necessary, create the aws-auth ConfigMap. This causes the kubernetes provider to always fail if you create the cluster and node groups from scratch in one run. The k8s provider needs resources to not exist or to be imported. There's no force create option.
Solved this problem with a null_data_source to enforce the ordering. So the main issue remaining is the endpoint not always being available after first creation. #639 looks like it might be the solution but that PR needs a little cleaning up.
dpiddockcmp
on 2 Jan 2020
Solved this problem with a null_data_source to enforce the ordering
Great!
max-rocket-internet
on 3 Jan 2020
Hit another issue with this damn provider when fiddling about with node_groups:
EKS Node Groups modify, and if necessary, create the aws-auth ConfigMap. This causes the kubernetes provider to always fail if you create the cluster and node groups from scratch in one run. The k8s provider needs resources to not exist or to be imported. There's no force create option.Solved this problem with a
null_data_sourceto enforce the ordering. So the main issue remaining is the endpoint not always being available after first creation. #639 looks like it might be the solution but that PR needs a little cleaning up.
Just run into this on master - can you give an example of how we can fix using a null_data_source?
jamescross91
on 3 Jan 2020
Just run into this on master - can you give an example of how we can fix using a
null_data_source?
Basically:
- Have a
data null_data_source ndswith inputs:
cluster_name. Doesn't matter where this comes fromkubernetes_config_map.aws_auth.id
- Use
data.null_data_source.nds.outputs["cluster_name"]as your cluster_name variable in the node groups
You'd have to fork the module to do it. It's not usable from outside. It only solves the race condition on cluster first creation with node groups.
dpiddockcmp
on 3 Jan 2020
Today I ran into both this issue and the Error: configmaps "my-config" already exists issue @dpiddockcmp described. I switched to the branch from #639 and it solved the problem 馃帀
erabug
on 6 Jan 2020
max-rocket-internet
on 9 Jan 2020
@max-rocket-internet I am not familiar with the release cadence. When is this fix going to be available for regular users?
ngortheone
on 14 Jan 2020
@ngortheone it's already available in v8.0.0
barryib
on 14 Jan 2020
Related issues
linxcat
路
4Comments
max-rocket-internet
路
4Comments
jimmiebtlr
路
3Comments
discordianfish
路
3Comments
gb-ckedzierski
路
5Comments
Most helpful comment
Hit another issue with this damn provider when fiddling about with node_groups:
EKS Node Groups modify, and if necessary, create the aws-auth ConfigMap. This causes the kubernetes provider to always fail if you create the cluster and node groups from scratch in one run. The k8s provider needs resources to not exist or to be imported. There's no force create option.
Can we go back to kubectl and forget this whole exercise?
Edit: Raised an issue with the k8s provider. See what they say. https://github.com/terraform-providers/terraform-provider-kubernetes/issues/719