Terraform-aws-eks: Again - Provider produced inconsistent final plan
I have issues
With a new module version 6.0 we still have an issue with:
Error: Provider produced inconsistent final plan
This issue previously has been address in: https://github.com/terraform-providers/terraform-provider-aws/issues/9882 and https://github.com/terraform-aws-modules/terraform-aws-eks/issues/481 but in version 6.0 of Module it back again
What is the current behavior?
Error: Provider produced inconsistent final plan
When expanding the plan for
module.eks.aws_autoscaling_group.workers_launch_template[0] to include new
values learned so far during apply, provider "aws" produced an invalid new
value for .initial_lifecycle_hook: planned set element
cty.ObjectVal(map[string]cty.Value{"default_result":cty.UnknownVal(cty.String),
"heartbeat_timeout":cty.UnknownVal(cty.Number),
"lifecycle_transition":cty.UnknownVal(cty.String),
"name":cty.UnknownVal(cty.String),
"notification_metadata":cty.UnknownVal(cty.String),
"notification_target_arn":cty.UnknownVal(cty.String),
"role_arn":cty.UnknownVal(cty.String)}) does not correlate with any element in
actual.
If this is a bug, how to reproduce? Please include a code sample if relevant.
worker_groups_launch_template = [
{
instance_type = "c5.large"
key_name = aws_key_pair.worker_management.key_name
subnets = var.subnet_ids
asg_desired_capacity = var.minimal_cluster_size
asg_min_size = var.minimal_cluster_size
asg_max_size = "5"
pre_userdata = <<EOFT
}
]
module "eks" {
source = "terraform-aws-modules/eks/aws"
cluster_name = local.cluster_name
cluster_version = var.cluster_version
subnets = concat(var.subnet_ids, var.underlay_subnet_ids)
tags = local.tags
vpc_id = var.vpc_id
worker_groups_launch_template = local.worker_groups_launch_template
worker_additional_security_group_ids = [aws_security_group.all_worker_mgmt.id]
map_roles = var.map_roles
map_users = var.map_users
map_accounts = var.map_accounts
config_output_path = "./generated/"
cluster_endpoint_private_access = var.private
permissions_boundary = var.private ? <ARN> : ""
}
What's the expected behavior?
With empty list (var.input.asg_initial_lifecycle_hooks), the dynamic block initial_lifecycle_hook must not be created and terraform should finish without panic.
Are you able to fix this problem and submit a PR? Link here if you have already.
Environment details
- Affected module version:
6.0.0 - Terraform version:
Terraform v0.12.6
+ provider.aws v2.29.0
+ provider.local v1.3.0
+ provider.null v2.1.2
+ provider.random v2.2.0
+ provider.template v2.1.2
timota
👍6
👀2
All 41 comments
I found issue, it was previously created by @barryib in 489 and somehow missed in version 6.x. So i created diff (cant create PR - no permissions). Can these changes be added to current code.
diff --git a/workers.tf b/workers.tf
index 45db1e4..f89511b 100644
--- a/workers.tf
+++ b/workers.tf
@@ -75,7 +75,7 @@ resource "aws_autoscaling_group" "workers" {
)
dynamic "initial_lifecycle_hook" {
- for_each = lookup(var.worker_groups[count.index], "asg_initial_lifecycle_hooks", local.workers_group_defaults["asg_initial_lifecycle_hooks"])
+ for_each = length(lookup(var.worker_groups_launch_template[count.index], "asg_initial_lifecycle_hooks", local.workers_group_defaults["asg_initial_lifecycle_hooks"])) > 0 ? lookup(var.worker_groups_launch_template[count.index], "asg_initial_lifecycle_hooks", local.workers_group_defaults["asg_initial_lifecycle_hooks"]) : []
content {
name = lookup(initial_lifecycle_hook.value, "name", null)
lifecycle_transition = lookup(initial_lifecycle_hook.value, "lifecycle_transition", null)
diff --git a/workers_launch_template.tf b/workers_launch_template.tf
index 258d536..320b62f 100644
--- a/workers_launch_template.tf
+++ b/workers_launch_template.tf
@@ -151,7 +151,7 @@ resource "aws_autoscaling_group" "workers_launch_template" {
}
dynamic "initial_lifecycle_hook" {
- for_each = lookup(var.worker_groups_launch_template[count.index], "asg_initial_lifecycle_hooks", local.workers_group_defaults["asg_initial_lifecycle_hooks"])
+ for_each = length(lookup(var.worker_groups_launch_template[count.index], "asg_initial_lifecycle_hooks", local.workers_group_defaults["asg_initial_lifecycle_hooks"])) > 0 ? lookup(var.worker_groups_launch_template[count.index], "asg_initial_lifecycle_hooks", local.workers_group_defaults["asg_initial_lifecycle_hooks"]) : []
content {
name = lookup(initial_lifecycle_hook.value, "name", null)
lifecycle_transition = lookup(initial_lifecycle_hook.value, "lifecycle_transition", null)
Thanks in advance
timota
on 25 Sep 2019
@timota weird. i'm gonna have a look.
barryib
on 25 Sep 2019
maybe this comes from
worker_groups_launch_template = [
{
key_name = aws_key_pair.worker_management.key_name
# ....
}
]
And during your plan, aws_key_pair.worker_management.key_name doesn't exist during your first plan.
barryib
on 25 Sep 2019
I have the same issue. key_name is supposed to be optional. eks hosts dont need a key to be brought up.
manpreetsnehra
on 25 Sep 2019
How it related to this block dynamic "initial_lifecycle_hook" ?
timota
on 25 Sep 2019
because of https://github.com/terraform-aws-modules/terraform-aws-eks/issues/481#issuecomment-524817067
The issue: if a variable or local contains an unknown value then the whole variable/local is tagged as unknown in the diff state. Fine for simple variables but really annoying when dealing with collections. This makes terraform do weird things in dynamic blocks. It assumes it can't use anything from the collection until apply time. When it turns out that the referred to list is in fact empty and.. oops.
barryib
on 25 Sep 2019
but this doesnt explain why after fix #483 - everything starts working.
Anyway - let me try apply with vanilla module without key.
timota
on 25 Sep 2019
The key_name is mandatory. For tests, can you please create your key before and apply your eks module with the key created previously.
I'm gonna have a look also on a new cluster with your configuration.
barryib
on 25 Sep 2019
Error: Provider produced inconsistent final plan
When expanding the plan for module.eks.aws_autoscaling_group.workers[0] to
include new values learned so far during apply, provider "aws" produced an
invalid new value for .initial_lifecycle_hook: planned set element
cty.ObjectVal(map[string]cty.Value{"default_result":cty.UnknownVal(cty.String),
"heartbeat_timeout":cty.UnknownVal(cty.Number),
"lifecycle_transition":cty.UnknownVal(cty.String),
"name":cty.UnknownVal(cty.String),
"notification_metadata":cty.UnknownVal(cty.String),
"notification_target_arn":cty.UnknownVal(cty.String),
"role_arn":cty.UnknownVal(cty.String)}) does not correlate with any element in
actual.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.
with my config
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "6.0.0"
cluster_name = var.eks_cluster.name
subnets = concat(module.vpc.public_subnets,module.vpc.private_subnets)
vpc_id = module.vpc.vpc_id
cluster_create_security_group = true
cluster_endpoint_private_access = true
cluster_log_retention_in_days = 7
write_kubeconfig = true
worker_groups = [
{
instance_type = var.eks_cluster.instance_type
asg_max_size = var.eks_cluster.autoscaling_group_max_size
tags = [ for key,value in var.tags: {
key = "${key}"
value = "${value}"
propagate_at_launch = true
}]
}
]
tags = var.tags
}
Also according to module documentation only mandatory options are
cluster_name
subnets
vpc_id
manpreetsnehra
on 25 Sep 2019
Can you provide the plan output before the error. Do you have "(known after apply)" in your ASG, LT or LC resource ?
Also according to module documentation only mandatory options are
cluster_name
subnets
vpc_id
Maybe the default should be null and not ""
barryib
on 25 Sep 2019
# module.eks.aws_autoscaling_group.workers[0] will be created
+ resource "aws_autoscaling_group" "workers" {
+ arn = (known after apply)
+ availability_zones = (known after apply)
+ default_cooldown = (known after apply)
+ desired_capacity = (known after apply)
+ enabled_metrics = (known after apply)
+ force_delete = (known after apply)
+ health_check_grace_period = 300
+ health_check_type = (known after apply)
+ id = (known after apply)
+ launch_configuration = (known after apply)
+ load_balancers = (known after apply)
+ max_size = 10
+ metrics_granularity = "1Minute"
+ min_size = (known after apply)
+ name = (known after apply)
+ name_prefix = (known after apply)
+ placement_group = (known after apply)
+ protect_from_scale_in = (known after apply)
+ service_linked_role_arn = (known after apply)
+ suspended_processes = (known after apply)
+ tags = [
+ {
+ "key" = "Name"
+ "propagate_at_launch" = "true"
+ "value" = "project-k8s-cluster-0-eks_asg"
},
+ {
+ "key" = "kubernetes.io/cluster/project-k8s-cluster"
+ "propagate_at_launch" = "true"
+ "value" = "owned"
},
+ {
+ "key" = "k8s.io/cluster/project-k8s-cluster"
+ "propagate_at_launch" = "true"
+ "value" = "owned"
},
+ (known after apply),
+ {
+ "key" = "k8s.io/cluster-autoscaler/project-k8s-cluster"
+ "propagate_at_launch" = "false"
+ "value" = "project-k8s-cluster"
},
+ (known after apply),
+ {
+ "key" = "environment"
+ "propagate_at_launch" = "true"
+ "value" = "qa"
},
+ {
+ "key" = "environment"
+ "propagate_at_launch" = "true"
+ "value" = "qa"
},
]
+ target_group_arns = (known after apply)
+ termination_policies = (known after apply)
+ vpc_zone_identifier = (known after apply)
+ wait_for_capacity_timeout = "10m"
+ initial_lifecycle_hook {
+ default_result = (known after apply)
+ heartbeat_timeout = (known after apply)
+ lifecycle_transition = (known after apply)
+ name = (known after apply)
+ notification_metadata = (known after apply)
+ notification_target_arn = (known after apply)
+ role_arn = (known after apply)
}
}
Just a note with 6.0.0 I'm hitting this every time I apply. 2nd time it gets past it.
davisford
on 26 Sep 2019
Same here, terraform-aws-eks version v6.0.1. Have to rollback to v5.1.0.
When expanding the plan for
module.eks.aws_autoscaling_group.workers_launch_template[1] to include new
values learned so far during apply, provider "aws" produced an invalid new
value for .initial_lifecycle_hook: planned set element
cty.ObjectVal(map[string]cty.Value{"default_result":cty.UnknownVal(cty.String),
"heartbeat_timeout":cty.UnknownVal(cty.Number),
"lifecycle_transition":cty.UnknownVal(cty.String),
"name":cty.UnknownVal(cty.String),
"notification_metadata":cty.UnknownVal(cty.String),
"notification_target_arn":cty.UnknownVal(cty.String),
"role_arn":cty.UnknownVal(cty.String)}) does not correlate with any element in
actual.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.
marcecaro
on 26 Sep 2019
Ok, i have tested with "vanilla" module version 6.0.1 and with key_name explicitly defined:
TF plan
# module.eks.aws_launch_template.workers_launch_template[0] will be created
+ resource "aws_launch_template" "workers_launch_template" {
+ arn = (known after apply)
+ default_version = (known after apply)
+ ebs_optimized = (known after apply)
+ id = (known after apply)
+ image_id = (known after apply)
+ instance_type = (known after apply)
+ key_name = "ansible"
+ latest_version = (known after apply)
+ name = (known after apply)
+ name_prefix = (known after apply)
+ tags = {
+ "Environment" = "test"
+ "Workspace" = "default"
}
+ user_data = (known after apply)
+ block_device_mappings {
+ device_name = (known after apply)
+ ebs {
+ delete_on_termination = "true"
+ encrypted = (known after apply)
+ iops = (known after apply)
+ kms_key_id = (known after apply)
+ volume_size = (known after apply)
+ volume_type = (known after apply)
}
}
+ credit_specification {
+ cpu_credits = (known after apply)
}
+ iam_instance_profile {
+ name = (known after apply)
}
+ instance_market_options {
+ market_type = (known after apply)
}
+ monitoring {
+ enabled = (known after apply)
}
+ network_interfaces {
+ associate_public_ip_address = (known after apply)
+ delete_on_termination = (known after apply)
+ security_groups = (known after apply)
}
+ placement {
+ group_name = (known after apply)
+ tenancy = (known after apply)
}
+ tag_specifications {
+ resource_type = "volume"
+ tags = (known after apply)
}
}
where key_name = "ansible"
and got the same error
Error: Provider produced inconsistent final plan
When expanding the plan for
module.eks.aws_autoscaling_group.workers_launch_template[0] to include new
values learned so far during apply, provider "aws" produced an invalid new
value for .initial_lifecycle_hook: planned set element
cty.ObjectVal(map[string]cty.Value{"default_result":cty.UnknownVal(cty.String),
"heartbeat_timeout":cty.UnknownVal(cty.Number),
"lifecycle_transition":cty.UnknownVal(cty.String),
"name":cty.UnknownVal(cty.String),
"notification_metadata":cty.UnknownVal(cty.String),
"notification_target_arn":cty.UnknownVal(cty.String),
"role_arn":cty.UnknownVal(cty.String)}) does not correlate with any element in
actual.
second run - i defined key inside module, just in case
# key_name = lookup(
# var.worker_groups_launch_template[count.index],
# "key_name",
# local.workers_group_defaults["key_name"],
# )
key_name = "ansible"
Result the same.
Now, i guess this means that issue not with local key at all.
timota
on 26 Sep 2019
Thanks all for your feedbacks. The issue is not about local keys, but the unknown (known after apply) not handled correctly by the AWS provider or Terraform core for initial_lifecycle_hooks.
There are already issues on the aws provider or terraform core repository for inconstant plans :
- https://github.com/terraform-providers/terraform-provider-aws/issues/9882
- https://github.com/terraform-providers/terraform-provider-aws/issues/10088
- https://github.com/terraform-providers/terraform-provider-aws/issues/10107
- https://github.com/hashicorp/terraform/issues/22409
More info on inaccurate plans https://www.terraform.io/docs/extend/terraform-0.12-compatibility.html#inaccurate-plans.
To workaround on this, I'm going to add new variable asg_create_initial_lifecycle_hooks to use as switch for lifecycle hooks. And let users who want to use lifecycle hooks to provide consistent values across Terraform plans.
I'm going to have also a look on the AWS provider side if a customization diff will help to solve this issue.
barryib
on 26 Sep 2019
@timota @marcecaro @davisford @manpreetsnehra Is this PR work for you https://github.com/terraform-aws-modules/terraform-aws-eks/pull/532
barryib
on 26 Sep 2019
@barryib I was having same issue when I just picked one example and ran it. I can confirm this patch in #532 solved it.
poush
on 26 Sep 2019
@barryib I was also having the same issue and your PR works as intended. :+1:
mfin
on 26 Sep 2019
@poush @mfin thanks for your feedbacks.
closing since #532 is now merged. Feel free to re-open this issue if the provided solution doesn't work for you.
barryib
on 26 Sep 2019
@barryib will there be a 6.0.2 release soon?
davisford
on 27 Sep 2019
👍1
Thanks @barryib, Same question here. Any ETA for the release with the fix?
marcecaro
on 27 Sep 2019
👍1
Yes... In the next couple of days.
barryib
on 27 Sep 2019
👍4
👀1
This is causing ongoing issues for us, is there any update on when it is likely to be fixed?
colijack
on 7 Oct 2019
👀1
@colijack @marcecaro @davisford This fix has been released in v6.0.2
barryib
on 7 Oct 2019
🚀3
👍3
🎉1
The issue continues , the first time it triggers:
Provider produced inconsistent final plan
When expanding the plan for
module.eks.aws_autoscaling_group.workers_launch_template[0] to include new
values learned so far during apply, provider "aws" produced an invalid new
value for .initial_lifecycle_hook: planned set element
cty.ObjectVal(map[string]cty.Value{"default_result":cty.UnknownVal(cty.String),
"heartbeat_timeout":cty.UnknownVal(cty.Number),
"lifecycle_transition":cty.UnknownVal(cty.String),
"name":cty.UnknownVal(cty.String),
"notification_metadata":cty.UnknownVal(cty.String),
"notification_target_arn":cty.UnknownVal(cty.String),
"role_arn":cty.UnknownVal(cty.String)}) does not correlate with any element in
actual.
This error gets triggered for any single launch_template. But if y do an apply again then it works fine.
It seem like a race condition or something similar.
Some additional information:
Downloading terraform-aws-modules/eks/aws 6.0.2 for eks...
- eks in .terraform/modules/eks/terraform-aws-modules-terraform-aws-eks-1be1a02
Downloading terraform-aws-modules/vpc/aws 2.17.0 for vpc...
- vpc in .terraform/modules/vpc/terraform-aws-modules-terraform-aws-vpc-5358041
- Checking for available provider plugins...
- Downloading plugin for provider "template" (hashicorp/template) 2.1.2...
- Downloading plugin for provider "local" (hashicorp/local) 1.4.0...
- Downloading plugin for provider "null" (hashicorp/null) 2.1.2...
- Downloading plugin for provider "random" (hashicorp/random) 2.2.1...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.31.0...
Any clue if there is a workaround?
marcecaro
on 9 Oct 2019
👍1
@marcecaro Can you please share your configuration and plan output ?
barryib
on 9 Oct 2019
@barryib I can confirm the issue is still happening.
plan.txt
kkapoor1987
on 10 Oct 2019
👍1
Also can confirm that the issue still present. @barryib can we reopen this ticket?
aLekSer
on 11 Oct 2019
@barryib Did you get a chance to look into it ?
kkapoor1987
on 17 Oct 2019
Not yet. I have to figure out why #532 is not working as expected for everyone.
barryib
on 17 Oct 2019
@aLekSer @timota can you please share your configuration ? I would like to reproduce this locally.
barryib
on 20 Oct 2019
@barryib I can share part of it, I think you will be able to figureout the variables meaning (I am extracting this form other tools that uses terraform, that is the reason that i don't have access to all the files). I hope you can reproduce the issue with this configuration.
locals {
worker_groups = [
## three autoscale groups for regular instance type
{
instance_type = "${var.node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 0)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.protect_from_scale_in))}"
root_volume_type = "${var.root_volume_type}"
root_volume_size = "${var.root_volume_size}"
root_encrypted = "${tobool(lower(var.root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
{
instance_type = "${var.node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 1)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.protect_from_scale_in))}"
root_volume_type = "${var.root_volume_type}"
root_volume_size = "${var.root_volume_size}"
root_encrypted = "${tobool(lower(var.root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
{
instance_type = "${var.node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 2)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.protect_from_scale_in))}"
root_volume_type = "${var.root_volume_type}"
root_volume_size = "${var.root_volume_size}"
root_encrypted = "${tobool(lower(var.root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
## three autoscale groups for high cpu instance type
{
instance_type = "${var.high_cpu_node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 0)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.high_cpu_desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.high_cpu_max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.high_cpu_mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.high_cpu_protect_from_scale_in))}"
root_volume_type = "${var.high_cpu_root_volume_type}"
root_volume_size = "${var.high_cpu_root_volume_size}"
root_encrypted = "${tobool(lower(var.high_cpu_root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
{
instance_type = "${var.high_cpu_node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 1)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.high_cpu_desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.high_cpu_max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.high_cpu_mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.high_cpu_protect_from_scale_in))}"
root_volume_type = "${var.high_cpu_root_volume_type}"
root_volume_size = "${var.high_cpu_root_volume_size}"
root_encrypted = "${tobool(lower(var.high_cpu_root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
{
instance_type = "${var.high_cpu_node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 2)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.high_cpu_desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.high_cpu_max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.high_cpu_mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.high_cpu_protect_from_scale_in))}"
root_volume_type = "${var.high_cpu_root_volume_type}"
root_volume_size = "${var.high_cpu_root_volume_size}"
root_encrypted = "${tobool(lower(var.high_cpu_root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
## three autoscale groups for high memory instance type
{
instance_type = "${var.high_memory_node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 0)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.high_memory_desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.high_memory_max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.high_memory_mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.high_memory_protect_from_scale_in))}"
root_volume_type = "${var.high_memory_root_volume_type}"
root_volume_size = "${var.high_memory_root_volume_size}"
root_encrypted = "${tobool(lower(var.high_memory_root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
{
instance_type = "${var.high_memory_node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 1)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.high_memory_desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.high_memory_max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.high_memory_mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.high_memory_protect_from_scale_in))}"
root_volume_type = "${var.high_memory_root_volume_type}"
root_volume_size = "${var.high_memory_root_volume_size}"
root_encrypted = "${tobool(lower(var.high_memory_root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
{
instance_type = "${var.high_memory_node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 2)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.high_memory_desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.high_memory_max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.high_memory_mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.high_memory_protect_from_scale_in))}"
root_volume_type = "${var.high_memory_root_volume_type}"
root_volume_size = "${var.high_memory_root_volume_size}"
root_encrypted = "${tobool(lower(var.high_memory_root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
## three autoscale groups for high cpu and memory instance type
{
instance_type = "${var.high_cpu_memory_node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 0)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.high_cpu_memory_desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.high_cpu_memory_max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.high_cpu_memory_mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.high_cpu_memory_protect_from_scale_in))}"
root_volume_type = "${var.high_cpu_memory_root_volume_type}"
root_volume_size = "${var.high_cpu_memory_root_volume_size}"
root_encrypted = "${tobool(lower(var.high_cpu_memory_root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
{
instance_type = "${var.high_cpu_memory_node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 1)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.high_cpu_memory_desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.high_cpu_memory_max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.high_cpu_memory_mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.high_cpu_memory_protect_from_scale_in))}"
root_volume_type = "${var.high_cpu_memory_root_volume_type}"
root_volume_size = "${var.high_cpu_memory_root_volume_size}"
root_encrypted = "${tobool(lower(var.high_cpu_memory_root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
{
instance_type = "${var.high_cpu_memory_node_instance_type}"
subnets = ["${element(module.vpc.private_subnets, 2)}"]
asg_desired_capacity = "${max(ceil(tonumber(var.high_cpu_memory_desired_machines) / 3), 0)}"
asg_max_size = "${max(ceil(tonumber(var.high_cpu_memory_max_machines) / 3), 0)}"
asg_min_size = "${max(ceil(tonumber(var.high_cpu_memory_mim_machines) / 3), 0)}"
autoscaling_enabled = true
protect_from_scale_in = "${tobool(lower(var.high_cpu_memory_protect_from_scale_in))}"
root_volume_type = "${var.high_cpu_memory_root_volume_type}"
root_volume_size = "${var.high_cpu_memory_root_volume_size}"
root_encrypted = "${tobool(lower(var.high_cpu_memory_root_volume_encrypted))}"
root_kms_key_id = "${data.aws_kms_key.default.arn}"
},
]
}
module "eks" {
source = "terraform-aws-modules/eks/aws"
cluster_name = "${var.namespace}"
subnets = "${concat(module.vpc.private_subnets, module.vpc.public_subnets)}"
vpc_id = "${module.vpc.vpc_id}"
worker_sg_ingress_from_port = "0"
workers_additional_policies = ["${aws_iam_policy.eks_alb_policy.arn}"] ## Policy for ALB management
worker_groups_launch_template = [
for asg in local.worker_groups :
asg
if asg.instance_type != "none" && asg.instance_type != ""
]
worker_create_initial_lifecycle_hooks = false
tags = {
"KubernetesCluster" = "${var.namespace}"
"Start_Stop" = "NO"
"kubernetes.io/cluster/${var.namespace}" = "owned"
}
}
@barryib Can you advice on what can we change to make it work and avoid this issue ?
kkapoor1987
on 13 Nov 2019
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] picture](https://avatars3.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 11 Feb 2020
I was getting a similar error message, except that my inconsistancy was attached to an aws_lambda_function that had a source_code_hash = filebase64sha256(local.lambda_zipfilename) directive.
Part of my terraform script was a local-exec provisioner to create a lambda-deployment.
resource "null_resource" "pip_requirements" {
provisioner "local-exec" {
command = "rm -rf venv && python3 -m venv ./ && . ./bin/activate && python3 -m pip install -r requirements.txt"
working_dir = local.lambda_sourcepath
}
}
So it makes a certain amount of sense that the return value of filebase64sha256 would be one value at the beginning of the run, and a completely different value later, after my zip file got updated.
I was able to get it to work by running terraform plan -out plan.tfplan followed by terraform apply -auto-approve plan.tfplan, instead of just terraform apply without doing a plan first.
Is there a better way I could have handled it?
timbaileyjones
on 11 May 2020
@timbaileyjones this repo is for an EKS module. Please find a general terraform support forum for your issue.
dpiddockcmp
on 12 May 2020
I have an issue with inconsistent plan as well, but am unsure if this is the same reason as being discussed here. Potentially I am doing wrong, if not I am curious for workarounds.
With following configuration:
module "eks" {
source = "terraform-aws-modules/eks/aws"
cluster_name = var.cluster_name
cluster_version = "1.16"
cluster_enabled_log_types = ["audit", "authenticator"]
subnets = module.vpc.private_subnets
vpc_id = module.vpc.vpc_id
enable_irsa = true
map_users = var.eks_map_users
worker_groups = [
{
name = "worker-group-1"
instance_type = var.eks_node_ec2_type
# changes to asg sizing will have no effect, see here:
# https://github.com/terraform-aws-modules/terraform-aws-eks/blob/6c42d23c23d501c62bac6d8f0d9039f7c46a8307/docs/faq.md#why-does-changing-the-worker-groups-desired-count-not-do-anything
asg_desired_capacity = 2
asg_min_size = 2
asg_max_size = 4
additional_security_group_ids = [
aws_security_group.worker_group_mgmt_one.id
]
tags = [
{
key = "k8s.io/cluster-autoscaler/enabled"
propagate_at_launch = "false"
value = "true"
},
{
key = "k8s.io/cluster-autoscaler/${var.cluster_name}"
propagate_at_launch = "false"
value = "true"
}
]
}
]
worker_additional_security_group_ids = [
aws_security_group.all_worker_mgmt.id
]
tags = {
Cluster = var.cluster_name
Environment = var.environment
Terraform = "true"
}
}
I am getting on each run differences in tag ordering. I am using latest EKS module version 12.0, terraform-provider-aws_v2.63.0_x4.
User-Agent: aws-sdk-go/1.25.3 (go1.12.13; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.24
Click to expand for details
Terraform will perform the following actions:
# module.eks.aws_autoscaling_group.workers[0] will be updated in-place
~ resource "aws_autoscaling_group" "workers" {
arn = "arn:aws:autoscaling:eu-west-1:MYACCOUNT:autoScalingGroup:f99f4afd-e8c2-4219-a7cf-0b7de7df061a:autoScalingGroupName/cf-tf-eks-worker-group-120200508085635599200000001"
availability_zones = [
"eu-west-1a",
"eu-west-1b",
"eu-west-1c",
]
default_cooldown = 300
desired_capacity = 2
enabled_metrics = []
force_delete = false
health_check_grace_period = 300
health_check_type = "EC2"
id = "cf-tf-eks-worker-group-120200508085635599200000001"
launch_configuration = "cf-tf-eks-worker-group-120200512081006828300000001"
load_balancers = []
max_instance_lifetime = 0
max_size = 4
metrics_granularity = "1Minute"
min_size = 2
name = "cf-tf-eks-worker-group-120200508085635599200000001"
name_prefix = "cf-tf-eks-worker-group-1"
protect_from_scale_in = false
service_linked_role_arn = "arn:aws:iam::MYACCOUNT:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
suspended_processes = [
"AZRebalance",
]
~ tags = [
+ {
+ "key" = "Name"
+ "propagate_at_launch" = "true"
+ "value" = "cf-tf-eks-worker-group-1-eks_asg"
},
+ {
+ "key" = "kubernetes.io/cluster/cf-tf-eks"
+ "propagate_at_launch" = "true"
+ "value" = "owned"
},
+ {
+ "key" = "k8s.io/cluster/cf-tf-eks"
+ "propagate_at_launch" = "true"
+ "value" = "owned"
},
{
"key" = "Cluster"
"propagate_at_launch" = "true"
"value" = "cf-tf-eks"
},
{
"key" = "Environment"
"propagate_at_launch" = "true"
"value" = "test"
},
- {
- "key" = "Name"
- "propagate_at_launch" = "true"
- "value" = "cf-tf-eks-worker-group-1-eks_asg"
},
{
"key" = "Terraform"
"propagate_at_launch" = "true"
"value" = "true"
},
- {
- "key" = "k8s.io/cluster-autoscaler/cf-tf-eks"
- "propagate_at_launch" = "false"
- "value" = "true"
},
{
"key" = "k8s.io/cluster-autoscaler/enabled"
"propagate_at_launch" = "false"
"value" = "true"
},
- {
- "key" = "k8s.io/cluster/cf-tf-eks"
- "propagate_at_launch" = "true"
- "value" = "owned"
},
- {
- "key" = "kubernetes.io/cluster/cf-tf-eks"
- "propagate_at_launch" = "true"
- "value" = "owned"
},
+ {
+ "key" = "k8s.io/cluster-autoscaler/cf-tf-eks"
+ "propagate_at_launch" = "false"
+ "value" = "true"
},
]
target_group_arns = []
termination_policies = []
vpc_zone_identifier = [
"subnet-053e8b0b5c54b3316",
"subnet-090ae11ad7538c381",
"subnet-0bd52ba6f6dbb034b",
]
wait_for_capacity_timeout = "10m"
}
And in debug output:
2020/05/28 07:25:18 [WARN] Provider "registry.terraform.io/-/aws" produced an invalid plan for aws_security_group.worker_group_mgmt_one, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .ingress: planned value cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"cidr_blocks":cty.ListVal([]cty.Value{cty.StringVal("10.0.0.0/8")}), "description":cty.StringVal(""), "from_port":cty.NumberIntVal(22), "ipv6_cidr_blocks":cty.ListValEmpty(cty.String), "prefix_list_ids":cty.ListValEmpty(cty.String), "protocol":cty.Stri
ngVal("tcp"), "security_groups":cty.SetValEmpty(cty.String), "self":cty.False, "to_port":cty.NumberIntVal(22)})}) does not match config value cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"cidr_blocks":cty.ListVal([]cty.Value{cty.StringVal("10.0.0.0/8")}), "description":cty.NullVal(cty.String), "from_port":cty.NumberIntVal(22),
"ipv6_cidr_blocks":cty.NullVal(cty.List(cty.String)), "prefix_list_ids":cty.NullVal(cty.List(cty.String)), "protocol":cty.StringVal("tcp"), "security_groups":cty.NullVal(cty.Set(cty.String)), "self":cty.NullVal(cty.Bool), "to_port":cty.NumberIntVal(22)})}) nor prior value cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"cidr_block
s":cty.ListVal([]cty.Value{cty.StringVal("10.0.0.0/8")}), "description":cty.StringVal(""), "from_port":cty.NumberIntVal(22), "ipv6_cidr_blocks":cty.ListValEmpty(cty.String), "prefix_list_ids":cty.ListValEmpty(cty.String), "protocol":cty.StringVal("tcp"), "security_groups":cty.SetValEmpty(cty.String), "self":cty.False, "to_port":cty.NumberIn
tVal(22)})})
2020/05/28 07:25:18 [WARN] Provider "registry.terraform.io/-/aws" produced an invalid plan for aws_security_group.all_worker_mgmt, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .ingress: planned value cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"cidr_blocks":cty.ListVal([]cty.Value{cty.StringVal("10.0.0.0/8"), cty.StringVal("172.16.0.0/12"), cty.StringVal("192.168.0.0/16")}), "description":cty.StringVal(""), "from_port":cty.NumberIntVal(22), "ipv6_cidr_blocks":cty.ListValEmpty(cty.String), "p
refix_list_ids":cty.ListValEmpty(cty.String), "protocol":cty.StringVal("tcp"), "security_groups":cty.SetValEmpty(cty.String), "self":cty.False, "to_port":cty.NumberIntVal(22)})}) does not match config value cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"cidr_blocks":cty.ListVal([]cty.Value{cty.StringVal("10.0.0.0/8"), cty.String
Val("172.16.0.0/12"), cty.StringVal("192.168.0.0/16")}), "description":cty.NullVal(cty.String), "from_port":cty.NumberIntVal(22), "ipv6_cidr_blocks":cty.NullVal(cty.List(cty.String)), "prefix_list_ids":cty.NullVal(cty.List(cty.String)), "protocol":cty.StringVal("tcp"), "security_groups":cty.NullVal(cty.Set(cty.String)), "self":cty.NullVal(c
ty.Bool), "to_port":cty.NumberIntVal(22)})}) nor prior value cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"cidr_blocks":cty.ListVal([]cty.Value{cty.StringVal("10.0.0.0/8"), cty.StringVal("172.16.0.0/12"), cty.StringVal("192.168.0.0/16")}), "description":cty.StringVal(""), "from_port":cty.NumberIntVal(22), "ipv6_cidr_blocks":cty
.ListValEmpty(cty.String), "prefix_list_ids":cty.ListValEmpty(cty.String), "protocol":cty.StringVal("tcp"), "security_groups":cty.SetValEmpty(cty.String), "self":cty.False, "to_port":cty.NumberIntVal(22)})})
Update: After more research it seems the issue is caused by AWS Provider, see issue https://github.com/terraform-providers/terraform-provider-aws/issues/13469
cf-sewe
on 28 May 2020
I am currently having this problem:
- provider 2.70.0
- eks module 12.2.0
- terraform 0.13.0
Everything was already applied with no changes in plan. Then I needed to add this:
worker_additional_security_group_ids = [module.eks_nodes_custom_security_group.this_security_group_id]
And ran into:
Error: Provider produced inconsistent final plan
When expanding the plan for module.eks.random_pet.workers_launch_template[0]
to include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/random" changed the planned action from
CreateThenDelete to DeleteThenCreate.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.
Error: Provider produced inconsistent final plan
When expanding the plan for module.eks.random_pet.workers_launch_template[2]
to include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/random" changed the planned action from
CreateThenDelete to DeleteThenCreate.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.
Error: Provider produced inconsistent final plan
When expanding the plan for module.eks.random_pet.workers_launch_template[1]
to include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/random" changed the planned action from
CreateThenDelete to DeleteThenCreate.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.
After a couple of re-tries this is the only thing remaining in diff:
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+/- create replacement and then destroy
Terraform will perform the following actions:
# module.eks.random_pet.workers_launch_template[0] must be replaced
+/- resource "random_pet" "workers_launch_template" {
~ id = "creative-roughy" -> (known after apply)
~ keepers = { # forces replacement
~ "lt_name" = "us-prod-eks-cluster-us-prod-eks-worker-spot-subnet-03ed27ff8fe41e75d20200812074859681300000021-1" -> "us-prod-eks-cluster-us-prod-eks-worker-spot-subnet-03ed27ff8fe41e75d20200812074859681300000021-2"
}
length = 2
separator = "-"
}
# module.eks.random_pet.workers_launch_template[1] must be replaced
+/- resource "random_pet" "workers_launch_template" {
~ id = "liked-egret" -> (known after apply)
~ keepers = { # forces replacement
~ "lt_name" = "us-prod-eks-cluster-us-prod-eks-worker-spot-subnet-0dd51036e98e5befa2020081207485967580000001f-1" -> "us-prod-eks-cluster-us-prod-eks-worker-spot-subnet-0dd51036e98e5befa2020081207485967580000001f-2"
}
length = 2
separator = "-"
}
# module.eks.random_pet.workers_launch_template[2] must be replaced
+/- resource "random_pet" "workers_launch_template" {
~ id = "present-buzzard" -> (known after apply)
~ keepers = { # forces replacement
~ "lt_name" = "us-prod-eks-cluster-us-prod-eks-worker-spot-subnet-039f7a3983b29c6a620200812074859682800000023-1" -> "us-prod-eks-cluster-us-prod-eks-worker-spot-subnet-039f7a3983b29c6a620200812074859682800000023-2"
}
length = 2
separator = "-"
}
lebenitza
on 12 Aug 2020
❤1
👍1
Because I use worker_groups_launch_template only for spot instances and I could live without them for a while, the only option was to comment it out, apply, add it back, apply again.
Plan is consistent now.
lebenitza
on 12 Aug 2020
👍1
Almost Same environment like @lebenitza
- eks module 12.2.0
- terraform 0.13.2
Exactly Same error.
Error: Provider produced inconsistent final plan
When expanding the plan for module.eks.random_pet.workers_launch_template[1]
to include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/random" changed the planned action from
CreateThenDelete to DeleteThenCreate.
I was able to make this error disappear by removing TF state (instead of comment out->apply-> add back ->apply).
terraform state rm module.eks.aws_launch_template.workers_launch_template[0] \
module.eks.aws_launch_template.workers_launch_template[1] \
module.eks.random_pet.workers_launch_template[0] \
module.eks.random_pet.workers_launch_template[1]
# if you want to know how i get the full name of the state, you can do this :
terraform state list | grep launch
Note that when you terraform apply after this removal, expect that all autoscaling groups will be replaced.
I don't have issue because i have 5 groups and each group has set of machines.
This means if a one or two group(s) is deleted during terraform apply, the cluster will not be throttled
abdennour
on 19 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
Pratima
路
4Comments
max-rocket-internet
路
4Comments
discordianfish
路
3Comments
mojochao
路
3Comments
tokiwong
路
4Comments
Most helpful comment
@colijack @marcecaro @davisford This fix has been released in v6.0.2