Terraform-aws-eks: data.template_file.userdata Merge Error

[dannyleesmith]

I have issues

I'm submitting a...

• [X] bug report
• [ ] feature request
• [X] support request - read the FAQ first!
• [ ] kudos, thank you, warm fuzzy

What is the current behavior?

When attempting to update a Windows worker_group map to include kubelet_extra_args an error is thrown for:

Error: Invalid function argument

  on .terraform/modules/eks_cluster/terraform-aws-eks-12.2.0/data.tf line 99, in data "template_file" "userdata":
  96: 
  97: 
  98: 
  99:       local.workers_group_defaults["userdata_template_extra_args"]
 100: 
    |----------------
    | local.workers_group_defaults["userdata_template_extra_args"] is object with no attributes

Invalid value for "default" parameter: the default value must have the same
type as the map elements.

If this is a bug, how to reproduce? Please include a code sample if relevant.

Working worker_groups:

worker_groups = [
  {
    name                 = "linux"
    additional_userdata  = "yum update -y ; yum install -y kernel-headers kernel-devel dkms ; reboot"
    asg_max_size         = 4
    asg_min_size         = 1
    asg_desired_capacity = 1
    instance_type        = "t3.xlarge"
    root_volume_size     = 50
    platform             = "linux"
  },
  {
    name                 = "windows"
    additional_userdata  = "echo windows"
    instance_type        = "t3.xlarge"
    platform             = "windows"
    asg_max_size         = 4
    asg_min_size         = 1
    asg_desired_capacity = 1
    root_volume_size     = 50
  },
]

I want to supplement this by updating my worker_groups with kubelet_extra_args which should be a string as seen in local.tf.

My new Windows map looks like this:

  {
    name                 = "windows"
    additional_userdata  = "echo windows"
    kubelet_extra_args   = "--register-with-taints=windowsContainers=true:NoSchedule"
    instance_type        = "t3.xlarge"
    platform             = "windows"
    asg_max_size         = 4
    asg_min_size         = 1
    asg_desired_capacity = 1
    root_volume_size     = 50
  },

This should now mean my extra arg is a part of the the data.template_file.userdata object as seen here: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/data.tf#L90-L94

What I actually get is the error above in What is the current behavior?.

What's the expected behavior?

No error.

Are you able to fix this problem and submit a PR? Link here if you have already.

Environment details

• Affected module version: v12.2.0
• OS: MacOS and Ubuntu
• Terraform version: 0.12.26

Any other relevant info

This is the bit that's failing: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/data.tf#L96-L100

Having not passed userdata_template_extra_args in my worker_groups map it is of course backing off to the default which is: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/local.tf#L58

You can see this is an empty map (which the error suggests may be part of the problem), and if I try to pass in this map (as it seems I should be able to do for an override) I get this error which makes me think it's not possible to pass in some of these overrides:

The given value is not valid for variable "worker_groups": all list elements
must have the same type.

[daroga0002]

does issue is for new worker group creation or updating existing definition?

If you can please perform test where you adding third group with different parameters and then rerun TF, if succeed then try to remove legacy definition.

[dannyleesmith]

If I do this original I get no errors:

worker_groups = [
  {
    name                 = "linux"
    additional_userdata  = "yum update -y ; yum install -y kernel-headers kernel-devel dkms ; reboot"
    asg_max_size         = 4
    asg_min_size         = 1
    asg_desired_capacity = 1
    instance_type        = "t3.xlarge"
    root_volume_size     = 50
    platform             = "linux"
  },
  {
    name                 = "windows"
    additional_userdata  = "echo windows"
    instance_type        = "t3.xlarge"
    platform             = "windows"
    asg_max_size         = 4
    asg_min_size         = 1
    asg_desired_capacity = 1
    root_volume_size     = 50
  },
]

If I edit the Windows one to include the kubelet_extra_args I get these errors (note 2):

Error: Invalid function argument

  on .terraform/modules/eks_cluster/terraform-aws-eks-12.2.0/data.tf line 99, in data "template_file" "userdata":
  96:
  97:
  98:
  99:       local.workers_group_defaults["userdata_template_extra_args"]
 100:
    |----------------
    | local.workers_group_defaults["userdata_template_extra_args"] is object with no attributes

Invalid value for "default" parameter: the default value must have the same
type as the map elements.


Error: Invalid function argument

  on .terraform/modules/eks_cluster/terraform-aws-eks-12.2.0/data.tf line 99, in data "template_file" "userdata":
  96:
  97:
  98:
  99:       local.workers_group_defaults["userdata_template_extra_args"]
 100:
    |----------------
    | local.workers_group_defaults["userdata_template_extra_args"] is object with no attributes

Invalid value for "default" parameter: the default value must have the same
type as the map elements.

If I reset the original Windows map and add the third as requested I get these errors (note now 3):

Error: Invalid function argument

  on .terraform/modules/eks_cluster/terraform-aws-eks-12.2.0/data.tf line 99, in data "template_file" "userdata":
  96:
  97:
  98:
  99:       local.workers_group_defaults["userdata_template_extra_args"]
 100:
    |----------------
    | local.workers_group_defaults["userdata_template_extra_args"] is object with no attributes

Invalid value for "default" parameter: the default value must have the same
type as the map elements.


Error: Invalid function argument

  on .terraform/modules/eks_cluster/terraform-aws-eks-12.2.0/data.tf line 99, in data "template_file" "userdata":
  96:
  97:
  98:
  99:       local.workers_group_defaults["userdata_template_extra_args"]
 100:
    |----------------
    | local.workers_group_defaults["userdata_template_extra_args"] is object with no attributes

Invalid value for "default" parameter: the default value must have the same
type as the map elements.


Error: Invalid function argument

  on .terraform/modules/eks_cluster/terraform-aws-eks-12.2.0/data.tf line 99, in data "template_file" "userdata":
  96:
  97:
  98:
  99:       local.workers_group_defaults["userdata_template_extra_args"]
 100:
    |----------------
    | local.workers_group_defaults["userdata_template_extra_args"] is object with no attributes

Invalid value for "default" parameter: the default value must have the same
type as the map elements.

[tinyzimmer]

For whatever it's worth I just hit this same error with the example pulled right out of the README.

EDIT: Same bug on multiple terraform versions, 0.12 and 0.13. If I fork and take out the problematic lookup the next ones are additional_security_group_ids and additional_ebs_volumes for the dynamic block devices in workers.tf.

The underlying issue appears to be the fact that there are different types in the map in question.

[tinyzimmer]

Actually...well I haven't tested this theory thoroughly...but I think it's because I was passing the worker_groups variable the wrong type of object. I was passing it a list(map(string)) and I should have been passing it a list(object({})).

EDIT: This was actually the issue, and I think the best way to fix it would be to statically define the type passed to that variable.

variable "worker_groups" {
  description = "A list of maps defining worker group configurations to be defined using AWS Launch Configurations. See workers_group_defaults for valid keys."
  type        = any  # here
  default     = []
}

I assume terraform would have yelled at me earlier on if that was set to type = list(object({})). Just hopefully, that wouldn't cause people passing typed objects any pain.

[dannyleesmith]

Hi @tinyzimmer, apologies for the delay in responding. Yep, absolutely it turns out to be an error on our side with the var we were populating and trying to pass in.

Closing.

[amitsehgal]

@dannyleesmith I'm getting similar error while worker groups are passed as a parameter. I have tried list of maps and objects..does not help. could you share , how did you fix it ?
variable declaration

variable "additional_nodes" {
  type = map
}

```
additional_nodes = {
name = "worker-group-win"
instance_type = "t2.large"
platform = "windows"
asg_desired_capacity = 3
}

usage:

module "eks-cluster" {
.
.
.
worker_groups_launch_template =[
{
name = var.worker_group_name
desired_capacity = var.od_desired_capacity
max_capacity = var.od_max_capacity
min_capacity = var.od_min_capacity
public_ip = false
pre_userdata = templatefile("${path.module}/workers_user_data.sh.tpl", { pre_userdata = var.pre_userdata })
instance_type = var.instance_type
platform = "linux"
# Addition of tags indicate that this ASG will participate in auto-scaling
# through the cluster-autoscaler
tags = [for k, v in local.cluster_autoscaler.asg_tags : {
key = k
value = v
propagate_at_launch = false
}]
}, var.additional_nodes
]
}
```

[dannyleesmith]

@amitsehgal in our case the module we were deploying had a var of worker_groups which should have been set to list(map(object{})) but was set to list(map) so the mixed types were not allowed. In your case it looks like you are not passing in your var properly to worker_groups_launch_template? If you can provider a full example of what you are doing and the error you are facing I can try to assist.

[amitsehgal]

Thanks for prompt response. Let me try to elaborate issue. I have a submodule on top of terraform-aws-eks ... let's say my-module; which calls terraform-aws-eks.. There are sevral users of my-module. So far; my-module has only linux nodes and it only takes variables such as worker group name, desired capacity etc. As a next step, i'm planing to add windows nodes. And would like caller of my-module to pass the entire block

{
     name                          = "worker-group-win"
     instance_type                 = "t2.large"
     platform                      = "windows"
     asg_desired_capacity          = 3
 }
 ```
as variable.
So, i declared a variable in my-module 

variable "additional_nodes" {
type = map
}

caller of my module is passing it as:

additional_nodes = {
name = "worker-group-win"
instance_type = "t2.large"
platform = "windows"
asg_desired_capacity = 3
}

module "eks-cluster" {
source = "github.com/terraform-aws-modules/terraform-aws-eks?ref=v12.2.0"
cluster_name = local.cluster_name
subnets = data.aws_subnet_ids.public_private.ids
vpc_id = data.aws_vpc.main.id
enable_irsa = var.enable_irsa
wait_for_cluster_interpreter = ["C:/Program Files/Git/bin/sh.exe", "-c"]
wait_for_cluster_cmd = "until curl -k -s $ENDPOINT/healthz >/dev/null; do sleep 4; done"
cluster_version = var.cluster_version
tags = local.common_tags

workers_additional_policies = var.workers_additional_policies
workers_role_name = var.workers_role_name

write_kubeconfig = true
config_output_path = "~/kubeconfig"
map_roles = local.map_roles

workers_group_defaults = {
subnets = data.aws_subnet_ids.private.ids
}

worker_groups_launch_template = [
{
name = var.worker_group_name
desired_capacity = var.od_desired_capacity
max_capacity = var.od_max_capacity
min_capacity = var.od_min_capacity
public_ip = false
pre_userdata = templatefile("${path.module}/workers_user_data.sh.tpl", { pre_userdata = var.pre_userdata })
instance_type = var.instance_type
platform = "linux"
# Addition of tags indicate that this ASG will participate in auto-scaling
# through the cluster-autoscaler
tags = [for k, v in local.cluster_autoscaler.asg_tags : {
key = k
value = v
propagate_at_launch = false
}]
}, var.additional_nodes // This is where i'm using variable
]
}
```

if I replace var.additional_nodes // This is where i'm using variable line with block. it works..but the variable is above error.
If I use the block instead of variable it works....only while passing it as variable from caller. it throws the error

Error: Invalid function argument

on .terraform/modules/eks.eks-cluster/data.tf line 145, in data "template_file" "launch_template_userdata":
142:
143:
144:
145: local.workers_group_defaults["userdata_template_extra_args"]
146:
|----------------
| local.workers_group_defaults["userdata_template_extra_args"] is object with no attributes

Invalid value for "default" parameter: the default value must have the same
type as the map elements.

[dannyleesmith]

It shouldn't make a difference in my opinion but can you try:

additional_nodes = {
     name                          = "worker-group-win"
     instance_type                 = "t2.large"
     platform                      = "windows"
     asg_desired_capacity          = "3"
 }

It should make that 3 a string automatically in my opinion but just in case...

Alternatively:

variable "additional_nodes" {
  type = map(string)
}

And also try splitting the use of the var onto its own line:

    }, 
    var.additional_nodes,

Beyond that I would have thought it'd be fine.

[amitsehgal]

no luck with map(string) ... removing the type all together works fine :)

variable "additional_nodes" {
}

[amitsehgal]

it worked for happy path....when I pass values to additional_nodes variable....does not work when variableis empty...seems like module takes all default and create another worker launch template even for null object..weired