Terraform-provider-aws: AWS provider InvalidParameterException: Creation of service was not idempotent - ECS with Spot instances

[macdrorepo]

Hello,

Do not know if that is a bug or my code is wrong. I want to create ECS cluster with EC2 spot instances with help of launch template and ASG. My code is as follows:

For ECS service, cluster, task definition:

resource "aws_ecs_cluster" "main" {
  name = "test-ecs-cluster"
}

resource "aws_ecs_service" "ec2_service" {
  for_each = data.aws_subnet_ids.all_subnets.ids
  name                              = "test-ec2-service-qaz"
  task_definition                   = aws_ecs_task_definition.task_definition.arn
  cluster                           = aws_ecs_cluster.main.id
  desired_count                     = 1
  launch_type                       = "EC2"
  health_check_grace_period_seconds = 10

  load_balancer {
    container_name   = "test-container"
    container_port   = 80
    target_group_arn = aws_lb_target_group.alb_ec2_ecs_tg.id
  }

  network_configuration {
    security_groups  = [aws_security_group.ecs_ec2.id]
    subnets          = [each.value]
    assign_public_ip = "false"
  }

  ordered_placement_strategy {
    type  = "binpack"
    field = "cpu"
  }
}

resource "aws_ecs_task_definition" "task_definition" {
  container_definitions    = data.template_file.task_definition_template.rendered
  family                   = "test-ec2-task-family"
  execution_role_arn       = aws_iam_role.ecs_task_exec_role_ec2_ecs.arn
  task_role_arn            = aws_iam_role.ecs_task_exec_role_ec2_ecs.arn
  network_mode             = "awsvpc"
  memory                   = 1024
  cpu                      = 1024
  requires_compatibilities = ["EC2"]

  lifecycle {
    create_before_destroy = true
  }
}

data "template_file" "task_definition_template" {
  template = file("${path.module}/templates/task_definition.json.tpl")
  vars = {
    container_port = var.container_port
    region         = var.region
    log_group      = var.cloudwatch_log_group
  }
}

Launch template:

resource "aws_launch_template" "template_for_spot" {
  name = "test-spor-ecs-launch-template"
  disable_api_termination = false
  instance_type = "t3.small"
  image_id = data.aws_ami.amazon_linux_2_ecs_optimized.id
  key_name = "FrankfurtRegion"
  user_data = data.template_file.user_data.rendered
  vpc_security_group_ids = [aws_security_group.ecs_ec2.id]
  monitoring {
    enabled = var.enable_spot == "true" ? false : true
  }
  block_device_mappings {
    device_name = "/dev/sda1"
    ebs {
      volume_size = 30
    }
  }
  iam_instance_profile {
    arn = aws_iam_instance_profile.ecs_instance_profile.arn
  }
  lifecycle {
    create_before_destroy = true
  }
}

data "template_file" "user_data" {
  template = file("${path.module}/user_data.tpl")
  vars = {
    cluster_name = aws_ecs_cluster.main.name
  }
}

ASG with scaling policy:

resource "aws_autoscaling_group" "ecs_spot_asg" {
  name = "test-asg-for-ecs"
  max_size = 4
  min_size = 2
  desired_capacity = 2
  termination_policies = [
    "OldestInstance"]
  vpc_zone_identifier = data.aws_subnet_ids.all_subnets.ids
  health_check_type = "ELB"
  health_check_grace_period = 300

  mixed_instances_policy {
    instances_distribution {
      on_demand_percentage_above_base_capacity = 0
      spot_instance_pools = 2
      spot_max_price = "0.03"
    }
    launch_template {
      launch_template_specification {
        launch_template_id = aws_launch_template.template_for_spot.id
        version = "$Latest"
      }
      override {
        instance_type = "t3.large"
      }
      override {
        instance_type = "t3.medium"
      }
      override {
        instance_type = "t3a.large"
      }
      override {
        instance_type = "t3a.medium"
      }
    }
  }
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_autoscaling_policy" "ecs_cluster_scale_policy" {
  autoscaling_group_name = aws_autoscaling_group.ecs_spot_asg.name
  name = "test-ecs-cluster-scaling-policy"
  policy_type = "TargetTrackingScaling"
  adjustment_type = "ChangeInCapacity"

  target_tracking_configuration {
    target_value = 70
    customized_metric_specification {
      metric_name = "ECS-cluster-metric"
      namespace = "AWS/ECS"
      statistic = "Average"
      metric_dimension {
        name = aws_ecs_cluster.main.name
        value = aws_ecs_cluster.main.name
      }
    }
  }
}

I'm getting :

Error: InvalidParameterException: Creation of service was not idempotent. "test-ec2-service-qaz"

on ecs.tf line 5, in resource "aws_ecs_service" "ec2_service": 5: resource "aws_ecs_service" "ec2_service" {

Was reading from other issues that it is becouse of usage lifecycle with create_before_destroy statement but it is not declared in my code.

[macdrorepo]

No one is doing ECS on spot instances :( ? Help :D

[karnauskas]

Why "false" is quoted as a string? Also I think no one is looking into issues, way too many.

[macdrorepo]

"False" is because I didn't want public IP as instances are suppose to be in private subnets, just because :) and well I dont know it doesnt change if the error is or not there.
Yes I can see that questions are completely skipped by community but didn't have better idea to ask for help. (stackoverflow is done no answer also there). Any way hard to believe no one has terraform code for such use case, spot instances and ecs.

[karnauskas]

If you had all terraform code shared, perhaps I could have a look.

[karnauskas]

So general rule is that if you have lifecycle { create_before_destroy = true } for a resource with unique name like name = "test-ec2-service-qaz", such resource can't be created before deleting old one, simple because you can't have 2 resources with same unique name. I would say try use name_prefix instead name.

And in this aws_ecs_service case, are you creating 3 services with same name using for_each?

[macdrorepo]

I've pushed update to ECS service, sorry previous version was older. Basically was trying with name_prefix but name i must have for this resource, so i did name = "myservice_${replace(timestamp(), ":", "-")}" which ends up with same error that service is idempotent. Dont know how implement this name_prfix. Also I gave up on for_each

[karnauskas]

I would be easier if question had yes or no answer. Try name = "myservice_${count.index}". I stilll have feeling you're creating 3 services with same name, and don't see any proof which would say otherwise. https://www.terraform.io/docs/configuration/functions/timestamp.html is not recommending using timestamp() as resource attribute, mainly because that would change every single time you run terraform plan/apply.

[macdrorepo]

Sorry - still learning all of this. Answering question from previous post - no i didn't want to create three services, just one.
EDIT: Did manage to start all of this as required, thank you for your time and help.

[karnauskas]

Cool, I assume this is not provider's issue – this issue can be closed.