Terraform-provider-aws: Terraform forcing replacement of ECS task definition - (only on task definitions with mount points)

[hashibot[bot]]

_This issue was originally opened by @im-lcoupe as hashicorp/terraform#23780. It was migrated here as a result of the provider split. The original body of the issue is below._

---

Summary

Hi there,

So this only seems to have become a problem since upgrading my code to the latest version - and strangely only seems to happen on the task definitions with mount points (however, the format of them hasn't changed...)

Terraform will constantly try and replace the two task definitions regardless of whether any changes have been made to them...

Any guidance on this would be greatly appreciated, as it means the task definition revision is changing on every run (which of course is not ideal)...

Terraform Version

0.12.18

Terraform Configuration Files - 1st problematic task definition - followed by the second

[
  {
    "name": "${container_name_nginx}",
    "image": "${container_image_nginx}",
    "memory": ${container_memory},
    "cpu": ${container_cpu},
    "networkMode": "awsvpc",
    "volumesFrom": [],
    "essential": true,
    "portMappings": [
      {
        "containerPort": 443,
        "hostPort": 443,
        "protocol": "tcp"
      }
    ],
    "mountPoints": [
     {
    "readOnly": false,
    "containerPath": "/var/www/symfony/var/log",
    "sourceVolume": "shared_symfony_logs"
     },
     {
     "readOnly": false,
     "containerPath": "/var/log/nginx",
     "sourceVolume": "shared_nginx_logs"
     }
    ],
    "environment": [
      {
        "name": "${env_var_1_name}",
        "value": "${env_var_value_1}"
      },
      {
        "name": "${env_var_2_name}",
        "value": "${env_var_value_2}"
      },
      {
        "name": "${env_var_3_name}",
        "value": "${env_var_value_3}"
      }
    ],
  "logConfiguration" : {
    "logDriver" : "awslogs",
    "options" :{
      "awslogs-create-group": "true",
      "awslogs-group": "${container_name_nginx}",
      "awslogs-region": "${platform_region}",
      "awslogs-stream-prefix": "ecs"
    }
  }
},

{
  "name": "${container_name_php}",
  "image": "${container_image_php}",
  "memory": ${container_memory},
  "cpu": ${container_cpu},
  "networkMode": "awsvpc",
  "volumesFrom": [],
  "essential": true,
  "portMappings": [
    {
      "containerPort": 9000,
      "hostPort": 9000,
      "protocol": "tcp"
    }
  ],
  "mountPoints": [
   {
  "readOnly": false,
  "containerPath": "/var/www/symfony/var/log",
  "sourceVolume": "shared_symfony_logs"
   },
   {
   "readOnly": false,
   "containerPath": "/var/log/nginx",
   "sourceVolume": "shared_nginx_logs"
   }
  ],
  "environment": [
    {
      "name": "${env_var_4_name}",
      "value": "${env_var_value_4}"
    },
    {
      "name": "${env_var_5_name}",
      "value": "${env_var_value_5}"
    },
    {
      "name": "${env_var_6_name}",
      "value": "${env_var_value_6}"
    },
    {
      "name": "${env_var_7_name}",
      "value": "${env_var_value_7}"
    },
    {
      "name": "${env_var_8_name}",
      "value": "${env_var_value_8}"
    },
    {
      "name": "${env_var_10_name}",
      "value": "${env_var_value_10}"
    },
    {
      "name": "${env_var_11_name}",
      "value": "${env_var_value_11}"
    },
    {
      "name": "${env_var_12_name}",
      "value": "${env_var_value_12}"
    },
    {
      "name": "${env_var_13_name}",
      "value": "${env_var_value_13}"
    },
    {
      "name": "${env_var_14_name}",
      "value": "${env_var_value_14}"
    },
    {
      "name": "${env_var_15_name}",
      "value": "${env_var_value_15}"
    },
    {
      "name": "${env_var_16_name}",
      "value": "${env_var_value_16}"
    },
    {
      "name": "${env_var_17_name}",
      "value": "${env_var_value_17}"
    },
    {
      "name": "${env_var_18_name}",
      "value": "${env_var_value_18}"
    },
    {
      "name": "${env_var_19_name}",
      "value": "${env_var_value_19}"
    },
    {
      "name": "${env_var_20_name}",
      "value": "${env_var_value_20}"
    },
    {
      "name": "${env_var_21_name}",
      "value": "${env_var_value_21}"
    },
    {
      "name": "${env_var_22_name}",
      "value": "${env_var_value_22}"
    },
    {
      "name": "${env_var_1_name}",
      "value": "${env_var_value_1}"
    },
    {
      "name": "${env_var_28_name}",
      "value": "${env_var_value_28}"
    },
    {
      "name": "${env_var_29_name}",
      "value": "${env_var_value_29}"
    },
    {
      "name": "${env_var_30_name}",
      "value": "${env_var_value_30}"
    },
    {
      "name": "${env_var_31_name}",
      "value": "${env_var_value_31}"
    },
    {
      "name": "${env_var_32_name}",
      "value": "${env_var_value_32}"
    },
    {
      "name": "${env_var_33_name}",
      "value": "${env_var_value_33}"
    },
    {
      "name": "${env_var_34_name}",
      "value": "${env_var_value_34}"
    },
    {
      "name": "${env_var_35_name}",
      "value": "${env_var_value_35}"
    },
    {
      "name": "${env_var_36_name}",
      "value": "${env_var_value_36}"
    },
    {
      "name": "${env_var_37_name}",
      "value": "${env_var_value_37}"
    },
    {
      "name": "${env_var_38_name}",
      "value": "${env_var_value_38}"
    }
  ],
  "secrets":[
    {
      "name":"${sensitive_var_1}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_1}"
    },
    {
      "name":"${sensitive_var_2}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_2}"
    },
    {
      "name":"${sensitive_var_3}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_3}"
    },
    {
      "name":"${sensitive_var_4}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_4}"
    },
    {
      "name":"${sensitive_var_5}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_5}"
    },
    {
      "name":"${sensitive_var_6}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_6}"
    },
    {
      "name":"${sensitive_var_7}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_7}"
    },
    {
      "name":"${sensitive_var_8}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_8}"
    },
    {
      "name":"${sensitive_var_9}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_9}"
    },
    {
      "name":"${sensitive_var_10}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_10}"
    },
    {
      "name":"${sensitive_var_11}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_11}"
    },
    {
      "name":"${sensitive_var_12}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_12}"
    },
    {
      "name":"${sensitive_var_13}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_13}"
    }
  ],
"logConfiguration" : {
  "logDriver" : "awslogs",
  "options" :{
    "awslogs-create-group": "true",
    "awslogs-group": "${container_name_php}",
    "awslogs-region": "${platform_region}",
    "awslogs-stream-prefix": "ecs"
  }
}
},


{
  "name": "${container_name_logstash}",
  "image": "${container_image_logstash}",
  "memory": ${container_memory},
  "cpu": ${container_cpu},
  "networkMode": "awsvpc",
  "volumesFrom": [],
  "essential": true,
  "portMappings": [
    {
      "containerPort": 9600,
      "hostPort": 9600,
      "protocol": "tcp"
    }
  ],
  "mountPoints": [
   {
  "readOnly": false,
  "containerPath": "/var/www/symfony/var/log",
  "sourceVolume": "shared_symfony_logs"
   },
   {
   "readOnly": false,
   "containerPath": "/var/log/nginx",
   "sourceVolume": "shared_nginx_logs"
   }
 ],
 "environment": [
   {
     "name": "${env_var_23_name}",
     "value": "${env_var_value_23}"
   },
   {
     "name": "${env_var_1_name}",
     "value": "${env_var_value_1}"
   }
 ],
"logConfiguration" : {
  "logDriver" : "awslogs",
  "options" :{
    "awslogs-create-group": "true",
    "awslogs-group": "${container_name_logstash}",
    "awslogs-region": "${platform_region}",
    "awslogs-stream-prefix": "ecs"
    }
  }
}
]

Second Task definition

[
{
  "name": "${container_name_php}",
  "image": "${container_image_php}",
  "memory": ${container_memory},
  "cpu": ${container_cpu},
  "networkMode": "awsvpc",
  "volumesFrom": [],
  "essential": true,
  "command": ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"],
  "portMappings": [
    {
      "containerPort": 9001,
      "hostPort": 9001,
      "protocol": "tcp"
    }
  ],
  "mountPoints": [
   {
  "readOnly": false,
  "containerPath": "/var/www/symfony/var/log",
  "sourceVolume": "shared_worker_logs"
   }
 ],
  "environment": [
    {
      "name": "${env_var_4_name}",
      "value": "${env_var_value_4}"
    },
    {
      "name": "${env_var_5_name}",
      "value": "${env_var_value_5}"
    },
    {
      "name": "${env_var_6_name}",
      "value": "${env_var_value_6}"
    },
    {
      "name": "${env_var_7_name}",
      "value": "${env_var_value_7}"
    },
    {
      "name": "${env_var_8_name}",
      "value": "${env_var_value_8}"
    },
    {
      "name": "${env_var_10_name}",
      "value": "${env_var_value_10}"
    },
    {
      "name": "${env_var_11_name}",
      "value": "${env_var_value_11}"
    },
    {
      "name": "${env_var_12_name}",
      "value": "${env_var_value_12}"
    },
    {
      "name": "${env_var_13_name}",
      "value": "${env_var_value_13}"
    },
    {
      "name": "${env_var_14_name}",
      "value": "${env_var_value_14}"
    },
    {
      "name": "${env_var_15_name}",
      "value": "${env_var_value_15}"
    },
    {
      "name": "${env_var_16_name}",
      "value": "${env_var_value_16}"
    },
    {
      "name": "${env_var_17_name}",
      "value": "${env_var_value_17}"
    },
    {
      "name": "${env_var_18_name}",
      "value": "${env_var_value_18}"
    },
    {
      "name": "${env_var_19_name}",
      "value": "${env_var_value_19}"
    },
    {
      "name": "${env_var_20_name}",
      "value": "${env_var_value_20}"
    },
    {
      "name": "${env_var_21_name}",
      "value": "${env_var_value_21}"
    },
    {
      "name": "${env_var_22_name}",
      "value": "${env_var_value_22}"
    },
    {
      "name": "${env_var_1_name}",
      "value": "${env_var_value_1}"
    },
    {
      "name": "${env_var_28_name}",
      "value": "${env_var_value_28}"
    },
    {
      "name": "${env_var_29_name}",
      "value": "${env_var_value_29}"
    },
    {
      "name": "${env_var_30_name}",
      "value": "${env_var_value_30}"
    },
    {
      "name": "${env_var_31_name}",
      "value": "${env_var_value_31}"
    },
    {
      "name": "${env_var_32_name}",
      "value": "${env_var_value_32}"
    },
    {
      "name": "${env_var_33_name}",
      "value": "${env_var_value_33}"
    },
    {
      "name": "${env_var_34_name}",
      "value": "${env_var_value_34}"
    },
    {
      "name": "${env_var_35_name}",
      "value": "${env_var_value_35}"
    },
    {
      "name": "${env_var_36_name}",
      "value": "${env_var_value_37}"
    },
    {
      "name": "${env_var_37_name}",
      "value": "${env_var_value_37}"
    },
    {
      "name": "${env_var_38_name}",
      "value": "${env_var_value_38}"
    }
  ],
  "secrets":[
    {
      "name":"${sensitive_var_1}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_1}"
    },
    {
      "name":"${sensitive_var_2}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_2}"
    },
    {
      "name":"${sensitive_var_3}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_3}"
    },
    {
      "name":"${sensitive_var_4}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_4}"
    },
    {
      "name":"${sensitive_var_5}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_5}"
    },
    {
      "name":"${sensitive_var_6}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_6}"
    },
    {
      "name":"${sensitive_var_7}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_7}"
    },
    {
      "name":"${sensitive_var_8}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_8}"
    },
    {
      "name":"${sensitive_var_9}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_9}"
    },
    {
      "name":"${sensitive_var_10}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_10}"
    },
    {
      "name":"${sensitive_var_11}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_11}"
    },
    {
      "name":"${sensitive_var_12}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_12}"
    },
    {
      "name":"${sensitive_var_13}",
      "valueFrom": "arn:aws:ssm:${platform_region}:${aws_account_number}:parameter/${product}/${service_1}/${environment}/${sensitive_var_13}"
    }
  ],
"logConfiguration" : {
  "logDriver" : "awslogs",
  "options" :{
    "awslogs-create-group": "true",
    "awslogs-group": "${container_name_php}",
    "awslogs-region": "${platform_region}",
    "awslogs-stream-prefix": "ecs"
  }
}
},


{
  "name": "${container_name_logstash}",
  "image": "${container_image_logstash}",
  "memory": ${container_memory},
  "cpu": ${container_cpu},
  "networkMode": "awsvpc",
  "volumesFrom": [],
  "essential": true,
  "portMappings": [
    {
      "containerPort": 9600,
      "hostPort": 9600,
      "protocol": "tcp"
    }
  ],
  "mountPoints": [
   {
  "readOnly": false,
  "containerPath": "/var/www/symfony/var/log",
  "sourceVolume": "shared_worker_logs"
   }
 ],
 "environment": [
   {
     "name": "${env_var_23_name}",
     "value": "${env_var_value_23}"
   },
   {
     "name": "${env_var_1_name}",
     "value": "${env_var_value_1}"
   }
 ],
"logConfiguration" : {
  "logDriver" : "awslogs",
  "options" :{
    "awslogs-create-group": "true",
    "awslogs-group": "${container_name_logstash}",
    "awslogs-region": "${platform_region}",
    "awslogs-stream-prefix": "ecs"
    }
  }
}
]

Example plan output - to me it isn't clear what exactly it needs to change which requires a forced replacement - it looks to remove vars that are already there and then re add them? in other places it seems to re order them also. It also looks to be adding the network mode (awsvpc) which is already defined in the task definition?

      ~ container_definitions    = jsonencode(
          ~ [ # forces replacement
              ~ {
                    cpu              = 512
                  ~ environment      = [
                      - {
                          - name  = "PHP_FPM_PORT"
                          - value = "9000"
                        },
                      - {
                          - name  = "PHP_FPM_HOST"
                          - value = "localhost"
                        },
                        {
                            name  = "APP_ENV"
                            value = "prod"
                        },
                      + {
                          + name  = "PHP_FPM_HOST"
                          + value = "localhost"
                        },
                      + {
                          + name  = "PHP_FPM_PORT"
                          + value = "9000"
                        },
                    ]
                    essential        = true
                    image            = "###########################:latest"
                    logConfiguration = {
                        logDriver = "awslogs"
                        options   = {
                            awslogs-create-group  = "true"
                            awslogs-group         = "###############"
                            awslogs-region        = "eu-west-1"
                            awslogs-stream-prefix = "ecs"
                        }
                    }
                    memory           = 1024
                    mountPoints      = [
                        {
                            containerPath = "/var/www/symfony/var/log"
                            readOnly      = false
                            sourceVolume  = "shared_symfony_logs"
                        },
                        {
                            containerPath = "/var/log/nginx"
                            readOnly      = false
                            sourceVolume  = "shared_nginx_logs"
                        },
                    ]
                    name             = "##################"
                  + networkMode      = "awsvpc"
                    portMappings     = [
                        {
                            containerPort = 443
                            hostPort      = 443
                            protocol      = "tcp"
                        },
                    ]
                    volumesFrom      = []
                } # forces replacement,

Expected Behavior

Terraform should not try and replace the task definitions on every plan.

Actual Behavior

Terraform forces replacement of the task definition on every plan.

Steps to Reproduce

Terraform plan

[moyuanhuang]

I'm having the same issue. I think it's ordering the custom environment variables, plus adding some default configurations if you don't already have them in your task definition. For my case I had to add these default options and reorder the environment variable according to the diff output.

It'd be nice if terraform can

1. compare the environment variables as a real hash (so that the order doesn't matter)
2. avoid updating the task definitions because of the absence of some default variables.

[reedflinch]

Going off of @moyuanhuang I also suspect the issue is the ordering of environment variables. I do NOT see the issue with secrets. One thing to note for my use case is I am changing the image of the task definition. So I do expect a new task definition to be created with the new image, but I do not expect to see a diff for unchanging environment variables.

This makes evaluating diffs for task definitions extremely difficult.

I notice the that AWS API + CLI do return these arrays in a consistent order (from what I can see), so perhaps this is something that Terraform or the provider itself is doing.

[LeComptoirDesPharmacies]

Hi, I having the same issue without mount points.
In addition of reordering custom environment variables, I have some variable set at null who indicate terraform to recreate the task definition.

Exemple with docker health check :

~ healthCheck      = {
                        command     = [
                            "CMD-SHELL",
                            "agent health",
                        ]
                        interval    = 15
                        retries     = 10
                        startPeriod = 15
                      - timeout     = 5 -> null
                    }

[moyuanhuang]

@LeComptoirDesPharmacies That probably means there's a default value set for this particular config to be 5. However because you don't specify that config in your task definition so terraform thinks that you're trying to set it to null (which is never gonna happen since there is an enforced default). Add

timeout = 5

...to your task definition and you should be able to avoid terraform recreating the task.

[LeComptoirDesPharmacies]

@moyuanhuang Yes thanks.
But I have the problems with custom environment variables too.
I found this fix who is waiting to be merge :

• https://github.com/terraform-providers/terraform-provider-aws/pull/11463

[jonesmac]

I found that alphabetizing my env variables by name seems to keep it out of the plan. Noticed that the ecs task definition stores them that way in the json output in the console.