Terraform: Terraform keeps reapplying same change every time

[polasekr]

Terraform version:
Terraform v0.7.13

Issue:
I use terraform with AWS provider. Every time I run terraform apply, it applies same changes over and over again. Specifically it relates to aws_route_table and aws_opsworks_stack resource listed below. From the terraform plan output is appears that terraform thinks that ssh key and nat gateway have changed. See below. It looks to me like a some sort of bug. Your help is appreciated.

`~ aws_opsworks_stack.stg-hubub-infra
    custom_cookbooks_source.0.ssh_key: "" => "###content has been sanitized since it contans sensitive information###"

~ aws_route_table.Infra-private-route-table
    route.1167944804.cidr_block:                "" => "0.0.0.0/0"
    route.1167944804.gateway_id:                "" => "nat-0539455f0961a2b3f"
    route.1167944804.instance_id:               "" => ""
    route.1167944804.nat_gateway_id:            "" => ""
    route.1167944804.network_interface_id:      "" => ""
    route.1167944804.vpc_peering_connection_id: "" => ""
    route.1239881238.cidr_block:                "172.30.0.0/16" => "172.30.0.0/16"
    route.1239881238.gateway_id:                "" => ""
    route.1239881238.instance_id:               "" => ""
    route.1239881238.nat_gateway_id:            "" => ""
    route.1239881238.network_interface_id:      "" => ""
    route.1239881238.vpc_peering_connection_id: "pcx-0e50c067" => "pcx-0e50c067"
    route.1456406231.cidr_block:                "172.20.0.0/16" => "172.20.0.0/16"
    route.1456406231.gateway_id:                "" => ""
    route.1456406231.instance_id:               "" => ""
    route.1456406231.nat_gateway_id:            "" => ""
    route.1456406231.network_interface_id:      "" => ""
    route.1456406231.vpc_peering_connection_id: "pcx-9a0a8ff3" => "pcx-9a0a8ff3"
    route.1671617090.cidr_block:                "0.0.0.0/0" => ""
    route.1671617090.gateway_id:                "" => ""
    route.1671617090.instance_id:               "" => ""
    route.1671617090.nat_gateway_id:            "nat-0539455f0961a2b3f" => ""
    route.1671617090.network_interface_id:      "" => ""
    route.1671617090.vpc_peering_connection_id: "" => ""`

Affected resources:

resource "aws_route_table" "Infra-private-route-table" {
    vpc_id = "${aws_vpc.infrastructure-vpc.id}"
    //ignore_changes = [ "route" ]
    route {
                cidr_block = "0.0.0.0/0"
                gateway_id = "${aws_nat_gateway.Infra-NAT.id}"
    }
    route {
                cidr_block = "${aws_vpc.kubernetes-vpc.cidr_block}"
                vpc_peering_connection_id = "${aws_vpc_peering_connection.peer-infra-k8s.id}"
    }
    route {
                cidr_block = "${aws_vpc.kubernetes-stg-vpc.cidr_block}"
                vpc_peering_connection_id = "${aws_vpc_peering_connection.peer-infra-k8s_stg.id}"
    }
    tags  {
        Name = "Infra private route table (172.10.0.0)"
    }
}`
`resource "aws_opsworks_stack" "stg-hubub-infra" {
    //ignore_changes = [ "custom_cookbooks*" ]
    name = "stg-hubub-infra"
    region = "us-east-1"
    service_role_arn = "arn:aws:iam::533108516651:role/aws-opsworks-service-role"
    default_instance_profile_arn = "arn:aws:iam::533108516651:instance-profile/aws-opsworks-ec2-role"
    agent_version = "4019-20161019094031"
    color = "rgb(209, 105, 41)"
    configuration_manager_name = "Chef"
    configuration_manager_version = "12"
    //default_availability_zone = "us-east-1b"
    default_os = "Ubuntu 16.04 LTS"
    default_root_device_type = "ebs"
    default_ssh_key_name = "deployment-key"
    default_subnet_id = "${aws_subnet.infra-subnet-1b.id}"
    hostname_theme = "Layer_Dependent"
    name = "stg-hubub-infra"
    use_custom_cookbooks = "true"
    use_opsworks_security_groups = "false"
    vpc_id = "${aws_vpc.infrastructure-vpc.id}"
    custom_cookbooks_source = {
        revision = "production"
        ssh_key = "${file("gitblit.pem")}"
        type = "git"
        url = "[email protected]:/srv/git/opscookbooks.git"
    }
    custom_json = <<EOT
{
#### code sanitized here snce it contains sensitive information ####
}
EOT
}

[stack72]

Hi @polasekr

Thanks for the issue report here - the issue here is that you are trying to assign a NAT Gateway ID to the Internet Gateway ID

The config for your route table should look as follows:

resource "aws_route_table" "Infra-private-route-table" {
    vpc_id = "${aws_vpc.infrastructure-vpc.id}"
    //ignore_changes = [ "route" ]
    route {
                cidr_block = "0.0.0.0/0"
                nat_gateway_id = "${aws_nat_gateway.Infra-NAT.id}"
    }
    route {
                cidr_block = "${aws_vpc.kubernetes-vpc.cidr_block}"
                vpc_peering_connection_id = "${aws_vpc_peering_connection.peer-infra-k8s.id}"
    }
    route {
                cidr_block = "${aws_vpc.kubernetes-stg-vpc.cidr_block}"
                vpc_peering_connection_id = "${aws_vpc_peering_connection.peer-infra-k8s_stg.id}"
    }
    tags  {
        Name = "Infra private route table (172.10.0.0)"
    }
}

That should then fix the issue

Please let me know if this is not the case

Thanks

Paul

[polasekr]

Hi Stack72,

The change worked for aws_route_table. I still experience the issue with resource aws_opsworks_stack. From terraform plan output it looks like it is trying to replace blank ssh key coming back from AWS with private ssh key.

~ aws_opsworks_stack.stg-hubub-infra custom_cookbooks_source.0.ssh_key: "" => "###content has been sanitized since it contans sensitive information###"

[martin-flaregames]

@polasekr I have the same "issue" with the custom_cookbooks_source.0.ssh_key entry. The problem here is not terraform but the AWS API: You cannot fetch the SSH key over the API, and this intended and desired to be that way.

So, when terraform performs a refresh of the OpsWorks stack resource it correctly detects the delta, because Amazon is not delivering the SSH key.

It is also correct that terraform applies the changes again, because someone(tm) could manually have changed the key using the AWS console.

I personally got used to mentally ignore the change of the SSH key during plan and apply.

I don't think that this can be easily solved, if solved at all, on the terraform side

[martin-flaregames]

@polasekr Me again, if you are sure that the SSH key will never be manually changed using the AWS Console, then you may opt to ignore the changes on that attribute: https://www.terraform.io/docs/configuration/resources.html#ignore_changes

Just use terraform show to see the proper name of the attribute you want to ignore during plan and apply.

Google may give you some additional information code examples on how to achieve this.

[polasekr]

I tried ignore_changes, but it gives me an error:

robert@razer-laptop:~/src/hubub-terraform$ terraform plan
There are warnings and/or errors related to your configuration. Please
fix these before continuing.

Errors:

  * aws_opsworks_stack.stg-hubub-infra: : invalid or unknown key: ignore_changes

resource "aws_opsworks_stack" "stg-hubub-infra" {
    ignore_changes = [ "custom_cookbooks*" ]
    name = "stg-hubub-infra"
    region = "us-east-1"
    service_role_arn = "arn:aws:iam::533108516651:role/aws-opsworks-service-role"
    default_instance_profile_arn = "arn:aws:iam::533108516651:instance-profile/aws-opsworks-ec2-role"
    agent_version = "4019-20161019094031"
    color = "rgb(209, 105, 41)"
    configuration_manager_name = "Chef"
    configuration_manager_version = "12"
    //default_availability_zone = "us-east-1b"
    default_os = "Ubuntu 16.04 LTS"
    default_root_device_type = "ebs"
    default_ssh_key_name = "deployment-key"
    default_subnet_id = "${aws_subnet.infra-subnet-1b.id}"
    hostname_theme = "Layer_Dependent"
    name = "stg-hubub-infra"
    use_custom_cookbooks = "true"
    use_opsworks_security_groups = "false"
    vpc_id = "${aws_vpc.infrastructure-vpc.id}"
    custom_cookbooks_source = {
        revision = "production"
        ssh_key = "${file("gitblit.pem")}"
        type = "git"
        url = "[email protected]:/srv/git/opscookbooks.git"
    }
    custom_json = <<EOT
{

Considering that I am running the latest version, it should work.

[martin-flaregames]

@polasekr try replacing

ignore_changes = [ "custom_cookbooks*" ]

with

lifecycle {
  ignore_changes = ["custom_cookbooks_source.0.ssh_key"]
}

A simple way to find the keys to use in the list is by issuing the following command in the command line:

terraform show | grep -iP "custom_cookbooks"

in my stack i get the following candidates

  custom_cookbooks_source.# = 1
  custom_cookbooks_source.0.password = 
  custom_cookbooks_source.0.revision = feature/ALB
  custom_cookbooks_source.0.ssh_key = 
  custom_cookbooks_source.0.type = git
  custom_cookbooks_source.0.url = [email protected]:project/project-chef.git
  custom_cookbooks_source.0.username = 
  use_custom_cookbooks = true

[martin-flaregames]

With terraform 0.6 and 0.7 there was a catch with this strategy. If you ignore the custom_cookbooks_source.0.ssh_key while creating the aws_opsworks_stack element then the SSH key was not set and left empty. That's the reason I'm not using this.

You can try with terraform 0.8 and see if it works. If not you can first create the aws_opsworks_stack resource without the ignore_changes clause, and then activate the ignore_changes when the stack object already exists.

[polasekr]

@martin-flaregames Thank you for your tips. I set up for now ignore_changes clause and it works.

I also set up ignore_changes for database parameter since it was always popping up when I ran a plan. It works as intended and changes are ignored. I just wanted to mention it, for more people to comment whether it is expected behavior, or it is a bug.

resource "aws_db_parameter_group" "parameter_group_to_allow_triggers" {
    lifecycle {
        ignore_changes = ["parameter"]
    }
    name = "allow-triggers"
    family = "mysql5.6"

    parameter {
      name = "log_bin_trust_function_creators"
      value = "true"
    }
}

When I run the plan without ignore changes

~ aws_db_parameter_group.parameter_group_to_allow_triggers
    parameter.950177639.apply_method: "immediate" => ""
    parameter.950177639.name:         "log_bin_trust_function_creators" => ""
    parameter.950177639.value:        "1" => ""
    parameter.973660974.apply_method: "" => "immediate"
    parameter.973660974.name:         "" => "log_bin_trust_function_creators"
    parameter.973660974.value:        "" => "true"


Plan: 0 to add, 1 to change, 0 to destroy.

I am little puzzled by the output and that's why I am not certain whether it is a problem is with AWS interaction, or it is a bug.

Anyway, thank you all for your prompt help.

[martin-flaregames]

edit: removed lifecycle { ignore_changes = ["parameter"] } from all examples, because that made no sense.

After a short observation in your plan output I can see that the cause seems to be that the "true" values is somehow transformed into a "1" after execution. Not sure if this is a bug.

Try one of the following alternatives and see which one works better for you, and please check that the parameters are set as expected in the AWS parameter group

"1" instead of "true"

resource "aws_db_parameter_group" "parameter_group_to_allow_triggers" {
    name = "allow-triggers"
    family = "mysql5.6"

    parameter {
      name = "log_bin_trust_function_creators"
      value = "1"
    }
}

1 instead of "true"

resource "aws_db_parameter_group" "parameter_group_to_allow_triggers" {
    name = "allow-triggers"
    family = "mysql5.6"

    parameter {
      name = "log_bin_trust_function_creators"
      value = 1
    }
}

true instead of "true"

resource "aws_db_parameter_group" "parameter_group_to_allow_triggers" {
    name = "allow-triggers"
    family = "mysql5.6"

    parameter {
      name = "log_bin_trust_function_creators"
      value = true
    }
}

[martin-flaregames]

Addendum: AFAIK terraform treats true and false respectively as 1 and 0. It would be helpful if someone could confirm this assumption of mine.

[polasekr]

@martin-flaregames Thank you for your tremendous help. I tested the option with true without double quotes and it works as expected. Here is the configuration used:

resource "aws_db_parameter_group" "parameter_group_to_allow_triggers" {

    name = "allow-triggers"
    family = "mysql5.6"

    parameter {
      name = "log_bin_trust_function_creators"
      value = true
    }
}

[manyam1113]

Hi I tried to changes like true to 1 no use
Its still reapplying the changes to parameter groups.

Any ideas

[martin-flaregames]

Nice necroposting. Anyway. I could try to help you if you post a code snippet of the Terraform resource configuration and the output for that resource from the plan generated by Terraform.

Just a blind hint: use true instead of "true", that seemed to help the OP.

[Mykolaichenko]

I've replaced it:

  parameter {
    name         = "skip_name_resolve"
    value        = "ON"
    apply_method = "pending-reboot"
  }

by

  parameter {
    name         = "skip_name_resolve"
    value        = "1"
    apply_method = "pending-reboot"
  }

And problem is gone.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.