Terraform: Security group with cidr_blocks keep showing up as modified when planning

Created on 24 Jul 2015 · 25Comments · Source: hashicorp/terraform

Hi there,

We noticed that SGs with cidr_blocks keep trying to be modified, per the plan.

In reality, nothing changes but it keeps showing up in the plan.

Let me know what kind of log I can provide to help.

~ aws_security_group.dev-chef-clients
    ingress.1612988420.cidr_blocks.#:     "0" => "1"
    ingress.1612988420.cidr_blocks.0:     "" => "10.42.0.0/16"
    ingress.1612988420.from_port:         "" => "10000"
    ingress.1612988420.protocol:          "" => "TCP"
    ingress.1612988420.security_groups.#: "0" => "0"
    ingress.1612988420.self:              "" => "0"
    ingress.1612988420.to_port:           "" => "10003"
    ingress.2107046791.cidr_blocks.#:     "1" => "0"
    ingress.2107046791.cidr_blocks.0:     "10.42.0.0/16" => ""
    ingress.2107046791.from_port:         "10000" => "0"
    ingress.2107046791.protocol:          "tcp" => ""
    ingress.2107046791.security_groups.#: "0" => "0"
    ingress.2107046791.self:              "0" => "0"
    ingress.2107046791.to_port:           "10003" => "0"

~ aws_security_group.gz-dev-activity
    ingress.#:                            "1" => "3"
    ingress.1020046450.cidr_blocks.#:     "0" => "1"
    ingress.1020046450.cidr_blocks.0:     "" => "10.42.32.231/16"
    ingress.1020046450.from_port:         "" => "0"
    ingress.1020046450.protocol:          "" => "-1"
    ingress.1020046450.security_groups.#: "0" => "0"
    ingress.1020046450.self:              "" => "0"
    ingress.1020046450.to_port:           "" => "0"
    ingress.2098700667.cidr_blocks.#:     "0" => "1"
    ingress.2098700667.cidr_blocks.0:     "" => "10.52.0.0/16"
    ingress.2098700667.from_port:         "" => "0"
    ingress.2098700667.protocol:          "" => "-1"
    ingress.2098700667.security_groups.#: "0" => "0"
    ingress.2098700667.self:              "" => "0"
    ingress.2098700667.to_port:           "" => "0"
    ingress.4113246637.cidr_blocks.#:     "2" => "0"
    ingress.4113246637.cidr_blocks.0:     "10.42.0.0/16" => ""
    ingress.4113246637.cidr_blocks.1:     "10.52.0.0/16" => ""
    ingress.4113246637.from_port:         "0" => "0"
    ingress.4113246637.protocol:          "-1" => ""
    ingress.4113246637.security_groups.#: "0" => "0"
    ingress.4113246637.self:              "1" => "0"
    ingress.4113246637.to_port:           "0" => "0"
    ingress.753360330.cidr_blocks.#:      "0" => "0"
    ingress.753360330.from_port:          "" => "0"
    ingress.753360330.protocol:           "" => "-1"
    ingress.753360330.security_groups.#:  "0" => "0"
    ingress.753360330.self:               "" => "1"
    ingress.753360330.to_port:            "" => "0"

~ aws_security_group.gz-dev-activity-elb
    ingress.#:                            "3" => "6"
    ingress.2131821948.cidr_blocks.#:     "2" => "0"
    ingress.2131821948.cidr_blocks.0:     "10.42.0.0/16" => ""
    ingress.2131821948.cidr_blocks.1:     "10.52.0.0/16" => ""
    ingress.2131821948.from_port:         "50041" => "0"
    ingress.2131821948.protocol:          "tcp" => ""
    ingress.2131821948.security_groups.#: "0" => "0"
    ingress.2131821948.self:              "0" => "0"
    ingress.2131821948.to_port:           "50041" => "0"
    ingress.2202729228.cidr_blocks.#:     "0" => "1"
    ingress.2202729228.cidr_blocks.0:     "" => "10.52.0.0/16"
    ingress.2202729228.from_port:         "" => "50051"
    ingress.2202729228.protocol:          "" => "TCP"
    ingress.2202729228.security_groups.#: "0" => "0"
    ingress.2202729228.self:              "" => "0"
    ingress.2202729228.to_port:           "" => "50051"
    ingress.2425398671.cidr_blocks.#:     "2" => "0"
    ingress.2425398671.cidr_blocks.0:     "10.52.0.0/16" => ""
    ingress.2425398671.cidr_blocks.1:     "10.42.0.0/16" => ""
    ingress.2425398671.from_port:         "50011" => "0"
    ingress.2425398671.protocol:          "tcp" => ""
    ingress.2425398671.security_groups.#: "0" => "0"
    ingress.2425398671.self:              "0" => "0"
    ingress.2425398671.to_port:           "50011" => "0"
    ingress.2450400921.cidr_blocks.#:     "2" => "0"
    ingress.2450400921.cidr_blocks.0:     "10.42.0.0/16" => ""
    ingress.2450400921.cidr_blocks.1:     "10.52.0.0/16" => ""
    ingress.2450400921.from_port:         "50051" => "0"
    ingress.2450400921.protocol:          "tcp" => ""
    ingress.2450400921.security_groups.#: "0" => "0"
    ingress.2450400921.self:              "0" => "0"
    ingress.2450400921.to_port:           "50051" => "0"
    ingress.2489680930.cidr_blocks.#:     "0" => "1"
    ingress.2489680930.cidr_blocks.0:     "" => "10.42.32.231/16"
    ingress.2489680930.from_port:         "" => "50041"
    ingress.2489680930.protocol:          "" => "TCP"
    ingress.2489680930.security_groups.#: "0" => "0"
    ingress.2489680930.self:              "" => "0"
    ingress.2489680930.to_port:           "" => "50041"
    ingress.2509068337.cidr_blocks.#:     "0" => "1"
    ingress.2509068337.cidr_blocks.0:     "" => "10.52.0.0/16"
    ingress.2509068337.from_port:         "" => "50041"
    ingress.2509068337.protocol:          "" => "TCP"
    ingress.2509068337.security_groups.#: "0" => "0"
    ingress.2509068337.self:              "" => "0"
    ingress.2509068337.to_port:           "" => "50041"
    ingress.3629414392.cidr_blocks.#:     "0" => "1"
    ingress.3629414392.cidr_blocks.0:     "" => "10.52.0.0/16"
    ingress.3629414392.from_port:         "" => "50011"
    ingress.3629414392.protocol:          "" => "TCP"
    ingress.3629414392.security_groups.#: "0" => "0"
    ingress.3629414392.self:              "" => "0"
    ingress.3629414392.to_port:           "" => "50011"
    ingress.3876092861.cidr_blocks.#:     "0" => "1"
    ingress.3876092861.cidr_blocks.0:     "" => "10.42.32.231/16"
    ingress.3876092861.from_port:         "" => "50011"
    ingress.3876092861.protocol:          "" => "TCP"
    ingress.3876092861.security_groups.#: "0" => "0"
    ingress.3876092861.self:              "" => "0"
    ingress.3876092861.to_port:           "" => "50011"
    ingress.956058318.cidr_blocks.#:      "0" => "1"
    ingress.956058318.cidr_blocks.0:      "" => "10.42.32.231/16"
    ingress.956058318.from_port:          "" => "50051"
    ingress.956058318.protocol:           "" => "TCP"
    ingress.956058318.security_groups.#:  "0" => "0"
    ingress.956058318.self:               "" => "0"
    ingress.956058318.to_port:            "" => "50051"

bug provideaws

Source

scalp42

Most helpful comment

Thanks for all the detailed reports, everyone!

In @scalp42's latest example it looks like AWS is normalizing "TCP" to "tcp" and causing a diff. Fixing that should look similar to fixing #3120... just use StateFunc to make sure it always gets saved in the state as lowercase.

I think this is a bug in addition to the issue that @catsby described above. The normalization of the protocol should be easy to fix relative to the issue of the API collapsing multiple separate rules into one.

In both cases, the workaround is to write your Terraform config in the normalized form. Specifically:

If you have multiple rules with the same port range and different CIDR blocks, collapse them into a single rule with all of the CIDR blocks listed.
If you're writing the protocol in uppercase, write it in lowercase instead.

So for @scalp42's example:

resource "aws_security_group" "gz-prod-worker-elb" {
    name = "gz-prod-worker-elb"
    description = "gz-prod-worker-elb"
    vpc_id = "${aws_vpc.prod.id}"
    ingress {
        from_port = 50081
        to_port = 50081
        protocol = "tcp"
        cidr_blocks = [
            "${aws_eip.gz-infra-jumphost.private_ip}/32",
            "${var.prod_vpc_cidr_block}",
        ]
    }
    ingress {
        from_port = 50091
        to_port = 50091
        protocol = "tcp"
        cidr_blocks = [
                "${aws_eip.gz-infra-jumphost.private_ip}/32",
                "${var.prod_vpc_cidr_block}",
        ]
    }
    egress {
        from_port = 0
        to_port = 0
        protocol = "-1"
        cidr_blocks = ["0.0.0.0/0"]
    }
    tags {
        Name = "gz-prod-worker-elb"
        Description = "gz-prod-worker-elb"
        vpc = "prod"
        terraform = "true"
    }
}

The self attribute can also be combined with cidr_blocks in a single rule, as we see in @georgebashi's example:

resource "aws_security_group" "test_sg" {
    name = "test_sg"
    vpc_id = "vpc-XXXXXXXX"

    ingress {
        from_port = 9200
        to_port = 9400
        protocol = "tcp"
        cidr_blocks = ["10.0.0.0/8"]
        self = true
    }
}

By writing the configuration in the normalized form, the state retrieved from the API will match the configuration and so the diffs should go away.

apparentlymart on 2 Nov 2015

👍3

All 25 comments

Update:

Terraform does appear to be doing something, but we checked and nothing changed on AWS side:

aws_security_group.gz-dev-worker-elb: Modifying...
  ingress.#:                            "2" => "4"
  ingress.1852403216.cidr_blocks.#:     "0" => "1"
  ingress.1852403216.cidr_blocks.0:     "" => "10.52.0.0/16"
  ingress.1852403216.from_port:         "" => "50091"
  ingress.1852403216.protocol:          "" => "TCP"
  ingress.1852403216.security_groups.#: "0" => "0"
  ingress.1852403216.self:              "" => "0"
  ingress.1852403216.to_port:           "" => "50091"
  ingress.2024721709.cidr_blocks.#:     "0" => "1"
  ingress.2024721709.cidr_blocks.0:     "" => "10.52.0.0/16"
  ingress.2024721709.from_port:         "" => "50081"
  ingress.2024721709.protocol:          "" => "TCP"
  ingress.2024721709.security_groups.#: "0" => "0"
  ingress.2024721709.self:              "" => "0"
  ingress.2024721709.to_port:           "" => "50081"
  ingress.2024964166.cidr_blocks.#:     "2" => "0"
  ingress.2024964166.cidr_blocks.0:     "10.42.0.0/16" => ""
  ingress.2024964166.cidr_blocks.1:     "10.52.0.0/16" => ""
  ingress.2024964166.from_port:         "50081" => "0"
  ingress.2024964166.protocol:          "tcp" => ""
  ingress.2024964166.security_groups.#: "0" => "0"
  ingress.2024964166.self:              "0" => "0"
  ingress.2024964166.to_port:           "50081" => "0"
  ingress.2207325978.cidr_blocks.#:     "0" => "1"
  ingress.2207325978.cidr_blocks.0:     "" => "10.42.32.231/16"
  ingress.2207325978.from_port:         "" => "50091"
  ingress.2207325978.protocol:          "" => "TCP"
  ingress.2207325978.security_groups.#: "0" => "0"
  ingress.2207325978.self:              "" => "0"
  ingress.2207325978.to_port:           "" => "50091"
  ingress.2511184803.cidr_blocks.#:     "2" => "0"
  ingress.2511184803.cidr_blocks.0:     "10.52.0.0/16" => ""
  ingress.2511184803.cidr_blocks.1:     "10.42.0.0/16" => ""
  ingress.2511184803.from_port:         "50091" => "0"
  ingress.2511184803.protocol:          "tcp" => ""
  ingress.2511184803.security_groups.#: "0" => "0"
  ingress.2511184803.self:              "0" => "0"
  ingress.2511184803.to_port:           "50091" => "0"
  ingress.789107190.cidr_blocks.#:      "0" => "1"
  ingress.789107190.cidr_blocks.0:      "" => "10.42.32.231/16"
  ingress.789107190.from_port:          "" => "50081"
  ingress.789107190.protocol:           "" => "TCP"
  ingress.789107190.security_groups.#:  "0" => "0"
  ingress.789107190.self:               "" => "0"
  ingress.789107190.to_port:            "" => "50081"
aws_security_group.gz-dev-worker-elb: Modifications complete

scalp42 on 24 Jul 2015

Hey @scalp42 – do you have a config that shows just these changing blocks? If you could share that, minus any secrets, that would help.

Some things I noticed though:

10.42.32.231/16 isn't valid for /16... AWS is returning that as 10.42.0.0/16
if you send 10.42.32.231/16, check the console, you'll see 10.42.0.0/16

thats why you see this:

    ingress.1020046450.cidr_blocks.0:     "" => "10.42.32.231/16"
    [...]
    ingress.4113246637.cidr_blocks.0:     "10.42.0.0/16" => ""=

catsby on 24 Jul 2015

Hi @catsby, thanks for getting back to me! Indeed, we fixed the CIDR for the bastion host, but we're still hitting the issue.

Assuming aws_security_group.gz-prod-worker-elb, I went and deleted all the SGs rule in AWS.

This is the config for aws_security_group.gz-prod-worker-elb:

resource "aws_security_group" "gz-prod-worker-elb" {
    name = "gz-prod-worker-elb"
    description = "gz-prod-worker-elb"
    vpc_id = "${aws_vpc.prod.id}"
    ingress {
        from_port = 50081
        to_port = 50081
        protocol = "TCP"
        cidr_blocks = ["${aws_eip.gz-infra-jumphost.private_ip}/32"]
    }
    ingress {
        from_port = 50091
        to_port = 50091
        protocol = "TCP"
        cidr_blocks = ["${aws_eip.gz-infra-jumphost.private_ip}/32"]
    }
    ingress {
        from_port = 50081
        to_port = 50081
        protocol = "TCP"
        cidr_blocks = ["${var.prod_vpc_cidr_block}"]
    }
    ingress {
        from_port = 50091
        to_port = 50091
        protocol = "TCP"
        cidr_blocks = ["${var.prod_vpc_cidr_block}"]
    }
    egress {
        from_port = 0
        to_port = 0
        protocol = "-1"
        cidr_blocks = ["0.0.0.0/0"]
    }
    tags {
        Name = "gz-prod-worker-elb"
        Description = "gz-prod-worker-elb"
        vpc = "prod"
        terraform = "true"
    }
}

variable "prod_vpc_cidr_block" {
  description = "CIDR block for the prod VPC."
  default = "10.92.0.0/16"
}

Here is the summary for plan:

~ aws_security_group.gz-prod-worker-elb
    ingress.#:                            "0" => "4"
    ingress.1222688924.cidr_blocks.#:     "0" => "1"
    ingress.1222688924.cidr_blocks.0:     "" => "10.42.32.231/32"
    ingress.1222688924.from_port:         "" => "50081"
    ingress.1222688924.protocol:          "" => "TCP"
    ingress.1222688924.security_groups.#: "0" => "0"
    ingress.1222688924.self:              "" => "0"
    ingress.1222688924.to_port:           "" => "50081"
    ingress.1653747107.cidr_blocks.#:     "0" => "1"
    ingress.1653747107.cidr_blocks.0:     "" => "10.92.0.0/16"
    ingress.1653747107.from_port:         "" => "50081"
    ingress.1653747107.protocol:          "" => "TCP"
    ingress.1653747107.security_groups.#: "0" => "0"
    ingress.1653747107.self:              "" => "0"
    ingress.1653747107.to_port:           "" => "50081"
    ingress.1951764126.cidr_blocks.#:     "0" => "1"
    ingress.1951764126.cidr_blocks.0:     "" => "10.92.0.0/16"
    ingress.1951764126.from_port:         "" => "50091"
    ingress.1951764126.protocol:          "" => "TCP"
    ingress.1951764126.security_groups.#: "0" => "0"
    ingress.1951764126.self:              "" => "0"
    ingress.1951764126.to_port:           "" => "50091"
    ingress.3833138800.cidr_blocks.#:     "0" => "1"
    ingress.3833138800.cidr_blocks.0:     "" => "10.42.32.231/32"
    ingress.3833138800.from_port:         "" => "50091"
    ingress.3833138800.protocol:          "" => "TCP"
    ingress.3833138800.security_groups.#: "0" => "0"
    ingress.3833138800.self:              "" => "0"
    ingress.3833138800.to_port:           "" => "50091"

Here is the apply:

aws_security_group.gz-prod-worker-elb: Modifying...
  ingress.#:                            "0" => "4"
  ingress.1222688924.cidr_blocks.#:     "0" => "1"
  ingress.1222688924.cidr_blocks.0:     "" => "10.42.32.231/32"
  ingress.1222688924.from_port:         "" => "50081"
  ingress.1222688924.protocol:          "" => "TCP"
  ingress.1222688924.security_groups.#: "0" => "0"
  ingress.1222688924.self:              "" => "0"
  ingress.1222688924.to_port:           "" => "50081"
  ingress.1653747107.cidr_blocks.#:     "0" => "1"
  ingress.1653747107.cidr_blocks.0:     "" => "10.92.0.0/16"
  ingress.1653747107.from_port:         "" => "50081"
  ingress.1653747107.protocol:          "" => "TCP"
  ingress.1653747107.security_groups.#: "0" => "0"
  ingress.1653747107.self:              "" => "0"
  ingress.1653747107.to_port:           "" => "50081"
  ingress.1951764126.cidr_blocks.#:     "0" => "1"
  ingress.1951764126.cidr_blocks.0:     "" => "10.92.0.0/16"
  ingress.1951764126.from_port:         "" => "50091"
  ingress.1951764126.protocol:          "" => "TCP"
  ingress.1951764126.security_groups.#: "0" => "0"
  ingress.1951764126.self:              "" => "0"
  ingress.1951764126.to_port:           "" => "50091"
  ingress.3833138800.cidr_blocks.#:     "0" => "1"
  ingress.3833138800.cidr_blocks.0:     "" => "10.42.32.231/32"
  ingress.3833138800.from_port:         "" => "50091"
  ingress.3833138800.protocol:          "" => "TCP"
  ingress.3833138800.security_groups.#: "0" => "0"
  ingress.3833138800.self:              "" => "0"
  ingress.3833138800.to_port:           "" => "50091"
aws_security_group.gz-prod-worker-elb: Modifications complete

At this point, we can see the correct rules created in AWS, so far so good:

If we try to plan again:

aws_security_group.gz-prod-worker-elb: Refreshing state... (ID: sg-b38249d7)
~ aws_security_group.gz-prod-worker-elb
    ingress.#:                            "2" => "4"
    ingress.1222688924.cidr_blocks.#:     "0" => "1"
    ingress.1222688924.cidr_blocks.0:     "" => "10.42.32.231/32"
    ingress.1222688924.from_port:         "" => "50081"
    ingress.1222688924.protocol:          "" => "TCP"
    ingress.1222688924.security_groups.#: "0" => "0"
    ingress.1222688924.self:              "" => "0"
    ingress.1222688924.to_port:           "" => "50081"
    ingress.1653747107.cidr_blocks.#:     "0" => "1"
    ingress.1653747107.cidr_blocks.0:     "" => "10.92.0.0/16"
    ingress.1653747107.from_port:         "" => "50081"
    ingress.1653747107.protocol:          "" => "TCP"
    ingress.1653747107.security_groups.#: "0" => "0"
    ingress.1653747107.self:              "" => "0"
    ingress.1653747107.to_port:           "" => "50081"
    ingress.1951764126.cidr_blocks.#:     "0" => "1"
    ingress.1951764126.cidr_blocks.0:     "" => "10.92.0.0/16"
    ingress.1951764126.from_port:         "" => "50091"
    ingress.1951764126.protocol:          "" => "TCP"
    ingress.1951764126.security_groups.#: "0" => "0"
    ingress.1951764126.self:              "" => "0"
    ingress.1951764126.to_port:           "" => "50091"
    ingress.3019390024.cidr_blocks.#:     "2" => "0"
    ingress.3019390024.cidr_blocks.0:     "10.92.0.0/16" => ""
    ingress.3019390024.cidr_blocks.1:     "10.42.32.231/32" => ""
    ingress.3019390024.from_port:         "50091" => "0"
    ingress.3019390024.protocol:          "tcp" => ""
    ingress.3019390024.security_groups.#: "0" => "0"
    ingress.3019390024.self:              "0" => "0"
    ingress.3019390024.to_port:           "50091" => "0"
    ingress.3833138800.cidr_blocks.#:     "0" => "1"
    ingress.3833138800.cidr_blocks.0:     "" => "10.42.32.231/32"
    ingress.3833138800.from_port:         "" => "50091"
    ingress.3833138800.protocol:          "" => "TCP"
    ingress.3833138800.security_groups.#: "0" => "0"
    ingress.3833138800.self:              "" => "0"
    ingress.3833138800.to_port:           "" => "50091"
    ingress.795890810.cidr_blocks.#:      "2" => "0"
    ingress.795890810.cidr_blocks.0:      "10.42.32.231/32" => ""
    ingress.795890810.cidr_blocks.1:      "10.92.0.0/16" => ""
    ingress.795890810.from_port:          "50081" => "0"
    ingress.795890810.protocol:           "tcp" => ""
    ingress.795890810.security_groups.#:  "0" => "0"
    ingress.795890810.self:               "0" => "0"
    ingress.795890810.to_port:            "50081" => "0"

As you can see, there's a bug here (we have 60 changes like this one).

We noticed that the ordering in AWS changed:

Let me know if you need anything else, greatly appreciated.

scalp42 on 24 Jul 2015

We experience this same problem -- plan and apply have changes, even when _nothing_ has changed. These phantom changes seem to be security group & NACL entries which are changing position, e.g.:

ingress.#: "2" => "4"

Applying the plan doesn't seem to make any changes to the AWS resources, but it's still nerve-wracking to see so many changes.

simpsora on 7 Aug 2015

I'm seeing the same thing. Here's a test case:

resource "aws_security_group" "test_sg" {
    name = "test_sg"
    vpc_id = "vpc-XXXXXXXX"

    ingress {
        from_port = 9200
        to_port = 9400
        protocol = "tcp"
        self = true
    }

    ingress {
        from_port = 9200
        to_port = 9400
        protocol = "tcp"
        cidr_blocks = ["10.0.0.0/8"]
    }
}

You can terraform apply this over and over and it will still show modifications like this:

aws_security_group.test_sg: Refreshing state... (ID: sg-XXXXXXXX)
aws_security_group.test_sg: Modifying...
  ingress.#:                            "1" => "2"
  ingress.1907357802.cidr_blocks.#:     "0" => "1"
  ingress.1907357802.cidr_blocks.0:     "" => "10.0.0.0/8"
  ingress.1907357802.from_port:         "" => "9200"
  ingress.1907357802.protocol:          "" => "tcp"
  ingress.1907357802.security_groups.#: "0" => "0"
  ingress.1907357802.self:              "" => "0"
  ingress.1907357802.to_port:           "" => "9400"
  ingress.2549385326.cidr_blocks.#:     "1" => "0"
  ingress.2549385326.cidr_blocks.0:     "10.0.0.0/8" => ""
  ingress.2549385326.from_port:         "9200" => "0"
  ingress.2549385326.protocol:          "tcp" => ""
  ingress.2549385326.security_groups.#: "0" => "0"
  ingress.2549385326.self:              "1" => "0"
  ingress.2549385326.to_port:           "9400" => "0"
  ingress.3645327794.cidr_blocks.#:     "0" => "0"
  ingress.3645327794.from_port:         "" => "9200"
  ingress.3645327794.protocol:          "" => "tcp"
  ingress.3645327794.security_groups.#: "0" => "0"
  ingress.3645327794.self:              "" => "1"
  ingress.3645327794.to_port:           "" => "9400"
aws_security_group.test_sg: Modifications complete

georgebashi on 7 Aug 2015

Same here.

code:

resource "aws_security_group" "test-web" {
    name = "test.web"
    description = "test web"
    vpc_id = "${aws_vpc.test-prod.id}"

    # HTTP access from self
    ingress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        self = true
    }
    # HTTPS access from self
    ingress {
        from_port = 443
        to_port = 443
        protocol = "tcp"
        self = true
    }
    # HTTP access from anywhere
    ingress {
        from_port = 80
        to_port = 80
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
    # HTTPS access from anywhere
    ingress {
        from_port = 443
        to_port = 443
        protocol = "tcp"
        cidr_blocks = ["0.0.0.0/0"]
    }
}

terraform plan output:

~ aws_security_group.test-web
    ingress.#:                            "2" => "4"
    ingress.1844393598.cidr_blocks.#:     "0" => "0"
    ingress.1844393598.from_port:         "" => "80"
    ingress.1844393598.protocol:          "" => "tcp"
    ingress.1844393598.security_groups.#: "0" => "0"
    ingress.1844393598.self:              "" => "1"
    ingress.1844393598.to_port:           "" => "80"
    ingress.2214680975.cidr_blocks.#:     "0" => "1"
    ingress.2214680975.cidr_blocks.0:     "" => "0.0.0.0/0"
    ingress.2214680975.from_port:         "" => "80"
    ingress.2214680975.protocol:          "" => "tcp"
    ingress.2214680975.security_groups.#: "0" => "0"
    ingress.2214680975.self:              "" => "0"
    ingress.2214680975.to_port:           "" => "80"
    ingress.2617001939.cidr_blocks.#:     "0" => "1"
    ingress.2617001939.cidr_blocks.0:     "" => "0.0.0.0/0"
    ingress.2617001939.from_port:         "" => "443"
    ingress.2617001939.protocol:          "" => "tcp"
    ingress.2617001939.security_groups.#: "0" => "0"
    ingress.2617001939.self:              "" => "0"
    ingress.2617001939.to_port:           "" => "443"
    ingress.3167104115.cidr_blocks.#:     "0" => "0"
    ingress.3167104115.from_port:         "" => "443"
    ingress.3167104115.protocol:          "" => "tcp"
    ingress.3167104115.security_groups.#: "0" => "0"
    ingress.3167104115.self:              "" => "1"
    ingress.3167104115.to_port:           "" => "443"
    ingress.3666510777.cidr_blocks.#:     "1" => "0"
    ingress.3666510777.cidr_blocks.0:     "0.0.0.0/0" => ""
    ingress.3666510777.from_port:         "443" => "0"
    ingress.3666510777.protocol:          "tcp" => ""
    ingress.3666510777.security_groups.#: "0" => "0"
    ingress.3666510777.self:              "1" => "0"
    ingress.3666510777.to_port:           "443" => "0"
    ingress.4246188394.cidr_blocks.#:     "1" => "0"
    ingress.4246188394.cidr_blocks.0:     "0.0.0.0/0" => ""
    ingress.4246188394.from_port:         "80" => "0"
    ingress.4246188394.protocol:          "tcp" => ""
    ingress.4246188394.security_groups.#: "0" => "0"
    ingress.4246188394.self:              "1" => "0"
    ingress.4246188394.to_port:           "80" => "0"

Plan: 0 to add, 1 to change, 0 to destroy.

SG test.web is already good regarding the code:
test-web_screenshot

EDIT: I'm using last version as I'm writing this (0.6.3).

jeekajoo on 12 Aug 2015

Hey all – thank you everyone for contributing your examples that demonstrate this issue. I sincerely apologize for not getting to this sooner, but I'm on it now.

The gist of it, what I can see, is how AWS handles Security Group Rules with regards to ports. AWS will group all rules for a given port range into one "rule", even though the console shows differently. The console shows all the IP ranges, but they're grouped under one rule by port..

http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_IpPermission.html

I'm working on this and similar issue to try and narrow down a fix.
Sorry again for the delay

catsby on 14 Aug 2015

@catsby thanks for looking into it, do you know if it'll make it to the next release by any chance?

scalp42 on 27 Aug 2015

any update on this? it is really causing pain for us.

shayangz on 10 Oct 2015

For anyone having the same issue, our workaround currently is to make sure of the ignore_changes flag added in Terraform v0.6.4:

    lifecycle {
      create_before_destroy = false
      ignore_changes = ["ingress"]
    }

We added it to every single SG having cidr_blocks defined.

Where before we had close to 100 changes, now it's "working" as workaround:

No changes. Infrastructure is up-to-date. This means that Terraform
could not detect any differences between your configuration and
the real physical resources that exist. As a result, Terraform
doesn't need to do anything.

cc @phinze for updates (granted its just a workaround)

scalp42 on 20 Oct 2015

I've been having this same problem in 0.6.7-dev. I tried the workaround and got this error instead:

* aws_security_group.router: diffs didn't match during apply. This is a bug with Terraform and should be reported.

thrashr888 on 2 Nov 2015

Thanks for all the detailed reports, everyone!

In both cases, the workaround is to write your Terraform config in the normalized form. Specifically:

If you have multiple rules with the same port range and different CIDR blocks, collapse them into a single rule with all of the CIDR blocks listed.
If you're writing the protocol in uppercase, write it in lowercase instead.

So for @scalp42's example:

resource "aws_security_group" "gz-prod-worker-elb" {
    name = "gz-prod-worker-elb"
    description = "gz-prod-worker-elb"
    vpc_id = "${aws_vpc.prod.id}"
    ingress {
        from_port = 50081
        to_port = 50081
        protocol = "tcp"
        cidr_blocks = [
            "${aws_eip.gz-infra-jumphost.private_ip}/32",
            "${var.prod_vpc_cidr_block}",
        ]
    }
    ingress {
        from_port = 50091
        to_port = 50091
        protocol = "tcp"
        cidr_blocks = [
                "${aws_eip.gz-infra-jumphost.private_ip}/32",
                "${var.prod_vpc_cidr_block}",
        ]
    }
    egress {
        from_port = 0
        to_port = 0
        protocol = "-1"
        cidr_blocks = ["0.0.0.0/0"]
    }
    tags {
        Name = "gz-prod-worker-elb"
        Description = "gz-prod-worker-elb"
        vpc = "prod"
        terraform = "true"
    }
}

The self attribute can also be combined with cidr_blocks in a single rule, as we see in @georgebashi's example:

resource "aws_security_group" "test_sg" {
    name = "test_sg"
    vpc_id = "vpc-XXXXXXXX"

    ingress {
        from_port = 9200
        to_port = 9400
        protocol = "tcp"
        cidr_blocks = ["10.0.0.0/8"]
        self = true
    }
}

By writing the configuration in the normalized form, the state retrieved from the API will match the configuration and so the diffs should go away.

apparentlymart on 2 Nov 2015

👍3

The lowercase workaround seems to have worked for me. Thanks @apparentlymart!

thrashr888 on 2 Nov 2015

+1 for @apparentlymart's lower case "tcp" work around. Thanks!

brikis98 on 3 Nov 2015

Hey all – I'm going to close this for now.
ingress rules should be normalized as per @apparentlymart mentioned, so that rules of the same port range and protocol are together.

Let me know if there's anything else here!

catsby on 5 Nov 2015

Normalizing ingress rules worked for me. Thanks @apparentlymart!

simpsora on 6 Nov 2015

@catsby
I'm having the same issue but I'm not sure how to put my two rules into one:

ingress {
    from_port       = 22
    to_port         = 22
    protocol        = "tcp"
    cidr_blocks     = ["${split(",", var.external_ips)}"]
}   

ingress {
    from_port       = 22
    to_port         = 22
    protocol        = "tcp"
    cidr_blocks     = [ "X.X.Y.Y/32","X.X.Z.Z/32" ]
}

I cannot put the two additional IPs into external_ips since this rule is a special case.

mseiwald on 8 Feb 2016

@mseiwald, have you tried this:

ingress {
    from_port       = 22
    to_port         = 22
    protocol        = "tcp"
    cidr_blocks     = [
       "${split(",", var.external_ips)}",
       "X.X.Y.Y/32",
       "X.X.Z.Z/32"
    ]
}

scalp42 on 8 Feb 2016

@mseiwald the concat() function also could come in handy here:

cidr_blocks = ["${concat(split(",", var.external_ips), split(",", "X.X.Y.Y/32,X.X.Z.Z/32"))}]

phinze on 8 Feb 2016

Thanks, guys.

mseiwald on 9 Feb 2016

this thread helped TONS, thanks for the help.

luis-echegaray on 10 Feb 2016

I still get this error. see below, I tried all the above. I am adding aws_security_group_rule as well as shown below.
I have my security group xyz_sg_abc at the bottom.

33m~ aws_security_group.xyz_sg_abc
[0m egress.#: "14" => "4"
egress.1238134219.cidr_blocks.#: "0" => "0"
egress.1238134219.from_port: "3389" => "3389"
egress.1238134219.prefix_list_ids.#: "0" => "0"
egress.1238134219.protocol: "udp" => "udp"
egress.1238134219.security_groups.#: "1" => "1"
egress.1238134219.security_groups.3352223863: "sg-940721ee" => "sg-940721ee"
egress.1238134219.self: "false" => "false"
egress.1238134219.to_port: "3389" => "3389"
egress.136061899.cidr_blocks.#: "0" => "0"
egress.136061899.from_port: "443" => "443"
egress.136061899.prefix_list_ids.#: "0" => "0"
egress.136061899.protocol: "tcp" => "tcp"
egress.136061899.security_groups.#: "1" => "1"
egress.136061899.security_groups.2876704322: "sg-b80721c2" => "sg-b80721c2"
egress.136061899.self: "false" => "false"
egress.136061899.to_port: "443" => "443"
egress.1467575756.cidr_blocks.#: "0" => "0"
egress.1467575756.from_port: "61620" => "0"
egress.1467575756.prefix_list_ids.#: "0" => "0"
egress.1467575756.protocol: "tcp" => ""
egress.1467575756.security_groups.#: "1" => "0"
egress.1467575756.security_groups.2169191079: "sg-c70127bd" => ""
egress.1467575756.self: "false" => "false"
egress.1467575756.to_port: "61620" => "0"
egress.149989598.cidr_blocks.#: "0" => "0"
egress.149989598.from_port: "80" => "0"
egress.149989598.prefix_list_ids.#: "0" => "0"
egress.149989598.protocol: "tcp" => ""
egress.149989598.security_groups.#: "2" => "0"
egress.149989598.security_groups.2980972767: "sg-c20127b8" => ""
egress.149989598.security_groups.4111187599: "sg-c80127b2" => ""
egress.149989598.self: "false" => "false"
egress.149989598.to_port: "80" => "0"
egress.187082154.cidr_blocks.#: "0" => "0"
egress.187082154.from_port: "9042" => "0"
egress.187082154.prefix_list_ids.#: "0" => "0"
egress.187082154.protocol: "tcp" => ""
egress.187082154.security_groups.#: "1" => "0"
egress.187082154.security_groups.2169191079: "sg-c70127bd" => ""
egress.187082154.self: "false" => "false"
egress.187082154.to_port: "9042" => "0"
egress.2195547251.cidr_blocks.#: "0" => "0"
egress.2195547251.from_port: "22" => "0"
egress.2195547251.prefix_list_ids.#: "0" => "0"
egress.2195547251.protocol: "tcp" => ""
egress.2195547251.security_groups.#: "1" => "0"
egress.2195547251.security_groups.2169191079: "sg-c70127bd" => ""
egress.2195547251.self: "false" => "false"
egress.2195547251.to_port: "22" => "0"
egress.3400316176.cidr_blocks.#: "0" => "0"
egress.3400316176.from_port: "7000" => "0"
egress.3400316176.prefix_list_ids.#: "0" => "0"
egress.3400316176.protocol: "tcp" => ""
egress.3400316176.security_groups.#: "1" => "0"
egress.3400316176.security_groups.2169191079: "sg-c70127bd" => ""
egress.3400316176.self: "false" => "false"
egress.3400316176.to_port: "7000" => "0"
egress.357529062.cidr_blocks.#: "0" => "0"
egress.357529062.from_port: "9160" => "0"
egress.357529062.prefix_list_ids.#: "0" => "0"
egress.357529062.protocol: "tcp" => ""
egress.357529062.security_groups.#: "1" => "0"
egress.357529062.security_groups.2169191079: "sg-c70127bd" => ""
egress.357529062.self: "false" => "false"
egress.357529062.to_port: "9160" => "0"
egress.3827110669.cidr_blocks.#: "0" => "0"
egress.3827110669.from_port: "80" => "80"
egress.3827110669.prefix_list_ids.#: "0" => "0"
egress.3827110669.protocol: "tcp" => "tcp"
egress.3827110669.security_groups.#: "1" => "1"
egress.3827110669.security_groups.2876704322: "sg-b80721c2" => "sg-b80721c2"
egress.3827110669.self: "false" => "false"
egress.3827110669.to_port: "80" => "80"
egress.3929861091.cidr_blocks.#: "0" => "0"
egress.3929861091.from_port: "61621" => "0"
egress.3929861091.prefix_list_ids.#: "0" => "0"
egress.3929861091.protocol: "tcp" => ""
egress.3929861091.security_groups.#: "1" => "0"
egress.3929861091.security_groups.2169191079: "sg-c70127bd" => ""
egress.3929861091.self: "false" => "false"
egress.3929861091.to_port: "61621" => "0"
egress.4211759511.cidr_blocks.#: "0" => "0"
egress.4211759511.from_port: "1433" => "0"
egress.4211759511.prefix_list_ids.#: "0" => "0"
egress.4211759511.protocol: "tcp" => ""
egress.4211759511.security_groups.#: "1" => "0"
egress.4211759511.security_groups.3551840219: "sg-cc0127b6" => ""
egress.4211759511.self: "false" => "false"
egress.4211759511.to_port: "1433" => "0"
egress.680397075.cidr_blocks.#: "0" => "0"
egress.680397075.from_port: "7001" => "0"
egress.680397075.prefix_list_ids.#: "0" => "0"
egress.680397075.protocol: "tcp" => ""
egress.680397075.security_groups.#: "1" => "0"
egress.680397075.security_groups.2169191079: "sg-c70127bd" => ""
egress.680397075.self: "false" => "false"
egress.680397075.to_port: "7001" => "0"
egress.808665337.cidr_blocks.#: "0" => "0"
egress.808665337.from_port: "7199" => "0"
egress.808665337.prefix_list_ids.#: "0" => "0"
egress.808665337.protocol: "tcp" => ""
egress.808665337.security_groups.#: "1" => "0"
egress.808665337.security_groups.2169191079: "sg-c70127bd" => ""
egress.808665337.self: "false" => "false"
egress.808665337.to_port: "7199" => "0"
egress.922980179.cidr_blocks.#: "0" => "0"
egress.922980179.from_port: "3389" => "3389"
egress.922980179.prefix_list_ids.#: "0" => "0"
egress.922980179.protocol: "tcp" => "tcp"
egress.922980179.security_groups.#: "1" => "1"
egress.922980179.security_groups.3352223863: "sg-940721ee" => "sg-940721ee"
egress.922980179.self: "false" => "false"
egress.922980179.to_port: "3389" => "3389"
ingress.#: "14" => "4"
ingress.1238134219.cidr_blocks.#: "0" => "0"
ingress.1238134219.from_port: "3389" => "3389"
ingress.1238134219.protocol: "udp" => "udp"
ingress.1238134219.security_groups.#: "1" => "1"
ingress.1238134219.security_groups.3352223863: "sg-940721ee" => "sg-940721ee"
ingress.1238134219.self: "false" => "false"
ingress.1238134219.to_port: "3389" => "3389"
ingress.136061899.cidr_blocks.#: "0" => "0"
ingress.136061899.from_port: "443" => "443"
ingress.136061899.protocol: "tcp" => "tcp"
ingress.136061899.security_groups.#: "1" => "1"
ingress.136061899.security_groups.2876704322: "sg-b80721c2" => "sg-b80721c2"
ingress.136061899.self: "false" => "false"
ingress.136061899.to_port: "443" => "443"
ingress.1467575756.cidr_blocks.#: "0" => "0"
ingress.1467575756.from_port: "61620" => "0"
ingress.1467575756.protocol: "tcp" => ""
ingress.1467575756.security_groups.#: "1" => "0"
ingress.1467575756.security_groups.2169191079: "sg-c70127bd" => ""
ingress.1467575756.self: "false" => "false"
ingress.1467575756.to_port: "61620" => "0"
ingress.149989598.cidr_blocks.#: "0" => "0"
ingress.149989598.from_port: "80" => "0"
ingress.149989598.protocol: "tcp" => ""
ingress.149989598.security_groups.#: "2" => "0"
ingress.149989598.security_groups.2980972767: "sg-c20127b8" => ""
ingress.149989598.security_groups.4111187599: "sg-c80127b2" => ""
ingress.149989598.self: "false" => "false"
ingress.149989598.to_port: "80" => "0"
ingress.187082154.cidr_blocks.#: "0" => "0"
ingress.187082154.from_port: "9042" => "0"
ingress.187082154.protocol: "tcp" => ""
ingress.187082154.security_groups.#: "1" => "0"
ingress.187082154.security_groups.2169191079: "sg-c70127bd" => ""
ingress.187082154.self: "false" => "false"
ingress.187082154.to_port: "9042" => "0"
ingress.2195547251.cidr_blocks.#: "0" => "0"
ingress.2195547251.from_port: "22" => "0"
ingress.2195547251.protocol: "tcp" => ""
ingress.2195547251.security_groups.#: "1" => "0"
ingress.2195547251.security_groups.2169191079: "sg-c70127bd" => ""
ingress.2195547251.self: "false" => "false"
ingress.2195547251.to_port: "22" => "0"
ingress.3400316176.cidr_blocks.#: "0" => "0"
ingress.3400316176.from_port: "7000" => "0"
ingress.3400316176.protocol: "tcp" => ""
ingress.3400316176.security_groups.#: "1" => "0"
ingress.3400316176.security_groups.2169191079: "sg-c70127bd" => ""
ingress.3400316176.self: "false" => "false"
ingress.3400316176.to_port: "7000" => "0"
ingress.357529062.cidr_blocks.#: "0" => "0"
ingress.357529062.from_port: "9160" => "0"
ingress.357529062.protocol: "tcp" => ""
ingress.357529062.security_groups.#: "1" => "0"
ingress.357529062.security_groups.2169191079: "sg-c70127bd" => ""
ingress.357529062.self: "false" => "false"
ingress.357529062.to_port: "9160" => "0"
ingress.3827110669.cidr_blocks.#: "0" => "0"
ingress.3827110669.from_port: "80" => "80"
ingress.3827110669.protocol: "tcp" => "tcp"
ingress.3827110669.security_groups.#: "1" => "1"
ingress.3827110669.security_groups.2876704322: "sg-b80721c2" => "sg-b80721c2"
ingress.3827110669.self: "false" => "false"
ingress.3827110669.to_port: "80" => "80"
ingress.3929861091.cidr_blocks.#: "0" => "0"
ingress.3929861091.from_port: "61621" => "0"
ingress.3929861091.protocol: "tcp" => ""
ingress.3929861091.security_groups.#: "1" => "0"
ingress.3929861091.security_groups.2169191079: "sg-c70127bd" => ""
ingress.3929861091.self: "false" => "false"
ingress.3929861091.to_port: "61621" => "0"
ingress.4211759511.cidr_blocks.#: "0" => "0"
ingress.4211759511.from_port: "1433" => "0"
ingress.4211759511.protocol: "tcp" => ""
ingress.4211759511.security_groups.#: "1" => "0"
ingress.4211759511.security_groups.3551840219: "sg-cc0127b6" => ""
ingress.4211759511.self: "false" => "false"
ingress.4211759511.to_port: "1433" => "0"
ingress.680397075.cidr_blocks.#: "0" => "0"
ingress.680397075.from_port: "7001" => "0"
ingress.680397075.protocol: "tcp" => ""
ingress.680397075.security_groups.#: "1" => "0"
ingress.680397075.security_groups.2169191079: "sg-c70127bd" => ""
ingress.680397075.self: "false" => "false"
ingress.680397075.to_port: "7001" => "0"
ingress.808665337.cidr_blocks.#: "0" => "0"
ingress.808665337.from_port: "7199" => "0"
ingress.808665337.protocol: "tcp" => ""
ingress.808665337.security_groups.#: "1" => "0"
ingress.808665337.security_groups.2169191079: "sg-c70127bd" => ""
ingress.808665337.self: "false" => "false"
ingress.808665337.to_port: "7199" => "0"
ingress.922980179.cidr_blocks.#: "0" => "0"
ingress.922980179.from_port: "3389" => "3389"
ingress.922980179.protocol: "tcp" => "tcp"
ingress.922980179.security_groups.#: "1" => "1"
ingress.922980179.security_groups.3352223863: "sg-940721ee" => "sg-940721ee"
ingress.922980179.self: "false" => "false"
ingress.922980179.to_port: "3389" => "3389"
[0m
[0m[33m~ aws_security_group.trf_sg_schdtasks
[0m egress.#: "2" => "1"
egress.1685082047.cidr_blocks.#: "0" => "0"
egress.1685082047.from_port: "80" => "80"
egress.1685082047.prefix_list_ids.#: "0" => "0"
egress.1685082047.protocol: "tcp" => "tcp"
egress.1685082047.security_groups.#: "1" => "1"
egress.1685082047.security_groups.2739699671: "sg-7004220a" => "sg-7004220a"
egress.1685082047.self: "false" => "false"
egress.1685082047.to_port: "80" => "80"
egress.4211759511.cidr_blocks.#: "0" => "0"
egress.4211759511.from_port: "1433" => "0"
egress.4211759511.prefix_list_ids.#: "0" => "0"
egress.4211759511.protocol: "tcp" => ""
egress.4211759511.security_groups.#: "1" => "0"
egress.4211759511.security_groups.3551840219: "sg-cc0127b6" => ""
egress.4211759511.self: "false" => "false"
egress.4211759511.to_port: "1433" => "0"
ingress.#: "2" => "1"
ingress.1685082047.cidr_blocks.#: "0" => "0"
ingress.1685082047.from_port: "80" => "80"
ingress.1685082047.protocol: "tcp" => "tcp"
ingress.1685082047.security_groups.#: "1" => "1"
ingress.1685082047.security_groups.2739699671: "sg-7004220a" => "sg-7004220a"
ingress.1685082047.self: "false" => "false"
ingress.1685082047.to_port: "80" => "80"
ingress.4211759511.cidr_blocks.#: "0" => "0"
ingress.4211759511.from_port: "1433" => "0"
ingress.4211759511.protocol: "tcp" => ""
ingress.4211759511.security_groups.#: "1" => "0"
ingress.4211759511.security_groups.3551840219: "sg-cc0127b6" => ""
ingress.4211759511.self: "false" => "false"
ingress.4211759511.to_port: "1433" => "0"
[0m
[0m[33m~ aws_security_group.trf_sg_sqlserver
[0m egress.#: "5" => "4"
egress.1333604694.cidr_blocks.#: "0" => "0"
egress.1333604694.from_port: "1434" => "1434"
egress.1333604694.prefix_list_ids.#: "0" => "0"
egress.1333604694.protocol: "udp" => "udp"
egress.1333604694.security_groups.#: "1" => "1"
egress.1333604694.security_groups.2739699671: "sg-7004220a" => "sg-7004220a"
egress.1333604694.self: "false" => "false"
egress.1333604694.to_port: "1434" => "1434"
egress.395540511.cidr_blocks.#: "0" => "0"
egress.395540511.from_port: "4022" => "4022"
egress.395540511.prefix_list_ids.#: "0" => "0"
egress.395540511.protocol: "tcp" => "tcp"
egress.395540511.security_groups.#: "1" => "1"
egress.395540511.security_groups.2739699671: "sg-7004220a" => "sg-7004220a"
egress.395540511.self: "false" => "false"
egress.395540511.to_port: "4022" => "4022"
egress.4073516292.cidr_blocks.#: "0" => "0"
egress.4073516292.from_port: "1433" => "1433"
egress.4073516292.prefix_list_ids.#: "0" => "0"
egress.4073516292.protocol: "tcp" => "tcp"
egress.4073516292.security_groups.#: "1" => "1"
egress.4073516292.security_groups.2739699671: "sg-7004220a" => "sg-7004220a"
egress.4073516292.self: "false" => "false"
egress.4073516292.to_port: "1433" => "1433"
egress.4228220737.cidr_blocks.#: "0" => "0"
egress.4228220737.from_port: "1433" => "0"
egress.4228220737.prefix_list_ids.#: "0" => "0"
egress.4228220737.protocol: "tcp" => ""
egress.4228220737.security_groups.#: "1" => "0"
egress.4228220737.security_groups.2980972767: "sg-c20127b8" => ""
egress.4228220737.self: "false" => "false"
egress.4228220737.to_port: "1433" => "0"
egress.833803214.cidr_blocks.#: "0" => "0"
egress.833803214.from_port: "1434" => "1434"
egress.833803214.prefix_list_ids.#: "0" => "0"
egress.833803214.protocol: "tcp" => "tcp"
egress.833803214.security_groups.#: "1" => "1"
egress.833803214.security_groups.2739699671: "sg-7004220a" => "sg-7004220a"
egress.833803214.self: "false" => "false"
egress.833803214.to_port: "1434" => "1434"
ingress.#: "5" => "4"
ingress.1333604694.cidr_blocks.#: "0" => "0"
ingress.1333604694.from_port: "1434" => "1434"
ingress.1333604694.protocol: "udp" => "udp"
ingress.1333604694.security_groups.#: "1" => "1"
ingress.1333604694.security_groups.2739699671: "sg-7004220a" => "sg-7004220a"
ingress.1333604694.self: "false" => "false"
ingress.1333604694.to_port: "1434" => "1434"
ingress.395540511.cidr_blocks.#: "0" => "0"
ingress.395540511.from_port: "4022" => "4022"
ingress.395540511.protocol: "tcp" => "tcp"
ingress.395540511.security_groups.#: "1" => "1"
ingress.395540511.security_groups.2739699671: "sg-7004220a" => "sg-7004220a"
ingress.395540511.self: "false" => "false"
ingress.395540511.to_port: "4022" => "4022"
ingress.4073516292.cidr_blocks.#: "0" => "0"
ingress.4073516292.from_port: "1433" => "1433"
ingress.4073516292.protocol: "tcp" => "tcp"
ingress.4073516292.security_groups.#: "1" => "1"
ingress.4073516292.security_groups.2739699671: "sg-7004220a" => "sg-7004220a"
ingress.4073516292.self: "false" => "false"
ingress.4073516292.to_port: "1433" => "1433"
ingress.4228220737.cidr_blocks.#: "0" => "0"
ingress.4228220737.from_port: "1433" => "0"
ingress.4228220737.protocol: "tcp" => ""
ingress.4228220737.security_groups.#: "1" => "0"
ingress.4228220737.security_groups.2980972767: "sg-c20127b8" => ""
ingress.4228220737.self: "false" => "false"
ingress.4228220737.to_port: "1433" => "0"
ingress.833803214.cidr_blocks.#: "0" => "0"
ingress.833803214.from_port: "1434" => "1434"
ingress.833803214.protocol: "tcp" => "tcp"
ingress.833803214.security_groups.#: "1" => "1"
ingress.833803214.security_groups.2739699671: "sg-7004220a" => "sg-7004220a"
ingress.833803214.self: "false" => "false"
ingress.833803214.to_port: "1434" => "1434"
[0m
[0m
[0m[1mPlan:[0m 0 to add, 3 to change, 0 to destroy.[0m

resource "aws_security_group" "xyz_sg_abc" {
name = "xyz-${var.environment}-sg-abc"
description = "Security Group for whatever"
vpc_id = "${aws_vpc.xyz_vpc.id}"

# HTTP access from anywhere
ingress {
from_port = 80
to_port = 80
protocol = "tcp"
security_groups = ["${aws_security_group.xyz_sg_cfelb01.id}"]
}
ingress {
from_port = 3389
to_port = 3389
protocol = "tcp"
security_groups = ["${aws_security_group.xyz_sg_win_jh.id}"]
}
ingress {
from_port = 3389
to_port = 3389
protocol = "udp"
security_groups = ["${aws_security_group.xyz_sg_win_jh.id}"]
}
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
security_groups = ["${aws_security_group.xyz_sg_cfelb01.id}"]
}

outbound internet access

egress {
from_port = 80
to_port = 80
protocol = "tcp"
security_groups = ["${aws_security_group.xyz_sg_cfelb01.id}"]
}
egress {
from_port = 3389
to_port = 3389
protocol = "tcp"
security_groups = ["${aws_security_group.xyz_sg_win_jh.id}"]
}
egress {
from_port = 3389
to_port = 3389
protocol = "udp"
security_groups = ["${aws_security_group.xyz_sg_win_jh.id}"]
}
egress {
from_port = 443
to_port = 443
protocol = "tcp"
security_groups = ["${aws_security_group.xyz_sg_cfelb01.id}"]
}

tags {
Name = "xyz-${var.environment}-sg-abc"
Environment = "${var.environment}"
}
}

resource "aws_security_group_rule" "xyz_sg_abc_rule01" {
type = "ingress"
from_port = 1433
to_port = 1433
protocol = "tcp"
security_group_id = "${aws_security_group.xyz_sg_abc.id}"
source_security_group_id = "${aws_security_group.xyz_sg_sqlserver.id}"
}

resource "aws_security_group_rule" "xyz_sg_abc_rule02" {
type = "egress"
from_port = 1433
to_port = 1433
protocol = "tcp"
security_group_id = "${aws_security_group.xyz_sg_abc.id}"
source_security_group_id = "${aws_security_group.xyz_sg_sqlserver.id}"
}

karthickk100 on 5 Oct 2016

I was also getting this problem. Even after normalizing it was still occurring. The issue turned out to be a CIDR that was actually invalid but AWS was silently correcting it on their side. Naturally Terraform saw this as a change.

If this still occurs for you, please double check ALL of your CIDR blocks and make sure they're valid for the network you're specifying. You might be having this issue.

farvour on 31 Oct 2016

👍1

So to recap, what we need to do is

normalize our protocol specifiers to lowercase,
specify all CIDR blocks in a single list to a single ingress or egress block, rather than multiple ones; and
make sure that our CIDR specifiers actually just point to the network we are defining (i.e. not 1.2.3.4/8, because that specifier would also have IP address info in it).

Seems like things that terraform validate should be able to help us catch, right?

llarsson on 4 Nov 2016

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.