Terraform: Recreate "Security Group Rule" each time I execute “terrafrom apply/plan”

Created on 3 Jan 2017 · 9Comments · Source: hashicorp/terraform

When I execute terraform apply or plan without doing any changes into any terraform scripts, terraform is going to add same security group rules again and again.

let me describe about my terraform scripts.I have designed my terraform script as modules, security group is a module and security group rule is also a module.

++++++++++++++++Main.tf++++++++++++++++++++++++++++++++++++++++++++++++++
module "application_sg" {
source = "modules/securitygroups"
security_group_name = "Application Security Group"
vpc_id = "${module.vpc.vpc_id}"
}

module "rule2"{
source = "modules/securitygroups/rules"
type = "ingress"
from_port = 8080
to_port = 8080
protocol = "tcp"
cidr_blocks = ["20.0.1.0/24"]
security_group_id = "${module.application_sg.security_group}"
}

module "rule3"{
source = "modules/securitygroups/rules"
type = "ingress"
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["20.0.1.0/24"]
security_group_id = "${module.application_sg.security_group}"
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Lets say I am executing this scripts again and again using terraform plan.

This is the output I am getting.See it tries to add rule which is already existing.

~ module.application_sg.aws_security_group.security_group
ingress.#: "3" => "1"
ingress.2358522502.cidr_blocks.#: "1" => "0"
ingress.2358522502.cidr_blocks.0: "20.0.1.0/24" => ""
ingress.2358522502.from_port: "443" => "0"
ingress.2358522502.protocol: "tcp" => ""
ingress.2358522502.security_groups.#: "0" => "0"
ingress.2358522502.self: "false" => "false"
ingress.2358522502.to_port: "443" => "0"
ingress.3250959853.cidr_blocks.#: "1" => "0"
ingress.3250959853.cidr_blocks.0: "20.0.1.0/24" => ""
ingress.3250959853.from_port: "8080" => "0"
ingress.3250959853.protocol: "tcp" => ""
ingress.3250959853.security_groups.#: "0" => "0"
ingress.3250959853.self: "false" => "false"
ingress.3250959853.to_port: "8080" => "0"
ingress.753360330.cidr_blocks.#: "0" => "0"
ingress.753360330.from_port: "0" => "0"
ingress.753360330.protocol: "-1" => "-1"
ingress.753360330.security_groups.#: "0" => "0"
ingress.753360330.self: "true" => "true"
ingress.753360330.to_port: "0" => "0"

module.rule1.aws_security_group_rule.rule
cidr_blocks.#: "1"
cidr_blocks.0: "20.0.1.0/24"
from_port: "80"
protocol: "tcp"
security_group_id: "sg-17c13770"
self: "false"
source_security_group_id: ""
to_port: "80"

type: "ingress"

bug provideaws

Source

AdimUser

Most helpful comment

@aglover-zendesk This happens "by design". At some point in the future we may look into allowing all of the aws_security_group_rule resources to fully quantify all of the security group rules for a security group, but this will be an opt-in feature, and isn't on the roadmap yet. :smile:

The main purpose of designing the aws_security_group_rule to not describe the "full state" of a security group's rules, is that in a distributed Terraform setup, where multiple repositories and multiple state files makeup a user's infrastructure, the loose contract of an aws_security_group_rule works really well. Defining the security group rules inline allows for the strict contract where: "these are a definitive list of all of the security group rules for this security group resource".

The best solution is to either define _all_ of a security group's rules inline, or _none_ of the security group's rules inline. It's when a user defines both that the mismatch occurs, as the inline rules are parsed as "definitive" and attempt to overwrite the individually defined security group rules.

Hopefully this answers your question, happy to discuss further though!

grubernaut on 28 Feb 2017

👍5

All 9 comments

Seeing same behavior Terraform 0.8.5 & 0.8.7 where security group resource is defined in a module and references to security group from security group rule cause cyclical create (security group rule)/ change (security group) state. Might have to wait until this approach can persist being modularized.

My code is almost exact to what @AdimUser described.

shaunofneuron on 16 Feb 2017

Hi @AdimUser thanks for the issue!

Would you be able to clarify for me if you are specifying ingress/egress rules inside of an aws_security_group resource, as well as adding rules with an aws_security_group_rule resource that references the initial security group?

For example, the following psuedo-config will reproduce the error you're seeing:

resource "aws_security_group" "foo" {
  ingress {
    ....
  }
  egress {
    ...
  }
}

resource "aws_security_group_rule" "bar" {
  security_group_id = "${aws_security_group.foo.id}"
  type = "ingress"
  ...
}

This is a known issue, as the aws_security_group resource will attempt to manage _all_ of the security group rules for that resource, while the aws_security_group_rule resource does not establish that same contract between security group rules.

Does this match what you have in your configuration, or does the source security group not include any inline ingress/egress rules? Thanks!

grubernaut on 16 Feb 2017

Also seeing this behavior on Terraform v0.8.6

I have a module for creating an RDS cluster and two security groups that should be used by the RDS cluster. Each of my security groups are defined with an aws_security_group resource and a aws_security_group_rule rule, like this:

resource "aws_security_group" "main" {
  name        = "${var.name}--admin"
  description = "Allows traffic to rds"
  vpc_id      = "${var.vpc_id}"

  ingress {
    from_port       = "${var.port}"
    to_port         = "${var.port}"
    protocol        = "TCP"
    security_groups = ["${var.security_groups}"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = -1
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags {
    Name        = "RDS cluster (${var.name})"
    Environment = "${var.environment}"
  }
}

resource "aws_security_group_rule" "allow_app" {
  type              = "ingress"
  from_port         = 3306
  to_port           = 3306
  protocol          = "tcp"
  cidr_blocks       = ["${var.app_subnets}"]
  security_group_id = "${aws_security_group.main.id}"
}

When I run apply, it strips all of the ingress rules. The next time I run apply, it adds them all back. Unfortunately, since I have two security groups, they are staggered so TF is always stripping ingress rules from one SG and re-adding them to the other SG. In other words, I can't get to the desired state of having both SGs populated with their ingress rules. Example output (subnets replaced with 'dummy' values):

...
module.stack.rds.aws_security_group.main: Modifying...
  ingress.#:                                    "2" => "1"
  ingress.2000300587.cidr_blocks.#:             "0" => "0"
  ingress.2000300587.from_port:                 "3306" => "3306"
  ingress.2000300587.protocol:                  "TCP" => "tcp"
  ingress.2000300587.security_groups.#:         "1" => "1"
  ingress.2000300587.security_groups.571983794: "sg-abc12345 => "sg-abc12345
  ingress.2000300587.self:                      "false" => "false"
  ingress.2000300587.to_port:                   "3306" => "3306"
  ingress.554452638.cidr_blocks.#:              "21" => "0"
  ingress.554452638.cidr_blocks.0:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.1:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.10:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.11:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.12:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.13:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.14:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.15:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.16:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.17:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.18:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.19:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.2:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.20:             "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.3:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.4:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.5:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.6:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.7:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.8:              "10.0.0.0/8" => ""
  ingress.554452638.cidr_blocks.9:              "10.0.0.0/8" => ""
  ingress.554452638.from_port:                  "3306" => "0"
  ingress.554452638.protocol:                   "tcp" => ""
  ingress.554452638.security_groups.#:          "0" => "0"
  ingress.554452638.self:                       "false" => "false"
  ingress.554452638.to_port:                    "3306" => "0"
module.stack.rds.aws_security_group_rule.allow_zendesk_app_and_admin: Creating...
  cidr_blocks.#:            "" => "38"
  cidr_blocks.0:            "" => "10.0.0.0/8"
  cidr_blocks.1:            "" => "10.0.0.0/8"
  cidr_blocks.10:           "" => "10.0.0.0/8"
  cidr_blocks.11:           "" => "10.0.0.0/8"
  cidr_blocks.12:           "" => "10.0.0.0/8"
  cidr_blocks.13:           "" => "10.0.0.0/8"
  cidr_blocks.14:           "" => "10.0.0.0/8"
  cidr_blocks.15:           "" => "10.0.0.0/8"
  cidr_blocks.16:           "" => "10.0.0.0/8"
  cidr_blocks.17:           "" => "10.0.0.0/8"
  cidr_blocks.18:           "" => "10.0.0.0/8"
  cidr_blocks.19:           "" => "10.0.0.0/8"
  cidr_blocks.2:            "" => "10.0.0.0/8"
  cidr_blocks.20:           "" => "10.0.0.0/8"
  cidr_blocks.21:           "" => "10.0.0.0/8"
  cidr_blocks.22:           "" => "10.0.0.0/8"
  cidr_blocks.23:           "" => "10.0.0.0/8"
  cidr_blocks.24:           "" => "10.0.0.0/8"
  cidr_blocks.25:           "" => "10.0.0.0/8"
  cidr_blocks.26:           "" => "10.0.0.0/8"
  cidr_blocks.27:           "" => "10.0.0.0/8"
  cidr_blocks.28:           "" => "10.0.0.0/8"
  cidr_blocks.29:           "" => "10.0.0.0/8"
  cidr_blocks.3:            "" => "10.0.0.0/8"
  cidr_blocks.30:           "" => "10.0.0.0/8"
  cidr_blocks.31:           "" => "10.0.0.0/8"
  cidr_blocks.32:           "" => "10.0.0.0/8"
  cidr_blocks.33:           "" => "10.0.0.0/8"
  cidr_blocks.34:           "" => "10.0.0.0/8"
  cidr_blocks.35:           "" => "10.0.0.0/8"
  cidr_blocks.36:           "" => "10.0.0.0/8"
  cidr_blocks.37:           "" => "10.0.0.0/8"
  cidr_blocks.4:            "" => "10.0.0.0/8"
  cidr_blocks.5:            "" => "10.0.0.0/8"
  cidr_blocks.6:            "" => "10.0.0.0/8"
  cidr_blocks.7:            "" => "10.0.0.0/8"
  cidr_blocks.8:            "" => "10.0.0.0/8"
  cidr_blocks.9:            "" => "10.0.0.0/8"
  from_port:                "" => "3306"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-abc12345
  self:                     "" => "false"
  source_security_group_id: "" => "<computed>"
  to_port:                  "" => "3306"
  type:                     "" => "ingress"
module.stack.rds.aws_security_group.main: Modifications complete
...

aglover-zendesk on 28 Feb 2017

Just re-read your comment @grubernaut, looks like I fall squarely into the known issue space. What's the suggested workaround/solution? Don't use aws_security_group_rule resources for the time being?

aglover-zendesk on 28 Feb 2017

Hopefully this answers your question, happy to discuss further though!

grubernaut on 28 Feb 2017

👍5

Hey @grubernaut thanks, that does make sense. I see you already have a warning/notice at the top of the aws_security_group page in the docs too, so that's on me. Thanks for the quick answer.

aglover-zendesk on 28 Feb 2017

❤2

Hello all,

Going to close this for now, more than happy to discuss further if the issue persists. Thanks!

grubernaut on 13 Apr 2017

Thanks for your notes on this. The design does make sense.

I feel the user experience could be improved with a meaningful error though. If an aws_security_group has inline rules, could an error be thrown if it is ever referenced by an aws_security_group_rule?

cavaliercoder on 15 Jul 2017

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.