Terraform-provider-aws: Cycle error on security group creation

[DrHashi]

Version 0.11.8
`
resource "aws_security_group" "prometheus_sg" {
name = "${var.env}-prometheus-sg"
description = "${var.env}-prometheus-sg"
vpc_id = "${aws_vpc.vpc.id}"

ingress {
    from_port       = 9090
    to_port         = 9094
    protocol        = "tcp"
    security_groups = ["${aws_security_group.bastion.id}"]
    self            = false
}

ingress {
    from_port       = 22
    to_port         = 22
    protocol        = "tcp"
    security_groups = ["${aws_security_group.bastion.id}"]
    self            = false
}

egress {
    from_port       = 9100
    to_port         = 9491
    protocol        = "-1"
    security_groups = ["${aws_security_group.logging_prometheus_sg.id}"]
}

egress {
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    cidr_blocks     = ["0.0.0.0/0"]
}

    egress {
    from_port       = 443
    to_port         = 443
    protocol        = "tcp"
    cidr_blocks     = ["0.0.0.0/0"]
}

    egress {
    from_port       = 123
    to_port         = 123
    protocol        = "udp"
    security_groups = ["${aws_security_group.ntp.id}"]
}

}

resource "aws_security_group" "logging_prometheus_sg" {
name = "${var.env}-logging-prometheus-sg"
description = "${var.env}-logging-prometheus-sg"
vpc_id = "${aws_vpc.vpc.id}"

ingress {
    from_port       = 9100
    to_port         = 9491
    protocol        = "tcp"
    security_groups = ["${aws_security_group.prometheus_sg.id}"]
    self            = false
}

egress {
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    cidr_blocks     = ["0.0.0.0/0"]
}

    egress {
    from_port       = 443
    to_port         = 443
    protocol        = "tcp"
    cidr_blocks     = ["0.0.0.0/0"]
}    

depends_on = ["aws_security_group.prometheus_sg"]

}`

Output

$ terraform plan

`Error: Error asking for user input: 1 error(s) occurred:

• Cycle: aws_security_group.prometheus_sg, aws_security_group.logging_prometheus_sg`

[blckct]

The problem is pretty self-explanatory. Terraform can't create those resources because there's a cyclic dependence. Either use more that two security groups or use the aws_security_group_rule resource instead.

[DrHashi]

The problem is pretty self-explanatory. Terraform can't create those resources because there's a cyclic dependence. Either use more that two security groups or use the aws_security_group_rule resource instead.

Can this not be implemented as a feature? It seems to be allowed by AWS

[YakDriver]

The can be easily done with existing resources.

resource "aws_security_group" "prometheus_sg" {
  name        = "${var.env}-prometheus-sg"
  description = "${var.env}-prometheus-sg"
  vpc_id      = "${aws_vpc.vpc.id}"

  ingress {
    from_port       = 9090
    to_port         = 9094
    protocol        = "tcp"
    security_groups = ["${aws_security_group.bastion.id}"]
    self            = false
  }

  ingress {
    from_port       = 22
    to_port         = 22
    protocol        = "tcp"
    security_groups = ["${aws_security_group.bastion.id}"]
    self            = false
  }

  egress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port       = 123
    to_port         = 123
    protocol        = "udp"
    security_groups = ["${aws_security_group.ntp.id}"]
  }
}

# this rule depends on both security groups so separating it allows it
# to be created after both
resource "aws_security_group_rule" "extra_rule" {
  security_group_id        = "${aws_security_group.prometheus_sg.id}"
  from_port                = 9100
  to_port                  = 9491
  protocol                 = "-1"
  type                     = "egress"
  source_security_group_id = "${aws_security_group.logging_prometheus_sg.id}"
}

resource "aws_security_group" "logging_prometheus_sg" {
  name        = "${var.env}-logging-prometheus-sg"
  description = "${var.env}-logging-prometheus-sg"
  vpc_id      = "${aws_vpc.vpc.id}"

  ingress {
    from_port       = 9100
    to_port         = 9491
    protocol        = "tcp"
    security_groups = ["${aws_security_group.prometheus_sg.id}"]
    self            = false
  }

  egress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  # you don't need this
  #depends_on = ["aws_security_group.prometheus_sg"]
}

[bflad]

Mixing aws_security_group and aws_security_group_rule resources for the same "physical" security group will generate a perpetual difference of trying to add/remove rules between them as noted in the resource documentation: https://www.terraform.io/docs/providers/aws/r/security_group.html

To workaround that issue in the configuration noted above, convert (at least) all the egress rules in resource "aws_security_group" "prometheus_sg" { to separate aws_security_group_rule resources.

There is a _super_ long history behind this security group cycle issue, but here's a good starting point: https://github.com/terraform-providers/terraform-provider-aws/pull/1824

[aeschright]

Hi everyone, since there's a workaround documented above, I'm going to close the issue. If you encounter more problems in this area, we'd like you to open a new bug report. Thanks for taking the time to discuss the problem!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!