Terraform-provider-aws: aws_security_group_rule doesn't seem to support security groups from multiple aws accounts.

[hashibot]

_This issue was originally opened by @rayzorinc as hashicorp/terraform#6571. It was migrated here as part of the provider split. The original body of the issue is below._

---

Terraform Version

terraform --version
Terraform v0.6.15

Affected Resource(s)

Please list the resources as a list, for example:

• aws_security_group_rule
• aws_security_group

Terraform Configuration Files

resource "aws_security_group" "default" {
    name        = "default-group"
    vpc_id      = "${aws_vpc.vpc.id}"
}
resource "aws_security_group" "internal" {
    name        = "internal-access"
    vpc_id      = "${aws_vpc.vpc.id}"
}

resource "aws_security_group_rule" "port443" {
     type  = "ingress"
     from_port = 443
     to_port = 443
     protocol = "tcp"
     security_group_id = "${aws_security_group.internal.id}"
     source_security_group_id = "${aws_security_group.default.id}"
}

resource "aws_security_group_rule" "peer_access" {
     type  = "ingress"
     from_port = 443
     to_port = 443
     protocol = "tcp"
     security_group_id = "${aws_security_group.internal.id}"
     source_security_group_id = "123456789012/sg-12345678"
}

Expected Behavior

After running terraform apply, and it successfully ran, there should be no more infrastructure changes when running terraform plan

Actual Behavior

Running terraform plan keeps showing aws_security_group_rule.peer_access needs to be added.
output is:

-/+ module.network.aws_security_group_rule.peer_access
    from_port:                "443" => "443"
    protocol:                 "tcp" => "tcp"
    security_group_id:        "sg-12345678" => "sg-12345678"
    self:                     "false" => "0"
    source_security_group_id: "sg-12345678" => "123456789012/sg-12345678" (forces new resource)
    to_port:                  "443" => "443"
    type:                     "ingress" => "ingress"

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform apply
2. terraform plan

Important Factoids

This errors only when we're trying to allow access from different aws accounts that are vpc peered to each other.

In the terraform.tfstate file, I see it stores it without the other aws account number and just the security group id.

                "aws_security_group_rule.peer_access": {
                    "type": "aws_security_group_rule",
                    "depends_on": [
                        "aws_security_group.internal"
                    ],
                    "primary": {
                        "id": "sgrule-261220255",
                        "attributes": {
                            "cidr_blocks.#": "0",
                            "from_port": "443",
                            "id": "sgrule-261220255",
                            "protocol": "tcp",
                            "security_group_id": "sg-12345678",
                            "self": "false",
                            "source_security_group_id": "sg-12345678",
                            "to_port": "443",
                            "type": "ingress"
                        },
                        "meta": {
                            "schema_version": "2"
                        }
                    }
                },

[barnesm999]

+1
I am seeing this bug on Terraform v0.10.4, AWS Provider version 0.1.4.

Unlike the previous report I do actually see the AWS Acount number making it to the Terraform State:

"aws_security_group_rule.sg_rule": {
    "type": "aws_security_group_rule",
    "depends_on": [
        "aws_security_group.security_group"
    ],
    "primary": {
        "id": "sgrule-2562427353",
        "attributes": {
            "from_port": "8080",
            "id": "sgrule-2562427353",
            "protocol": "tcp",
            "security_group_id": "sg-12345678",
            "self": "false",
            "source_security_group_id": "123456789012/sg-12345678",
            "to_port": "8080",
            "type": "ingress"
        },
        "meta": {
            "schema_version": "2"
         },
        "tainted": false
    },
    "deposed": [ ],
    "provider": ""
},

Terraform State commands shows the same:

terraform state show ...aws_security_group_rule.sg_rule
id                       = sgrule-2562427353
from_port                = 8080
protocol                 = tcp
security_group_id        = sg-12345678
self                     = false
source_security_group_id = 123456789012/sg-12345678
to_port                  = 8080
type                     = ingress

But Terraform Plan seems to think that the source_security_group_id is just the Security Group Id without the Account Number.

terraform plan
-/+ ....aws_security_group_rule.sg_rule (new resource required)
      id:                       "sgrule-2562427353" => <computed>
      from_port:                "8080" => "8080"
      protocol:                 "tcp" => "tcp"
      security_group_id:        "sg-12345678" => "sg-12345678"
      self:                     "false" => "false"
      source_security_group_id: "sg-12345678" => "123456789012/sg-12345678"
      to_port:                  "8080" => "8080"
      type:                     "ingress" => "ingress"

I have validated that the created Security Group Rule is created properly with the AWS Account Number in the source field.

[davidfic]

I'm also seeing the same thing on v0.10.7. It's in the state file, just not able to be completed by Terraform.

[oparrish2]

Seeing the same issue with terraform v0.11.2 and aws provider v1.6.0_x4

[bflad]

Hi folks, are these cross-account rules successfully created with correct permissions (and its just the diff that's the issue now) or can someone please provide any relevant error output?

[oparrish2]

In my use case, it's VPC peering, everything is created fine but, the rules are recreated every subsequent apply even with no changes.

Example resource config

resource "aws_security_group_rule" "all-syslog" {
  type            = "egress"
  from_port       = 602
  to_port         = 602
  protocol        = "tcp"
  source_security_group_id = "${var.account-id}/${var.peer-syslog-group-id}"

  security_group_id = "${aws_security_group.all.id}"
}

The state

$ terraform state show module.cluster-us-east-1.aws_security_group_rule.all-syslog
id                       = sgrule-665881303
from_port                = 602
protocol                 = tcp
security_group_id        = sg-1234
self                     = false
source_security_group_id = REDACTED_ACCOUNT_ID/sg-5678
to_port                  = 602
type                     = egress

Running terraform apply:

-/+ aws_security_group_rule.all-syslog (new resource required)
      id:                       "sgrule-665881303" => <computed> (forces new resource)
      from_port:                "602" => "602"
      protocol:                 "tcp" => "tcp"
      security_group_id:        "sg-1234" => "sg-1234"
      self:                     "false" => "false"
      source_security_group_id: "sg-5678" => "REDACTED_ACCOUNT_ID/sg-5678" (forces new resource)
      to_port:                  "602" => "602"
      type:                     "egress" => "egress"

[oparrish2]

An actual problem that is generated by this behavior is that when a terraform apply is done to modify the groups, all the rules are destroyed and then re-added. If you have running instances in the security group that rely on the present of the security group rules, then there will be unexpected interruption in connectivity.

[ctrowat]

We are also having this problem. Every subsequent run of terraform recreates the security group rules which is disruptive and unnecessary. It looks like this should hopefully be a pretty easy fix since the necessary data to determine no actual change is requires is being returned by the AWS API call.

It looks like perhaps this:
https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_security_group_rule.go#L693
Needs to also be checking if the "UserId" property is set the same as the security group's "OwnerId" and if not, including that as a prefix with a joining slash.

Terraform version v0.11.3
provider.aws v1.11.0

[devonbleak]

From my testing I was able to reference another account's peered VPC security group just using the security group ID with no OwnerId - AWS did the right thing on the backend. This may be fixable with a documentation update indicating that, or if we want to be more explicit about it I would add a source_security_group_owner_id field so we're not dealing with adding/detecting delimiters like in aws_security_group and the CLI.

resource "aws_security_group" "test" {
  vpc_id = "vpc-03c3cdREDACTED"
  name = "xaccount_test"
  description = "xaccount test"
}

resource "aws_security_group_rule" "test" {
  security_group_id = "${aws_security_group.test.id}"
  type = "ingress"
  source_security_group_id = "sg-02c3REDACTED"
  protocol = -1
  from_port = 0
  to_port = 0
}

(aws:shared-dev/us-west-2)
~/tmp/tf devonb$ terraform plan -out test.plan                                    
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + aws_security_group.test
      id:                       <computed>
      arn:                      <computed>
      description:              "xaccount test"
      egress.#:                 <computed>
      ingress.#:                <computed>
      name:                     "xaccount_test"
      owner_id:                 <computed>
      revoke_rules_on_delete:   "false"
      vpc_id:                   "vpc-03c3cdREDACTED"

  + aws_security_group_rule.test
      id:                       <computed>
      from_port:                "0"
      protocol:                 "-1"
      security_group_id:        "${aws_security_group.test.id}"
      self:                     "false"
      source_security_group_id: "sg-02c3REDACTED"
      to_port:                  "0"
      type:                     "ingress"


Plan: 2 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

This plan was saved to: test.plan

To perform exactly these actions, run the following command to apply:
    terraform apply "test.plan"


(aws:shared-dev/us-west-2)
~/tmp/tf devonb$ terraform apply test.plan
aws_security_group.test: Creating...
  arn:                    "" => "<computed>"
  description:            "" => "xaccount test"
  egress.#:               "" => "<computed>"
  ingress.#:              "" => "<computed>"
  name:                   "" => "xaccount_test"
  owner_id:               "" => "<computed>"
  revoke_rules_on_delete: "" => "false"
  vpc_id:                 "" => "vpc-03c3cdREDACTED"
aws_security_group.test: Creation complete after 2s (ID: sg-017a07d781b9829d7)
aws_security_group_rule.test: Creating...
  from_port:                "" => "0"
  protocol:                 "" => "-1"
  security_group_id:        "" => "sg-017a07d781b9829d7"
  self:                     "" => "false"
  source_security_group_id: "" => "sg-02c3REDACTED"
  to_port:                  "" => "0"
  type:                     "" => "ingress"
aws_security_group_rule.test: Creation complete after 1s (ID: sgrule-1199968668)

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

(aws:shared-dev/us-west-2)
~/tmp/tf devonb$ terraform plan -out test.plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_security_group.test: Refreshing state... (ID: sg-017a07d781b9829d7)
aws_security_group_rule.test: Refreshing state... (ID: sgrule-1199968668)

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

(aws:shared-dev/us-west-2)
~/tmp/tf devonb$ aws ec2 describe-security-groups --group-ids sg-017a07d781b9829d7
{
    "SecurityGroups": [
        {
            "IpPermissionsEgress": [], 
            "Description": "xaccount test", 
            "IpPermissions": [
                {
                    "IpProtocol": "-1", 
                    "PrefixListIds": [], 
                    "IpRanges": [], 
                    "UserIdGroupPairs": [
                        {
                            "VpcId": "vpc-26a7REDACTED", 
                            "UserId": "0579REDACTED", 
                            "GroupId": "sg-02c3REDACTED", 
                            "VpcPeeringConnectionId": "pcx-0ef3f02bREDACTED", 
                            "PeeringStatus": "active"
                        }
                    ], 
                    "Ipv6Ranges": []
                }
            ], 
            "GroupName": "xaccount_test", 
            "VpcId": "vpc-03c3cdREDACTED", 
            "OwnerId": "8171REDACTED", 
            "GroupId": "sg-017a07d781b9829d7"
        }
    ]
}

[rverma-nikiai]

This is still an open issue, I even raised this to aws support team but they are not covering this. It's strange that such a bug exists for 3 years and is being completely ignored.

[scalp42]

Same issue here, constantly re-creating:

aws_security_group_rule.bastion_ssh: Creating...
  from_port:                "" => "22"
  protocol:                 "" => "tcp"
  security_group_id:        "" => "sg-04e93fb33cd928570"
  self:                     "" => "false"
  source_security_group_id: "" => "00000PEEREDVPCACCOUNT/sg-01d1ab630dcacf755"
  to_port:                  "" => "22"
  type:                     "" => "ingress"
aws_security_group_rule.bastion_ssh: Creation complete after 1s (ID: sgrule-870448872)

Related: https://aws.amazon.com/about-aws/whats-new/2016/03/announcing-support-for-security-group-references-in-a-peered-vpc/

[mwarkentin]

We are running into this as well, @devonbleak's workaround seems to work for us as well, although we would like to be able to specify the account ID if possible.

[scalp42]

@mwarkentin confused, which workaround? Not putting the owner ID is breaking for us.

I'll give it another try 😅

[red8888]

Not putting owner ID works for me. It makes sense because sg IDs are globally unique right? So AWS would know which account owns that. So this is still open because terraform doesn't support (or update state at least) with the string as seen in the web console right? which looks like 123456/sg-abc123

whats funny is specifying an inline rule in the aws_security_group resource with the 123456/sg-abc123 _does_ work and update state correctly. so aws_security_group already does support this

[SiahaanBernard]

From my testing, i found interesting fact, when referencing security group from another account where the security group id length is 19 characters (this is a new unique id convention for security group) is working fine, but when referencing the security group where the security group id is 11 characters (old implementation, which i assume only unique to an account) we need to specify the account id owner of that security group.
I think for some people this is not a problem because maybe their security group was created after the AWS changes how they created unique id of security group, so all of their security group have 19 characters unique id. But for us, we have lots of security group where security group id still use the old implementation (unique id with 11 characters). We actually consider use AWS console when we need to create the cross account security group, as if we are using terraform security_group_rule for cross account security group id, it will created and destroy the resource, and disrupt the connection.

Here is my code and plan to reproduce my testing scenario.

Terraform Version

terraform version
Terraform v0.11.13
+ provider.aws v2.18.0

Affected Resource(s)

• aws_security_group_rules

Terraform Configuration Files

resource "aws_security_group" "cross_account_rules" {
  name        = "cross_account_sg"
  vpc_id      = "vpc-1a2b3c4d"
  description = "security group for testing cross account security group rule"
}

resource "aws_security_group_rule" "short_sg_id_rule" {
  type                     = "ingress"
  from_port                = "443"
  to_port                  = "443"
  protocol                 = "tcp"
  security_group_id        = "${aws_security_group.cross_account_rules.id}"
  source_security_group_id = "123456789876/sg-1a2b3c4d"
}

resource "aws_security_group_rule" "long_sg_id_rule" {
  type                     = "ingress"
  from_port                = "443"
  to_port                  = "443"
  protocol                 = "tcp"
  security_group_id        = "${aws_security_group.cross_account_rules.id}"
  source_security_group_id = "sg-1a2b3c4d5e6f7g8h"
}

Expected Behavior

After running terraform apply, and it successfully ran, there should be no more infrastructure changes when running terraform plan

Actual Behavior

Running terraform plan keeps showing aws_security_group_rule.peer_access needs to be added.
output is:

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

-/+ aws_security_group_rule.short_sg_id_rule (new resource required)
      id:                       "sgrule-4096904641" => <computed> (forces new resource)
      from_port:                "443" => "443"
      protocol:                 "tcp" => "tcp"
      security_group_id:        "sg-h8g76f5e4d3c2b1a" => "sg-h8g76f5e4d3c2b1a"
      self:                     "false" => "false"
      source_security_group_id: "sg-1a2b3c4d" => "123456789876/sg-1a2b3c4d" (forces new resource)
      to_port:                  "443" => "443"
      type:                     "ingress" => "ingress"


Plan: 1 to add, 0 to change, 1 to destroy.

------------------------------------------------------------------------

Steps to Reproduce

1. terraform apply
2. terraform plan

Actually we can create the new security group to replace the old security group to handle this issue, but i think this cost so much effort to do, we need to create new security group, ensure all rules from old security group exist on new security group, ensure all security group that reference old security group also reference to this new security group, attach new security group to ec2 instances, and detach old security group.

[ketzacoatl]

Help, this is still present in the current release of the AWS provider!

[ktham]

@bflad I can verify that Terraform successfully creates the cross-account rules, but the problem is that the rule keeps getting deleted and re-created on every subsequent apply, resulting in a slight downtime when this happens. The problem is that on state refresh, the account ID prefix is removed from tf state. I resolved the issue here if you can take a look at it: https://github.com/terraform-providers/terraform-provider-aws/pull/11809

[anGie44]

Note: The fix for this has been merged and will release with version 2.62.0 of the Terraform AWS Provider, expected in the this week's release.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!