Terraform-provider-aws: Can't set allow_remote_vpc_dns_resolution = true for aws_vpc_peering_connection

[jangrewe]

Terraform Version

Terraform v0.11.2

Affected Resource(s)

• aws_vpc_peering_connection
• aws_vpc_peering_connection_accepter

Terraform Configuration Files

provider "aws" {
  alias               = "requester"
  region              = "${var.requester_region}"
  assume_role {
    role_arn = "arn:aws:iam::${var.requester_account_id}:role/OrganizationAdmin${var.requester_team}${var.requester_env}"
  }
}

provider "aws" {
  alias               = "accepter"
  region              = "${var.accepter_region}"
  assume_role {
    role_arn = "arn:aws:iam::${var.accepter_account_id}:role/OrganizationAdmin${var.accepter_team}${var.accepter_env}"
  }
}


data "aws_vpc" "requester" {
  provider = "aws.requester"
  <...>
}

data "aws_vpc" "accepter" {
  provider = "aws.accepter"
  <...>
}


resource "aws_vpc_peering_connection" "default" {
  provider      = "aws.requester"
  vpc_id        = "${data.aws_vpc.requester.id}"
  peer_vpc_id   = "${data.aws_vpc.accepter.id}"
  peer_owner_id = "${var.accepter_account_id}"
  peer_region   = "${var.accepter_region}"
  auto_accept   = false

  requester {
    allow_remote_vpc_dns_resolution = true
  }

  accepter {
    allow_remote_vpc_dns_resolution = true
  }
}

resource "aws_vpc_peering_connection_accepter" "default" {
  provider                  = "aws.accepter"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.default.id}"
  auto_accept               = true

  requester {
    allow_remote_vpc_dns_resolution = true
  }

  accepter {
    allow_remote_vpc_dns_resolution = true
  }
}

Expected Behavior

The peering connection gets established and DNS resolution works in both directions.

Actual Behavior

• Without the requester and accepter blocks in aws_vpc_peering_connection and aws_vpc_peering_connection_accepter to enable the DNS resolution, everything is peachy!
  ... except that there's no DNS resolution for either remote end, of course.
  Both VPCs have enable_dns_hostnames = true, btw.

• With both blocks present in both resources,
  or both blocks present in only the aws_vpc_peering_connection resource
  or only the requester block present in only the aws_vpc_peering_connection resource
  or only the accepter block present in only the aws_vpc_peering_connection resource:

* aws_vpc_peering_connection.default: 1 error(s) occurred:
* aws_vpc_peering_connection.default: Unable to modify peering optionsaws_vpc_peering_connection_accepter. The VPC Peering Connection "pcx-0123456798" is not active. Please set `auto_accept` attribute to `true`, or activate VPC Peering Connection manually.

• With both blocks present in only the aws_vpc_peering_connection_accepter resource
  or only the requester block present in only the aws_vpc_peering_connection_accepter resource:

* aws_vpc_peering_connection_accepter.default: 1 error(s) occurred:
* aws_vpc_peering_connection_accepter.default: Error modifying VPC Peering Connection options: OperationNotPermitted: User [AccepterAccountID] does not have permission to modify the requester side peering options
    status code: 400, request id: a162d-b2-4ba7-a-96d2a

• With only the accepter block present in the aws_vpc_peering_connection_accepter resource, the peering connection gets created, and DNS resolution for the remote end (=requester) is enabled on the accepter's side - but i can't get it to be enabled for the remote end (=accepter) on the requester's side.

So the short and not-so-sweet version: The following doesn't work.

resource "aws_vpc_peering_connection" "default" {
  auto_accept   = false
  requester {
    allow_remote_vpc_dns_resolution = true
  }
}

Setting auto_accept = true for aws_vpc_peering_connection, as instructed by the error message, is also not an option, because:

* aws_vpc_peering_connection.default: 1 error(s) occurred:
* aws_vpc_peering_connection.default: peer_region cannot be set whilst auto_accept is true when creating a vpc peering connection

Steps to Reproduce

1. terraform apply

[jangrewe]

As this is a peering between accounts, and the peering needs to be active for the remote DNS resolution to be activated, i'm guessing it can only be done on the accepter's side, because it is not yet active when the requester sets it up? If so, is there any way around it?

[ewbankkit]

I think this is the same problem with the allow_remote_vpc_dns_resolution attribute as in

• https://github.com/terraform-providers/terraform-provider-aws/issues/3850
• https://github.com/terraform-providers/terraform-provider-aws/issues/3676

See also https://github.com/terraform-providers/terraform-provider-aws/pull/3097.

[ewbankkit]

The third bullet

With both blocks present in only the aws_vpc_peering_connection_accepter resource...

is an invalid scenario according to the AWS docs:

You must modify the requester VPC peering options if you are the requester of the VPC peering
connection, and you must modify the accepter VPC peering options if you are the accepter of the VPC
peering connection.

I'll modify the code to block this during terraform plan.

The second bullet

...only the requester block present in only the aws_vpc_peering_connection...

is the same scenario as in https://github.com/terraform-providers/terraform-provider-aws/issues/3850.
Unfortunately this can't be resolved via any combination of attributes right now - The requester needs the accepter to accept its side of the peering before going ahead and modifying the DNS resolution options. As the accepter is in another account you can't set auto_accept on the requester and need the aws_vpc_peering_accepter resource, but the accepter can't set the requester's options once it has accepted its side of the peering.
A longer-term solution would be to add a new resource just for the peering connection options and then make the requester's options depend on the accepter resource so that the connection is accepted before options are set.

[cornfeedhobo]

@ewbankkit what about in the case that terraform is managing both ends of this process? Can't the respective accepter credentials be used to automate this, like #3676 suggests?

[ewbankkit]

@cornfeedhobo The issue is a sequencing one in the aws_vpc_peering_connection code.
Using the requester's credentials its side of the connection is created via aws_vpc_peering_connection. As the accepter is in another account the requester can't accept the connection via auto_accept = true on its side so the connection is in _PendingAcceptance_ when the requester comes to modify its DNS options.
The accepter can accept its side of the connection via aws_vpc_peering_connection_accepter (using its credentials) and if auto_accept = true here then the connection becomes active so the accepter CAN set its DNS options and the requester could now set its DNS options.
We need to not set the requester's DNS options during the creation of the aws_vpc_peering_connection resource but set them after the accepter has accepted the connection. A new resource that manages the VPC peering connection options would do this.

[TomNorth]

@ewbankkit thanks for investigating this. I was going to suggest that an aws_vpc_peering_connection_options resource was needed for post acceptance, but I am purely a consumer of terraform and lack the skills to contribute. In the meantime, I have simply accepted and then modified my template, but this of course breaks the modular nature where I intend to re-use this for the test and prod environments seamlessly, as we're utilizing multi account separation. Thanks again for confirming and explaining.

[cornfeedhobo]

@ewbankkit that logic makes sense, but I have been wondering why this type of sequencing requires another resource vs simply altering the way the graph is calculated. This pattern of creating more resources for the sub-components and options keeps making terraform more and more verbose :-(

[ewbankkit]

@cornfeedhobo It's not a sequencing in the dependency graph but a sequencing in the current implementation code. The additional resource will correct the dependency graph.
Verbosity == Flexibility 😄.

[bflad]

The new aws_vpc_peering_connection_options resource for managing these dependency issues has been merged via #3909 and will release with v1.17.0 of the AWS provider, likely in two days.

[bflad]

The new resource has been released in version 1.17.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[jangrewe]

I just tried it out, seems to work as expected!

The crucial info about the dependency on the connection accepter in both options resources (for cross-account connections) in the docs was very helpful, so simple copypasta saved me from having to figure that out myself.

[willieddy]

Documentation on https://www.terraform.io/docs/providers/aws/r/vpc_peering.html should be updated. It currently has a note on the top of the page that is misleading. It states

Note: For cross-account (requester's AWS account differs from the accepter's AWS account) or inter-region VPC Peering Connections use the aws_vpc_peering_connection resource to manage the requester's side of the connection and use the aws_vpc_peering_connection_accepter resource to manage the accepter's side of the connection.

It should be changed to using the new resources https://github.com/terraform-providers/terraform-provider-aws/pull/3909 to configure cross-account or cross-region vpn options

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!