Terraform-provider-aws: aws_api_gateway_rest_api private API does not set VPC Endpoint association or policy until SECOND apply.

[bassmanitram]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.14
AWS provider 2.67.0

Affected Resource(s)

• aws_api_gateway_rest_api

Terraform Configuration Files

resource "aws_api_gateway_rest_api" "service_api_private_vpce" {
  count = "${local.private_vpce ? 1 : 0}"
  name = "${var.api_name}"

  body = "${var.api_openapi_yaml == "" ? data.template_file.openapi_yaml.rendered : var.api_openapi_yaml}"

  policy = "${data.template_file.policy.rendered}"

  endpoint_configuration {
  #
  # This will always be PRIVATE here
  #
    types = ["${local.endpoint_type}"]
    vpc_endpoint_ids = ["${module.environment_vpc_endpoint_id.ssm_parameter_value}"]
  }
  lifecycle {
      #
      # Plans have a permanent difference here!
      #
      ignore_changes = [ "binary_media_types" ]
  }
}

Expected Behavior

API should be created with endpoint type PRIVATE, associated with the passed VPCE ID and with
the provided policy attached

Actual Behavior

API is created as PRIVATE but without the VPCE association or the policy. Then running terraform apply a second time, the policy and the VPCE association are set

Steps to Reproduce

• create the necessary infrastructure to have a VPCE associated with the API Gateway
• create a terraform config file with the contents as provided by the plan output below, substituting
  the VPCE ID in your config for the one currently in the content below
• terraform apply
• examine the API in the API Gateway console - no policy, no VPCE
• terraform apply
• examine the API in the API Gateway console - policy and VPCE exist

Important Factoids

While the above resource property settings use a lot of variables, the plan output shows that the values have been correctly resolved

  + module.main.module.console_service_api.module.api.aws_api_gateway_rest_api.service_api_private_vpce
      id:                                                    <computed>
      api_key_source:                                        "HEADER"
      arn:                                                   <computed>
      body:                                                  "---\nswagger: \"2.0\"\ninfo:\n  version: \"2019-02-19T16:23:40Z\"\n  title: \"Assure-Platform-Console-Service-private-1\"\nschemes:\n- \"https\"\npaths:\n  /:\n    get:\n      produces:\n      - \"application/json\"\n      responses:\n        200:\n          description: \"200 response\"\n          schema:\n            $ref: \"#/definitions/Empty\"\n      x-amazon-apigateway-integration:\n        uri: \"arn:aws:apigateway:us-west-2:lambda:path/2015-03-31/functions/arn:aws:lambda:us-west-2:665158502186:function:${stageVariables.integrationLambda}/invocations\"\n        credentials: \"arn:aws:iam::665158502186:role/${stageVariables.integrationLambdaExecRole}\"\n        responses:\n          default:\n            statusCode: \"200\"\n        passthroughBehavior: \"when_no_match\"\n        httpMethod: \"POST\"\n        contentHandling: \"CONVERT_TO_TEXT\"\n        type: \"aws_proxy\"\n    options:\n      consumes:\n      - \"application/json\"\n      produces:\n      - \"application/json\"\n      responses:\n        200:\n          description: \"200 response\"\n          schema:\n            $ref: \"#/definitions/Empty\"\n          headers:\n            Access-Control-Allow-Origin:\n              type: \"string\"\n            Access-Control-Allow-Methods:\n              type: \"string\"\n            Access-Control-Allow-Headers:\n              type: \"string\"\n      x-amazon-apigateway-integration:\n        responses:\n          default:\n            statusCode: \"200\"\n            responseParameters:\n              method.response.header.Access-Control-Allow-Methods: \"'POST,OPTIONS'\"\n              method.response.header.Access-Control-Allow-Headers: \"'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'\"\n              method.response.header.Access-Control-Allow-Origin: \"'*'\"\n        requestTemplates:\n          application/json: \"{\\\"statusCode\\\": 200}\"\n        passthroughBehavior: \"when_no_match\"\n        type: \"mock\"\ndefinitions:\n  Empty:\n    type: \"object\"\n    title: \"Empty Schema\"\n\n"
      created_date:                                          <computed>
      endpoint_configuration.#:                              "1"
      endpoint_configuration.0.types.#:                      "1"
      endpoint_configuration.0.types.0:                      "PRIVATE"
      endpoint_configuration.0.vpc_endpoint_ids.#:           "1"
      endpoint_configuration.0.vpc_endpoint_ids.1596079509:  "vpce-0efe2662207bd63a0"
      execution_arn:                                         <computed>
      minimum_compression_size:                              "-1"
      name:                                                  "Assure-Platform-Console-Service-private-1"
      policy:                                                "{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Deny\",\n            \"Principal\": \"*\",\n            \"Action\": \"execute-api:Invoke\",\n            \"Resource\": \"execute-api:/*\",\n            \"Condition\": {\n                \"StringNotEquals\": {\n                    \"aws:sourceVpce\": \"vpce-0efe2662207bd63a0\"\n                }\n            }\n        },\n        {\n            \"Effect\": \"Allow\",\n            \"Principal\": \"*\",\n            \"Action\": \"execute-api:Invoke\",\n            \"Resource\": \"execute-api:/*\"\n        }\n    ]\n}"
      root_resource_id:                                      <computed>

That VPCE ID has been confirmed to exist and be active at the time of the apply, and, when in the API Gateway console, is available for attachment to the API

On running terraform apply the second time (immediately after the first attempt that fails to set the policy and VPCE association), I get a diff for the VPCE and the policy.

  ~ module.main.module.console_service_api.module.api.aws_api_gateway_rest_api.service_api_private_vpce
      endpoint_configuration.0.vpc_endpoint_ids.#:           "0" => "1"
      endpoint_configuration.0.vpc_endpoint_ids.1596079509:  "" => "vpce-0efe2662207bd63a0"
      policy:                                                "" => "{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Effect\": \"Deny\",\n            \"Principal\": \"*\",\n            \"Action\": \"execute-api:Invoke\",\n            \"Resource\": \"execute-api:/*\",\n            \"Condition\": {\n                \"StringNotEquals\": {\n                    \"aws:sourceVpce\": \"vpce-0efe2662207bd63a0\"\n                }\n            }\n        },\n        {\n            \"Effect\": \"Allow\",\n            \"Principal\": \"*\",\n            \"Action\": \"execute-api:Invoke\",\n            \"Resource\": \"execute-api:/*\"\n        }\n    ]\n}"

Letting THAT run through, adds the VPCE association and the policy.

[bassmanitram]

This is VERY likely due to parameters being ignored, or overridden, if the body is provided - but my reasoning would be "I know the body YAML only provides the structure of the API and not the endpoint config requirements because the business team is building that. The operations team require APIs to have policies and be private, so I am asking TF to 'force' in other parameters (such as the policy and the VPCE config) by including them in the resource". If my reasoning rings true, importing the body and then patching the API with the policy and the VPCE config (and the binary types which also suffer from this issue) with any provided values would appear the most consistent solution.

[bassmanitram]

The other side of this is that if we do manage to the hack the body string that is passed to this module and add the x-amazon-apigateway-policy and x-amazon-apigateway-endpoint-configuration sections that are the equivalents of the policy and VPCE config properties of the TF resource, and then remove those properties from the TF resource, the second terraform apply will attempt to delete them. Rock and a hard place here.

[bassmanitram]

Looking at https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_api_gateway_rest_api.go I see what's happening - the API is created with all the specified properties, and then the body is imported, which overwrites the VPCE association, policy, and binary types. There isn't a different order to do that - you have to create it first, but a possible fix would be to create the API with no optional parameters, then import the body, then patch the provided (affected) parameters, the reasoning being that "whatever I specify in my TF resource configuration should override what's in the body". This would fix both issues (the non-setting when the parameters are specified, and the diff when the parameters are in the body).

[thenom]

Does anyone have a workaround until this is fixed as this mean that TF cannot be applied to a live service as it would cause an outage?

I am going to try the x-amazon-apigateway-endpoint-configuration option but it seems that this has been confirmed as a no go.

[thenom]

Yep, i now get no VPCE ID configured no matter how many times i run it with x-amazon-apigateway-endpoint-configuration set and this in TF:

  endpoint_configuration {
    types = ["PRIVATE"]
  }

i assume as there is no VPCE ID set here it is wiping it out.

[bassmanitram]

You can include the policy and vpc endpoint config in the OpenAPI YAML -
that fixes it.

On Tue, Nov 17, 2020 at 12:00 PM Simon Thorley [e-mail]
wrote:

Yep, i now get no VPCE ID configured no matter how many times i run it
with x-amazon-apigateway-endpoint-configuration set and this in TF:

endpoint_configuration {
types = ["PRIVATE"]
}

i assume as there is no VPCE ID set here it is wiping it out.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/terraform-provider-aws/issues/13841#issuecomment-728853086,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADAKLWQPOWFBDJ4UTKDHF4DSQJJUBANCNFSM4OCZWRKQ
.

[thenom]

Thanks @bassmanitram but thats what i tried above but i get no VPCE if i do it that way. Unless i am implementing this wrong:
https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-swagger-extensions-endpoint-configuration.html

i.e. OAS file:

{
  "openapi": "3.0.0",
  "x-amazon-apigateway-endpoint-configuration": {
    "vpcEndpointIds": ["vpce-0212a4ababd5b8c3e"]
  },
  ....

TF:
```
endpoint_configuration {
types = ["PRIVATE"]
}

[thenom]

Do i have to have both the policy AND the VPCEndpoint config in the OAS file for this to work?

[thenom]

It seems i missed this vital bit of documentation... :/

it should be present under the vendor extensions of the Server object.

Re-attempting

[thenom]

Ok, to confirm. Just the x-amazon-apigateway-endpoint-configuration on its own without the policy in the OAS body does not work. You need both in the body for this to work. As a bonus, TF does not keep re-writing the policy with the shortened execute-api resource name every time, i.e.:

~ Resource = "arn:aws:execute-api:eu-west-1:1234567890:a12b34c56/*/*/*" -> "execute-api:/*/*/*"

For future googlers, this works for me:

{
  "openapi": "3.0.0",
  "servers": [
    {
      "x-amazon-apigateway-endpoint-configuration": {
        "vpcEndpointIds": []
      }
    }
  ],
  "x-amazon-apigateway-policy": {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "execute-api:Invoke",
            "Resource": "execute-api:/*/*/*",
            "Condition": {
                "StringNotEquals": {
                    "aws:sourceVpc": []
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "execute-api:Invoke",
            "Resource": "execute-api:/*/*/*"
        }
    ]
  },

Then i just replace "vpcEndpointIds": [] and "aws:sourceVpc": [] at plan\apply time with the tfvars.

[bassmanitram]

Sorry - only just saw this ... but, yes, exactly :D

On Wed, Nov 18, 2020 at 1:17 PM Simon Thorley [e-mail]
wrote:

Ok, to confirm. Just the x-amazon-apigateway-endpoint-configuration on
its own without the policy in the OAS body does not work. You need both in
the body for this to work. As a bonus, TF does not keep re-writing the
policy with the shortened execute-api resource name every time, i.e.:

~ Resource = "arn:aws:execute-api:eu-west-1:1234567890:a12b34c56///" ->
"execute-api:///"

For future googlers, this works for me:

{
"openapi": "3.0.0",
"servers": [
{
"x-amazon-apigateway-endpoint-configuration": {
"vpcEndpointIds": []
}
}
],
"x-amazon-apigateway-policy": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Principal": "",
"Action": "execute-api:Invoke",
"Resource": "execute-api:///",
"Condition": {
"StringNotEquals": {
"aws:sourceVpc": []
}
}
},
{
"Effect": "Allow",
"Principal": "",
"Action": "execute-api:Invoke",
"Resource": "execute-api:///"
}
]
},

Then i just replace "vpcEndpointIds": [] and "aws:sourceVpc": [] at
plan\apply time with the tfvars.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/terraform-provider-aws/issues/13841#issuecomment-729640830,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADAKLWQDVK6IUTSXWRXDRALSQO3MVANCNFSM4OCZWRKQ
.

[thenom]

Almost, this seems to have fixed the VPCE issue but in resources now i am getting:

I am trying to find out why it is ignoring the policy extension i have applied.

[thenom]

This now works fine with the policy setup but i noticed that with removing this block:

  endpoint_configuration {
    types = ["PRIVATE"]
  }

It works fine with an existing API but when i was reattempted a redeploy it fails due to the fact that type private is not set anywhere. I have looked through the extensions docs and in x-amazon-apigateway-endpoint-configuration it just says that is needs it to be private:

It is only supported for REST APIs the PRIVATE endpoint type.

This seems to be what the TF block is for but there doesnt seem to be anyway to set this on an extension which i am assuming is because you need the API setup before you can import the OAS file, another rock and a hard place...

The only way i have around this is on initial deploy is to just re-add that block which breaks the VPCE setup and then remove it again and redeploy to fix the VPCE.