Terraform-provider-aws: `aws_api_gateway_stage` and `aws_api_gateway_method_settings` are not consistently handled

[mikaelrandy]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

$ terraform -v
Terraform v0.11.8

Provider version : 1.34.0

Affected Resource(s)

• aws_api_gateway_deployment
• aws_api_gateway_stage
• aws_api_gateway_method_settings

Global explaination

I want to handle some API Gateway configurations with Terraform.
Theses configurations are available through aws_api_gateway_stage and aws_api_gateway_method_settings resources.

But there are 2 issues using theses resources :

• aws_api_gateway_stage cannot be create in a whole API Gateway Terraform configuration (need to be imported)
• aws_api_gateway_method_settings and aws_api_gateway_stage configuration are dropped at each aws_api_gateway_deployment application.

Terraform Configuration Files

provider "aws" {
  region  = "eu-west-1"
  profile = "6cloud-dev"
  version = "~> 1.33"
}

resource "aws_api_gateway_rest_api" "test" {
  name        = "api_name"
  description = "Some API"
}

resource "aws_api_gateway_deployment" "test" {
  depends_on = [
    "aws_api_gateway_method.test_get",
  ]

  rest_api_id = "${aws_api_gateway_rest_api.test.id}"
  stage_name  = "test"
}

resource "aws_api_gateway_stage" "time" {
  rest_api_id   = "${aws_api_gateway_rest_api.test.id}"
  deployment_id = "${aws_api_gateway_deployment.test.id}"

  stage_name = "${aws_api_gateway_deployment.test.stage_name}"

  tags {
    project = "ProjectName"
  }
}

resource "aws_api_gateway_method_settings" "test" {
  rest_api_id = "${aws_api_gateway_rest_api.test.id}"
  stage_name  = "${aws_api_gateway_deployment.test.stage_name}"
  method_path = "*/*"

  settings {
    throttling_rate_limit = 1000
  }
}

resource "aws_api_gateway_resource" "test" {
  rest_api_id = "${aws_api_gateway_rest_api.test.id}"
  parent_id   = "${aws_api_gateway_rest_api.test.root_resource_id}"
  path_part   = "time"
}

resource "aws_api_gateway_method" "test_get" {
  rest_api_id   = "${aws_api_gateway_rest_api.test.id}"
  resource_id   = "${aws_api_gateway_resource.test.id}"
  http_method   = "GET"
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "test" {
  rest_api_id = "${aws_api_gateway_rest_api.test.id}"
  resource_id = "${aws_api_gateway_resource.test.id}"
  http_method = "${aws_api_gateway_method.test_get.http_method}"
  type        = "MOCK"
}

Debug Output

Complete gist : https://gist.github.com/mikaelrandy/a8c13555c7790742a03d865165d9f93a

• First terraform apply execution : terraform_apigateway_stage_creation-tf
• Second terraform apply, after importing "aws_api_gateway_stage.time" : terraform_apigateway_stage_update-tf
• Third terraform apply, after taint "aws_api_gateway_deployment.test" : terraform_apigateway_deployment_update-tf
• Last terrafor apply, to redeploy "aws_api_gateway_stage.time" : terraform_apigateway_deployment_update_stage-tf

Expected Behavior

• At first deploy, all resources are deployed in AWS and correctly linked in tfstate
• At update, no configuration are lost

Actual Behavior

At first deploy

Execution fail on stage deployment. Seems that API Gateway stage is already created by another resource, but not linked.

terraform apply -auto-approve
aws_api_gateway_rest_api.test: Creating...
  api_key_source:           "" => "HEADER"
  created_date:             "" => "<computed>"
  description:              "" => "Some API"
  endpoint_configuration.#: "" => "<computed>"
  execution_arn:            "" => "<computed>"
  minimum_compression_size: "" => "-1"
  name:                     "" => "api_name"
  root_resource_id:         "" => "<computed>"
aws_api_gateway_rest_api.test: Creation complete after 0s (ID: wosvgb26dh)
aws_api_gateway_resource.test: Creating...
  parent_id:   "" => "a2f0vzet6l"
  path:        "" => "<computed>"
  path_part:   "" => "test"
  rest_api_id: "" => "wosvgb26dh"
aws_api_gateway_resource.test: Creation complete after 1s (ID: 8u42yk)
aws_api_gateway_method.test_get: Creating...
  api_key_required: "" => "false"
  authorization:    "" => "NONE"
  http_method:      "" => "GET"
  resource_id:      "" => "8u42yk"
  rest_api_id:      "" => "wosvgb26dh"
aws_api_gateway_method.test_get: Creation complete after 0s (ID: agm-wosvgb26dh-8u42yk-GET)
aws_api_gateway_deployment.test: Creating...
  created_date:  "" => "<computed>"
  execution_arn: "" => "<computed>"
  invoke_url:    "" => "<computed>"
  rest_api_id:   "" => "wosvgb26dh"
  stage_name:    "" => "test"
aws_api_gateway_integration.test: Creating...
  cache_namespace:      "" => "<computed>"
  connection_type:      "" => "INTERNET"
  http_method:          "" => "GET"
  passthrough_behavior: "" => "<computed>"
  resource_id:          "" => "8u42yk"
  rest_api_id:          "" => "wosvgb26dh"
  timeout_milliseconds: "" => "29000"
  type:                 "" => "MOCK"
aws_api_gateway_integration.test: Creation complete after 0s (ID: agi-wosvgb26dh-8u42yk-GET)
aws_api_gateway_deployment.test: Creation complete after 0s (ID: qhgl0a)
aws_api_gateway_stage.time: Creating...
  deployment_id: "" => "qhgl0a"
  execution_arn: "" => "<computed>"
  invoke_url:    "" => "<computed>"
  rest_api_id:   "" => "wosvgb26dh"
  stage_name:    "" => "test"
  tags.%:        "" => "1"
  tags.project:  "" => "ProjectName"
aws_api_gateway_method_settings.test: Creating...
  method_path:                      "" => "*/*"
  rest_api_id:                      "" => "wosvgb26dh"
  settings.#:                       "" => "1"
  settings.0.throttling_rate_limit: "" => "1000"
  stage_name:                       "" => "test"
aws_api_gateway_method_settings.test: Creation complete after 0s (ID: wosvgb26dh-test-*/*)

Error: Error applying plan:

1 error(s) occurred:

* aws_api_gateway_stage.time: 1 error(s) occurred:

* aws_api_gateway_stage.time: Error creating API Gateway Stage: ConflictException: Stage already exists
    status code: 409, request id: a9c1d588-b043-11e8-9a76-5fb0898636e5

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Need to import stage and re-apply to succeed.

$ terraform import aws_api_gateway_stage.time wosvgb26dh/test
aws_api_gateway_stage.time: Importing from ID "wosvgb26dh/test"...
aws_api_gateway_stage.time: Import complete!
  Imported aws_api_gateway_stage (ID: ags-wosvgb26dh-test)
aws_api_gateway_stage.time: Refreshing state... (ID: ags-wosvgb26dh-test)

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

$ terraform apply -auto-approve
aws_api_gateway_rest_api.test: Refreshing state... (ID: wosvgb26dh)
aws_api_gateway_resource.test: Refreshing state... (ID: 8u42yk)
aws_api_gateway_method.test_get: Refreshing state... (ID: agm-wosvgb26dh-8u42yk-GET)
aws_api_gateway_integration.test: Refreshing state... (ID: agi-wosvgb26dh-8u42yk-GET)
aws_api_gateway_deployment.test: Refreshing state... (ID: qhgl0a)
aws_api_gateway_stage.time: Refreshing state... (ID: ags-wosvgb26dh-test)
aws_api_gateway_method_settings.test: Refreshing state... (ID: wosvgb26dh-test-*/*)
aws_api_gateway_stage.time: Modifying... (ID: ags-wosvgb26dh-test)
  tags.%:       "0" => "1"
  tags.project: "" => "ProjectName"
aws_api_gateway_stage.time: Modifications complete after 1s (ID: ags-wosvgb26dh-test)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

At update

First apply after aws_api_gateway_deployment update lead to remove aws_api_gateway_stage and aws_api_gateway_method_settings configuration.
Need to re-apply to re-create it.

$ terraform taint aws_api_gateway_deployment.test
The resource aws_api_gateway_deployment.test in the module root has been marked as tainted!

$ terraform apply -auto-approve
aws_api_gateway_rest_api.test: Refreshing state... (ID: wosvgb26dh)
aws_api_gateway_resource.test: Refreshing state... (ID: 8u42yk)
aws_api_gateway_method.test_get: Refreshing state... (ID: agm-wosvgb26dh-8u42yk-GET)
aws_api_gateway_deployment.test: Refreshing state... (ID: qhgl0a)
aws_api_gateway_integration.test: Refreshing state... (ID: agi-wosvgb26dh-8u42yk-GET)
aws_api_gateway_method_settings.test: Refreshing state... (ID: wosvgb26dh-test-*/*)
aws_api_gateway_stage.time: Refreshing state... (ID: ags-wosvgb26dh-test)
aws_api_gateway_deployment.test: Destroying... (ID: qhgl0a)
aws_api_gateway_deployment.test: Destruction complete after 0s
aws_api_gateway_deployment.test: Creating...
  created_date:  "" => "<computed>"
  execution_arn: "" => "<computed>"
  invoke_url:    "" => "<computed>"
  rest_api_id:   "" => "wosvgb26dh"
  stage_name:    "" => "test"
aws_api_gateway_deployment.test: Creation complete after 1s (ID: v7ue28)
aws_api_gateway_stage.time: Modifying... (ID: ags-wosvgb26dh-test)
  deployment_id: "qhgl0a" => "v7ue28"
aws_api_gateway_stage.time: Modifications complete after 0s (ID: ags-wosvgb26dh-test)

Apply complete! Resources: 1 added, 1 changed, 1 destroyed.

Add this point, the tag in the stage on AWS console is removed
And a re-apply make updates on "aws_api_gateway_stage"

terraform apply -auto-approve
aws_api_gateway_rest_api.test: Refreshing state... (ID: wosvgb26dh)
aws_api_gateway_resource.test: Refreshing state... (ID: 8u42yk)
aws_api_gateway_method.test_get: Refreshing state... (ID: agm-wosvgb26dh-8u42yk-GET)
aws_api_gateway_integration.test: Refreshing state... (ID: agi-wosvgb26dh-8u42yk-GET)
aws_api_gateway_deployment.test: Refreshing state... (ID: v7ue28)
aws_api_gateway_stage.time: Refreshing state... (ID: ags-wosvgb26dh-test)
aws_api_gateway_method_settings.test: Refreshing state... (ID: wosvgb26dh-test-*/*)
aws_api_gateway_stage.time: Modifying... (ID: ags-wosvgb26dh-test)
  tags.%:       "0" => "1"
  tags.project: "" => "ProjectName"
aws_api_gateway_method_settings.test: Creating...
  method_path:                      "" => "*/*"
  rest_api_id:                      "" => "wosvgb26dh"
  settings.#:                       "" => "1"
  settings.0.throttling_rate_limit: "" => "1000"
  stage_name:                       "" => "test"
aws_api_gateway_method_settings.test: Creation complete after 0s (ID: wosvgb26dh-test-*/*)
aws_api_gateway_stage.time: Modifications complete after 0s (ID: ags-wosvgb26dh-test)

Apply complete! Resources: 1 added, 1 changed, 0 destroyed.

Steps to Reproduce

1. terraform apply
2. terraform import aws_api_gateway_stage.time [id]/test
3. terraform apply
4. terraform taint aws_api_gateway_deployment.test
5. terraform apply
6. terraform apply

[fsoikin]

Is there a workaround currently?

[jurajseffer]

Not sure if it makes a difference here but deployment should depend on aws_api_gateway_integration not method.

[jurajseffer]

It looks like aws_api_gateway_deployment should interpolate aws_api_gateway_stage.stage_name because deployment auto creates a stage to which it should be deploying if it doesn't exist. Create stage, first, then deployment.

Edit: I see now this is impossible indeed because stage needs a deployment. So it would seem the only way is to manually import a stage resource if one needs it at all. I just use deployment to create a stage.

[mattbarry]

@jurajseffer how would you set aws_api_gateway_stage.xray_tracing_enabled = true on a stage if you use deployment to create it?

[jurajseffer]

@mattbarry I guess you can't without importing it. It seems to me that this is an AWS limitation - the logic of their resources is weird - stage requires a deployment and deployment requires a stage or auto creates it.

[mattbarry]

@jurajseffer for what it's worth, I did (somehow) get it working yesterday. AWS seems to be able to make sense of this:

resource "aws_api_gateway_stage" "stage" {
  stage_name    = "${aws_api_gateway_deployment.deployment.stage_name}"
  rest_api_id   = "${aws_api_gateway_rest_api.api.id}"
  deployment_id = "${aws_api_gateway_deployment.deployment.id}"
  xray_tracing_enabled = true
}

resource "aws_api_gateway_deployment" "deployment" {
  rest_api_id = "${aws_api_gateway_rest_api.api.id}"
  stage_name  = "${local.stage_name}"
}

resource "aws_api_gateway_method_settings" "settings" {
  rest_api_id = "${aws_api_gateway_rest_api.api.id}"
  stage_name  = "${aws_api_gateway_deployment.deployment.stage_name}"
  method_path = "*/*"
  depends_on = ["aws_api_gateway_stage.stage", "aws_api_gateway_deployment.deployment"]
}

[jurajseffer]

@mattbarry that's interesting because I believe I had a conflict of "stage already exists" when both deployment and stage were used.

[mattbarry]

@jurajseffer I did too, but this seems to be working. When I get some time I'll try to figure out why.

[hs-rendov]

guys, any news about this issue - seems there's no any workaround for now. empty stage_name isn't working using v2.23.0 -- getting "Error creating API Gateway Stage: BadRequestException: Stage name must be non-empty" or "Error creating API Gateway Stage: ConflictException: Stage already exists" if it is set.

[nogweii]

This is still happening, and is exacerbated when trying to fix #162 (for now, I'm using timestamp() in the description) -- now every apply is creating a new stage and thus losing the settings. I could run it again...but then that would create another new stage.

[KenBerg75]

The way we have been handling this is by using an origin stage for the deployment which is only used to stage the deployment object. We then create the actual stages we would use based on that deployment.

# The un-used stage simply exists to create the deployment
resource "aws_api_gateway_deployment" "rest_api_deployment" {
  rest_api_id       = aws_api_gateway_rest_api.rest_api_definition.id
  stage_name        = "origin"
  stage_description = "This stage is used to mount the deployment before rolling it to the active stages"

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_stage" "rest_api_dev_stage" {
  stage_name    = "DEV"
  rest_api_id   = aws_api_gateway_rest_api.rest_api_definition.id
  deployment_id = aws_api_gateway_deployment.rest_api_deployment.id
  xray_tracing_enabled = true

  variables = {
       ...
  }
}

resource "aws_api_gateway_stage" "rest_api_prd_stage" {
  stage_name    = "PRD"
  rest_api_id   = aws_api_gateway_rest_api.rest_api_definition.id
  deployment_id = aws_api_gateway_deployment.rest_api_deployment.id
  xray_tracing_enabled = true

  variables = {
       ...
  }
}

Not sure if thats the right way, but it works for us.