Terraform: aws-provider: "The input token is no longer current." when operating on multiple WAF objects

[keymon]

AWS WAF implements a token based system to ensure that no concurrent changes are applied on the WAF rules. The API expects the API client to take care of synchronisation.

If two or more threads/workers request a token at the same time, they might get the same token:

Each create, update, or delete request must use a unique change token. If your application submits a GetChangeToken request and then submits a second GetChangeToken request before submitting a create, update, or delete request, the second GetChangeToken request returns the same value as the first GetChangeToken request.

But only one can actually do one change, the token will be immediately marked as invalid, the other operations will fail with The input token is no longer current.

This is happening with terraform when handling multiple AWS WAF resources.

Terraform Version

0.7.10

Affected Resource(s)

aws-provider.aws_waf_*

Terraform Configuration Files

provider "aws" {
    region = "eu-west-1"
}
resource "aws_waf_ipset" "hector_waf_ipset_test" {
  count = 4
  name = "${format("hector_waf_ipset_test_%d", count.index)}"

  ip_set_descriptors {
    type  = "IPV4"
    value = "${format("193.0.7.%d/32", count.index)}"
  }
}

Debug Output

If you run the above code, it will create only one resource each time. The same for destroy:

keymon@keytocho:/tmp/test-tf2$ terraform apply
aws_waf_ipset.hector_waf_ipset_test.0: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2805378577.type:  "" => "IPV4"
  ip_set_descriptors.2805378577.value: "" => "192.0.7.0/32"
  name:                                "" => "hector_waf_ipset_test_0"
aws_waf_ipset.hector_waf_ipset_test.2: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.3723951473.type:  "" => "IPV4"
  ip_set_descriptors.3723951473.value: "" => "192.0.7.2/32"
  name:                                "" => "hector_waf_ipset_test_2"
aws_waf_ipset.hector_waf_ipset_test.1: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2589362081.type:  "" => "IPV4"
  ip_set_descriptors.2589362081.value: "" => "192.0.7.1/32"
  name:                                "" => "hector_waf_ipset_test_1"
aws_waf_ipset.hector_waf_ipset_test.3: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.3767981249.type:  "" => "IPV4"
  ip_set_descriptors.3767981249.value: "" => "192.0.7.3/32"
  name:                                "" => "hector_waf_ipset_test_3"
aws_waf_ipset.hector_waf_ipset_test.3: Creation complete
Error applying plan:

3 error(s) occurred:

* aws_waf_ipset.hector_waf_ipset_test.0: WAFStaleDataException: The input token is no longer current.
    status code: 400, request id: 4f3cc0fd-b22e-11e6-b7bd-1999591af44d
* aws_waf_ipset.hector_waf_ipset_test.1: WAFStaleDataException: The input token is no longer current.
    status code: 400, request id: 4f401c62-b22e-11e6-9046-29c04bbde95a
* aws_waf_ipset.hector_waf_ipset_test.2: WAFStaleDataException: The input token is no longer current.
    status code: 400, request id: 4f3d362e-b22e-11e6-b7bd-1999591af44d

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
$?=1 10:11:07 rb:2.2.1 py:paas-cf aws:gov-paas-dev
keymon@keytocho:/tmp/test-tf2$ terraform apply
aws_waf_ipset.hector_waf_ipset_test.3: Refreshing state... (ID: 68c042a9-5e8a-46a7-9b79-76a24cf6d7a0)
aws_waf_ipset.hector_waf_ipset_test.1: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2589362081.type:  "" => "IPV4"
  ip_set_descriptors.2589362081.value: "" => "192.0.7.1/32"
  name:                                "" => "hector_waf_ipset_test_1"
aws_waf_ipset.hector_waf_ipset_test.0: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2805378577.type:  "" => "IPV4"
  ip_set_descriptors.2805378578.value: "" => "192.0.7.0/32"
  name:                                "" => "hector_waf_ipset_test_0"
aws_waf_ipset.hector_waf_ipset_test.2: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.3723951473.type:  "" => "IPV4"
  ip_set_descriptors.3723951473.value: "" => "192.0.7.2/32"
  name:                                "" => "hector_waf_ipset_test_2"
aws_waf_ipset.hector_waf_ipset_test.2: Creation complete
Error applying plan:

2 error(s) occurred:

* aws_waf_ipset.hector_waf_ipset_test.0: WAFStaleDataException: The input token is no longer current.
    status code: 400, request id: 599956e5-b22e-11e6-9fce-255be3b03960
* aws_waf_ipset.hector_waf_ipset_test.1: WAFStaleDataException: The input token is no longer current.
    status code: 400, request id: 599c160c-b22e-11e6-9046-29c04bbde95a

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
$?=1 10:11:25 rb:2.2.1 py:paas-cf aws:gov-paas-dev
keymon@keytocho:/tmp/test-tf2$ terraform apply
aws_waf_ipset.hector_waf_ipset_test.2: Refreshing state... (ID: e1d5fbfe-5133-40be-876a-dbc95ad11f8a)
aws_waf_ipset.hector_waf_ipset_test.3: Refreshing state... (ID: 68c042a9-5e8a-46a7-9b79-76a24cf6d7a0)
aws_waf_ipset.hector_waf_ipset_test.0: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2805378577.type:  "" => "IPV4"
  ip_set_descriptors.2805378577.value: "" => "192.0.7.0/32"
  name:                                "" => "hector_waf_ipset_test_0"
aws_waf_ipset.hector_waf_ipset_test.1: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2589362081.type:  "" => "IPV4"
  ip_set_descriptors.2589362081.value: "" => "192.0.7.1/32"
  name:                                "" => "hector_waf_ipset_test_1"
aws_waf_ipset.hector_waf_ipset_test.0: Creation complete
Error applying plan:

1 error(s) occurred:

* aws_waf_ipset.hector_waf_ipset_test.1: WAFStaleDataException: The input token is no longer current.
    status code: 400, request id: 5eddc253-b22e-11e6-9046-29c04bbde95a

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
$?=1 10:11:34 rb:2.2.1 py:paas-cf aws:gov-paas-dev
keymon@keytocho:/tmp/test-tf2$ terraform apply
aws_waf_ipset.hector_waf_ipset_test.3: Refreshing state... (ID: 68c042a9-5e8a-46a7-9b79-76a24cf6d7a0)
aws_waf_ipset.hector_waf_ipset_test.2: Refreshing state... (ID: e1d5fbfe-5133-40be-876a-dbc95ad11f8a)
aws_waf_ipset.hector_waf_ipset_test.0: Refreshing state... (ID: aadd06d6-3171-4bd0-8c00-0e530f4a878e)
aws_waf_ipset.hector_waf_ipset_test.1: Creating...
  ip_set_descriptors.#:                "" => "1"
  ip_set_descriptors.2589362081.type:  "" => "IPV4"
  ip_set_descriptors.2589362081.value: "" => "192.0.7.1/32"
  name:                                "" => "hector_waf_ipset_test_1"
aws_waf_ipset.hector_waf_ipset_test.1: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Expected Behavior

Should create/update/delete all the resources sequentially without failing.

Actual Behavior

Only one change at a time is possible. More might be possible if they happen to run sequentially.

Steps to Reproduce

1. Configure multiple WAF resources that would run in parallel
2. terraform apply

Workaround

• Rerun terraform several times, it will create/update one object each time.
• Create dependencies between resources
• Use the option parallelism to limit the number of concurrent changes to 1: terraform apply -parallelism=1. But everything will be slower.

References

How WAF token works: http://docs.aws.amazon.com/waf/latest/APIReference/API_GetChangeToken.html

[stack72]

@optimisticanshul did you see this issue when you were testing?

Paul

[ventz]

Yep -- seeing the same thing.

[vicfl]

+1 still happening

[steve-jansen]

+1 also experiencing this issue with our environments. A 2nd apply seems to fix.

[sebolabs]

+1 same here

[kaaloo]

+1 on 0.8.8

[ventz]

@stack72 - Paul, it seems like this issue got lost -- would it be possible to assign someone on it. It's starting to gain a lot of traction, and the problem is very frustrating.

[sandyfox]

+1 on Terraform v0.9.3

[radeksimko]

The bugfix was merged and it will be released in 0.9.4. Thanks for the patience. 😅

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.