Terraform-provider-aws: global secondary index always recreated

[hashibot[bot]]

_This issue was originally opened by @umeat as hashicorp/terraform#17600. It was migrated here as a result of the provider split. The original body of the issue is below._

---

Terraform Version

Terraform v0.11.3

Terraform Configuration Files

resource "aws_dynamodb_table" "file-index-table" {
  name           = "${var.application}-file-index-${var.environment}"
  hash_key       = "StationId"
  range_key      = "QueryString"
  read_capacity  = 30
  write_capacity = 30

  attribute {
    name = "StationId"
    type = "S"
  }

  attribute {
    name = "QueryString"
    type = "S"
  }

  attribute {
    name = "EventId"
    type = "N"
  }

  attribute {
    name = "Position"
    type = "N"
  }

  global_secondary_index {
    name               = "EventIdIndex"
    hash_key           = "EventId"
    range_key          = "Position"
    write_capacity     = 20
    read_capacity      = 20
    projection_type    = "INCLUDE"
    non_key_attributes = ["StationId", "QueryString"]
  }

  tags {
    Name        = "File-Index-Table"
    Environment = "${var.environment}"
  }
}

Behavior

The global secondary index is destroyed and recreated when no changes have been made to it.

module.file-index.aws_dynamodb_table.file-index-table: Modifying... (ID: archive-file-index-dev)
  global_secondary_index.593012552.hash_key:             "" => "EventId"
  global_secondary_index.593012552.name:                 "" => "EventIdIndex"
  global_secondary_index.593012552.non_key_attributes.#: "0" => "2"
  global_secondary_index.593012552.non_key_attributes.0: "" => "StationId"
  global_secondary_index.593012552.non_key_attributes.1: "" => "QueryString"
  global_secondary_index.593012552.projection_type:      "" => "INCLUDE"
  global_secondary_index.593012552.range_key:            "" => "Position"
  global_secondary_index.593012552.read_capacity:        "" => "20"
  global_secondary_index.593012552.write_capacity:       "" => "20"
  global_secondary_index.611285021.hash_key:             "EventId" => ""
  global_secondary_index.611285021.name:                 "EventIdIndex" => ""
  global_secondary_index.611285021.non_key_attributes.#: "2" => "0"
  global_secondary_index.611285021.non_key_attributes.0: "QueryString" => ""
  global_secondary_index.611285021.non_key_attributes.1: "StationId" => ""
  global_secondary_index.611285021.projection_type:      "INCLUDE" => ""
  global_secondary_index.611285021.range_key:            "Position" => ""
  global_secondary_index.611285021.read_capacity:        "20" => "0"
  global_secondary_index.611285021.write_capacity:       "20" => "0"

[jmettes]

+1

[miesvanderrobot]

Seeing same behavior here on Terraform 0.10.7

[laurams]

+1

[pgieser]

Still seeing this behavior on 0.11.7

[ctippur]

Same here .. Observed this in Terraform v0.11.7.

[alekbarszczewski]

Same issue here - is someone planning to fix this?

[ctippur]

Yes. It is causing quite a few inconsistencies. Terraform times out creating indexes. :(

[alekbarszczewski]

Guys are you planning to fix this?

[ctippur]

Any updates on this please?

[stevebreault]

If you look at the output above, you'll notice that the order of the non_key_attributes is different in the second half of the block.

module.file-index.aws_dynamodb_table.file-index-table: Modifying... (ID: archive-file-index-dev)
global_secondary_index.593012552.hash_key:             "" => "EventId"
global_secondary_index.593012552.name:                 "" => "EventIdIndex"
global_secondary_index.593012552.non_key_attributes.#: "0" => "2"
global_secondary_index.593012552.non_key_attributes.0: "" => "StationId"          <-- 1
global_secondary_index.593012552.non_key_attributes.1: "" => "QueryString"        <-- 2
global_secondary_index.593012552.projection_type:      "" => "INCLUDE"
global_secondary_index.593012552.range_key:            "" => "Position"
global_secondary_index.593012552.read_capacity:        "" => "20"
global_secondary_index.593012552.write_capacity:       "" => "20"
global_secondary_index.611285021.hash_key:             "EventId" => ""
global_secondary_index.611285021.name:                 "EventIdIndex" => ""
global_secondary_index.611285021.non_key_attributes.#: "2" => "0"
global_secondary_index.611285021.non_key_attributes.0: "QueryString" => ""        <-- 2
global_secondary_index.611285021.non_key_attributes.1: "StationId" => ""          <-- 1
global_secondary_index.611285021.projection_type:      "INCLUDE" => ""
global_secondary_index.611285021.range_key:            "Position" => ""
global_secondary_index.611285021.read_capacity:        "20" => "0"
global_secondary_index.611285021.write_capacity:       "20" => "0"

If you change the order in your configuration file to match the order in the later part of the block, it will no longer try to recreate. That's what fixed it for me anyhow.

[davehowell]

@stevebreault nice spotting. Clearly the state file is storing sorted, and using that to compare against the unsorted list specified in the HIL.

I can add that it also does this for the aws_dynamodb_table_item resource that I am using to load a few initial values to a table. If you don't specify item argument with sorted attributes it thinks there is a change and updates to the exact same values. Very annoying when you have dynamodb streams or something similar triggered from the table.

The ordering of the attributes is not really important given all dynambodb values are named kv pairs, and the api responses give you json with named fields, so storing it sorted in the state file is fine, as long as the comparison is also done between sorted existing-state and sorted new-state.

[bam0382]

Related to https://github.com/terraform-providers/terraform-provider-aws/issues/3807

[ioancatana]

I have the same issue, on 3 global sec indexes. I didn't manage to edit the state file, so I put lifecycle on each table:
lifecycle {
ignore_changes = ["read_capacity", "write_capacity", "global_secondary_index"]
}

[red8888]

Still an issue. How is it sorting? Alphabetical? Be nice if I knew ahead of time so I could order the array instead of having to apply, then check order, then change config until I confirm the order is right

[ajfisher]

This still appears to be an issue. Ran into this today as well. Changed the order and it appears fine.

If I had to guess, it's not alphabetical, but appears to be the order by which the attributes are defined for the table? (Mine aren't defined in alphabetical order so it is not that). I have a table with 5 attributes and three GSIs and the only pattern I can notice is this as each of my indexes use different combinations of attributes.

[ghost]

Still occurs in 0.12.5. It seems to me after a brief glance at the code (structure.go, diffDynamoDbGSI function) that this is caused by billing_mode "on demand" where you don't have to specify capacities.

When they are not specified in the terraform resource, while diffing the read DynamoDb table, AWS returns 0 as read and write capacity for each global secondary index, while in terraform it is defined as null. I'm not really proficient at go right now, but it seems like this could also be a reason for the indexes updating every time.

After having added read/write capacities = 0 to my resource definition and reordered indexes and index attributes, terraform now doesn't update the table when there's no change.

[shalka]

Can I get additional reviewers and/or upvotes for this PR?
https://github.com/terraform-providers/terraform-provider-aws/pull/9988

[vishwanatham]

As mentioned by @stevebreault we have to tweak the order of the "non_key_attributes". Regarding the order, we have to apply once and then look at the order the keys are presented in the aws console (DynamoDB > Table > Indexes). And if we adjust the order according to that in the terraform code, it won't reapply every time.

[DOCS-kdu3]

Same behavior here on Terraform 0.12.23

[digitalfiz]

And on terraform 0.12.25 and aws provider version 2.62

[yannh]

Been using a fork with @shalka 's patch for several months because of this issue, seems :+1:

[twnbay78]

This issue is still prevalent in version 0.12.25. Forking is not an option for everyone. When can we expect @shalka 's patch to be approved and merged? It has been several months now since a fix has been made and it still has not been merged.

[cakejelly]

@ewbankkit @bflad apologies for the direct mentions but this issue hasn't received a response from anybody on the team and it has been open for over 2 years (as well as https://github.com/terraform-providers/terraform-provider-aws/issues/3807).

This issue has led us (and probably many others) to run a fork with the patch from @shalka https://github.com/terraform-providers/terraform-provider-aws/pull/9988, which is not a very nice solution and that PR has also been open for almost a year without any response. The patch works and we've been using it for several months now so could we please merge it or at least get an update on when it could potentially be merged?

Thanks

[gurre]

Using Terraform v0.12.26 + provider.aws v2.66.0 I'm able to workaround this issue by using the same ordering as returned by the API for non_key_attributes. It appears to be the sorting that is the diff.

# change
non_key_attributes = ["StationId", "QueryString"]
# to 
non_key_attributes = ["QueryString", "StationId"]

[cakejelly]

^ can confirm this works for me too with Terraform v0.11.14 and provider.aws v2.70.0

Get the API ordering for your indexes with aws dynamodb describe-table --table-name <table_name> | jq '.Table.GlobalSecondaryIndexes' and order the non_key_attributes in your config the same way and it should fix it.

[maryelizbeth]

Hi Y'all, we are currently tracking this issue via #671.

We've added #671 to our team's prioritization board and are planning to address it and its related PRs in a near-term release. As this issue is a duplicate, we'll close this and ask that y'all continue discussion on #671. If you believe this has been closed in error or that #671 does not adequately address the issue, please open a new issue.

Thanks!

[gurre]

671 is a workaround not a fix. If you would like to change the non_key_attributes they cannot be ignored at the same time. So for any changes to the GSI you would need to remove the ignore_changes = ["global_secondary_index"], apply the change then put it into ignore again. Not so elegant.

I've noticed that the sorting differs between AWS regions which makes any solution quite challenging to implement. I guess a workaround is resonable due to this.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!