Terragrunt: lock_table error when upgrading from terraform 0.12.29 to 0.13 with terragrunt 0.23.33

[edgan]

I tried to upgrade from terraform 0.12.29 to 0.13, and got a unsupported attribute "lock_table" error. Downgrading back to 0.12.29 worked after another terraform init.

[user@host us-west-2](master|✚2…) terragrunt apply
[terragrunt] [/home/user/git/terraform/production/us-west-2] 2020/08/19 14:29:50 Running command: terraform --version
[terragrunt] 2020/08/19 14:29:51 Terraform version: 0.13.0
[terragrunt] 2020/08/19 14:29:51 Reading Terragrunt config file at /home/user/git/terraform/production/us-west-2/terragrunt.hcl
[terragrunt] 2020/08/19 14:29:51 Running command: terraform apply -var-file=region.tfvars -var-file=/home/user/git/terraform/production/account.tfvars -var-file=/home/user/git/terraform/global.tfvars -lock-timeout=20m
Backend reinitialization required. Please run "terraform init".
Reason: Backend configuration changed for "s3"

The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.

Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.

If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.


Error: Initialization required. Please see the error message above.


[terragrunt] 2020/08/19 14:29:52 Hit multiple errors:
exit status 1
[user@host us-west-2](master|✚2…) terragrunt init
[terragrunt] [/home/user/git/terraform/production/us-west-2] 2020/08/19 14:29:57 Running command: terraform --version
[terragrunt] 2020/08/19 14:29:57 Terraform version: 0.13.0
[terragrunt] 2020/08/19 14:29:57 Reading Terragrunt config file at /home/user/git/terraform/production/us-west-2/terragrunt.hcl
[terragrunt] 2020/08/19 14:29:58 Running command: terraform init -backend-config=bucket=oe-prod-tfstate -backend-config=dynamodb_table=oe-prod-tflock -backend-config=encrypt=true -backend-config=key=vpc/terraform.tfstate -backend-config=profile=production -backend-config=region=us-west-2
Initializing modules...

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.



Error: Failed to decode current backend config

The backend configuration created by the most recent run of "terraform init"
could not be decoded: unsupported attribute "lock_table". The configuration
may have been initialized by an earlier version that used an incompatible
configuration structure. Run "terraform init -reconfigure" to force
re-initialization of the backend.

[terragrunt] 2020/08/19 14:29:59 Hit multiple errors:
exit status 1

[yorinasub17]

Thanks for reporting the bug! It looks like the lock_table attribute was renamed to dynamodb_table. We need to update the remote state logic to handle that in terragrunt.

[ahaffar]

i am getting the same error while upgrading from .12.25 to 0.13.0, i checked the local .terraform/terraform.tfstate and it has both dynamo_table and lock_table attributes

{
    "version": 3,
    "serial": 2,
    "lineage": "a7701610-8a94-ad84-502d-c5a0b43a225c",
    "backend": {
        "type": "s3",
        "config": {
            "access_key": null,
            "acl": null,
            "assume_role_policy": null,
            "bucket": "xxxxxxxxxxxx",
            "dynamodb_endpoint": null,
            "dynamodb_table": null,
            "encrypt": null,
            "endpoint": null,
            "external_id": null,
            "force_path_style": null,
            "iam_endpoint": null,
            "key": "xxxxxxxxxxxx",
            "kms_key_id": null,
            "lock_table": null,
            "max_retries": null,
            "profile": "xxxxxxxxx",
            "region": "us-east-1",
            "role_arn": null,
            "secret_key": null,
            "session_name": null,
            "shared_credentials_file": "~/.aws/credentials",
            "skip_credentials_validation": null,
            "skip_get_ec2_platforms": null,
            "skip_metadata_api_check": null,
            "skip_region_validation": null,
            "skip_requesting_account_id": null,
            "sse_customer_key": null,
            "sts_endpoint": null,
            "token": null,
            "workspace_key_prefix": null
        },
        "hash": 2120811086
    },
    "modules": [
        {
            "path": [
                "root"
            ],
            "outputs": {},
            "resources": {},
            "depends_on": []
        }
    ]
}

[ahaffar]

I solved this

• modifying the /.terraform/terraform.tfstate manually and remove the lines which the terraform_0_13 init reports such below lines
  "lock_table": null, "skip_get_ec2_platforms": null, "skip_requesting_account_id": null,

• Run terraform apply with the current version such as 0.12.25
• Change the required_versions in terraform block to be 0.13.0
• run terraform init -- terraform 0.13.0

hacky way 😜

[kwilczynski]

Hi @ahaffar, it's good to know that you were able to solve the problem! 🎉

To add more colour to what @yorinasub17 said.

The lock_table attribute was originally deprecated in the Terraform circa three years ago prior to the release of version 0.10 (as per the following Pull Request: hashicorp/terraform#14949), and then recently (see corresponding CHANGELOG.md entry and the _BREAKING CHANGES_ section for more details) finally retired leaving only the dynamodb_table to be used going forward (0.13+) (see Pull Request hashicorp/terraform#25134).

@yorinasub17, would you be open to adding a deprecation message when someone is using lock_table and then a little bit of code to simply always migrate the value from the lock_table attribute (when it was used) to dynamodb_table regardless of the version of Terraform?

I believe, this simple change would cover both cases, especially since both of the attributes are supported in the Terragrunt (see master/remote/remote_state_s3.go) and in the Terraform (and the deprecated attribute name even up to _three years_ back, so that users with older Terraform version would be covered - unless someone is still using Terraform older than version 0.10, albeit that would be quite problematic these days).

Terragrunt could simply always use the new attribute - this wouldn't require any special logic to detect which attribute has to be used based on the actual version of Terraform being used.

What do you think?

Update:

A comparison between state files originating from latest release of Terraform 0.12 and 0.13 showcases the removed of unused "lock_table" property (used to be NULL before removal):

[yorinasub17]

@kwilczynski Thanks for the detailed explanation!

would you be open to adding a deprecation message when someone is using lock_table and then a little bit of code to simply always migrate the value from the lock_table attribute (when it was used) to dynamodb_table regardless of the version of Terraform?

That sounds like a great idea! Would you be up for a quick PR to add this in? If not, we can add it to our backlog of items to handle.

[kwilczynski]

Hi @yorinasub17,

I would be delighted. Leave it with me. I will send a Pull Request later today.

[kwilczynski]

~Work in progress~Work completed: https://github.com/gruntwork-io/terragrunt/pull/1327

[kwilczynski]

Hi @edgan and @ahaffar,

A small Pull Request has been merged today to address the use of lock_table within Terragrunt which should allow Terraform release 0.12 and 0.13 to just work, as the new attribute dynamodb_table would be used instead. In other words, this fixes an issue within Terragrunt. Having said that, users who are upgrading from release 0.12 _to_ 0.13 would still need to complete the migration processes as per Terraform's request to run terraform init -reconfigure which would bring the local and remote state to be compliant with release 0.13 and newer.

I hope this will work for you going forward. 😄

[kwilczynski]

Hi @yorinasub17, I believe this issue can now be closed.