Terraform: State format upgrade failed

Created on 24 May 2019  ·  41Comments  ·  Source: hashicorp/terraform

I'm seeing the output below when running a plan with v0.12 for the first time (having upgraded from v0.11.9)

Terraform Version

0.12

Debug Output

I haven't included this as it showed nothing relevant

Expected Behavior

Either the state file upgrade should have succeeded or more detail should have been given in the error message, particularly a clue about where in the state file the failures occurred.

Actual Behavior

terraform plan -var-file=params/nonprod/params.tfvars -var-file=params/nonprod/secrets.tfvars

Error: Error loading state: State format upgrade failed: Error upgrading state file format from version 3 to version 4: 2 problems:

- Invalid character: This character is not used within the language.
- Attribute name required: Dot must be followed by attribute name..

How to go about troubleshooting Invalid character and Dot must be followed by attribute name when I can't see where in the state file to look?

bug core v0.12

Most helpful comment

Hi all!

We've just merged #22136 to add some additional context to these messages about what exactly Terraform was trying to do when it encountered these problems. Once that's released, we can hopefully collect some new versions of these error messages and finally track down which cases are not being handled by the state upgrade mechanism today.

The newer error messages will appear in the next release of Terraform.

All 41 comments

Hi @mf-lit,

It sounds like your state file includes some addresses in depends_on that the state upgrade logic isn't correctly upgrading. The 0.12 state format uses resource address syntax for depends_on, while 0.11 and prior used a special internal representation.

While I do agree that the error message is not helpful here, I'd like to start by figuring out what exactly the problem was, so that we can help you make progress and then know which error case needs improving. If you could take a look in your state file and share the addresses you find in depends_on properties -- particularly if any of them contain unusual punctuation characters rather than just attribute names separated by periods -- hopefully we can spot one that is in a format we didn't consider.

Thanks, and sorry for this oversight in the upgrade logic.

I was experiencing:

Error: Error loading state: 2 problems:

- Attribute name required: Dot must be followed by attribute name.
- Attribute name required: Dot must be followed by attribute name.

Upon manually deleting the follow two resources in the state, the error cleared up.

"aws_iam_role_policy_attachment.7shifts": {
        "type": "aws_iam_role_policy_attachment",
        "depends_on": [
                "aws_iam_role.app",
                "data.aws_iam_policy.7shifts"
        ],
        "primary": {
            ...
        },
        "deposed": [],
        "provider": "provider.aws"
},
"aws_iam_role_policy_attachment.7shifts_celery": {
        "type": "aws_iam_role_policy_attachment",
        "depends_on": [
                "aws_iam_role.celery",
                "data.aws_iam_policy.7shifts"
        ],
        "primary": {
            ...
        },
        "deposed": [],
        "provider": "provider.aws"
},

Hi @luhn! Thanks for sharing that example.

In your case, I think the problem is that you had a resource name starting with a digit. Terraform 0.12 can't parse that because it's ambiguous with a number token. This is one of the pre-upgrade steps that the 0.12checklist command should've warned you about before you upgraded, giving you an opportunity to rename it before upgrading.

Since your state hasn't been upgraded yet, you should still be able to do the steps that the checklist command would've recommended (using Terraform 0.11.14, _not_ Terraform 0.12.0):

  • Rename the data.aws_iam_policy.7shifts resource so that its name doesn't start with a digit.
  • Use terraform state mv to ensure that the state object is also renamed to match.

With that done, you should then be able to upgrade to Terraform 0.12.0 and have the state upgrade work correctly. (That is, unless terraform 0.12checklist on Terraform 0.11.14 identifies other things to fix, in which case I'd definitely recommend addressing those first too.)

Hi @apparentlymart,

Apologies that it's taken a few days to get back to you, holiday weekend here in the UK.

So I've eyeballed all the depends_on in my state file and don't really see anything that looks likely, other than a bit of camel case (e.g. aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy) and some wildcarding (e.g. aws_subnet.public.* )

Here is an extraction of all the resources.depends_on from the state file (with all the empty ones removed) in case it's of use to you:
https://gist.github.com/mf-lit/5b1397556c11291f7cdbd8bf219d6e24

Also, following Luhn's example I do see multiple (over 30) examples in the state file of resources that are named by their count. For example:

"aws_route_table_association.private.2": {
"aws_efs_mount_target.default.1": {

These aren't being flagged by terraform 0.12checklist however.

Thanks for sharing that, @mf-lit!

From reviewing your list, I wonder if Terraform is failing to correctly upgrade the ones that end in .*. I think I remember there being a special rule to deal with that in the state upgrade logic, but I'm not 100% sure. I too am away from things right now, so I won't be able to dig into that immediately here, but I'll take a closer look soon.

The cases where a resource instance itself has a trailing index followed by a period are not a problem here: that's the 0.11 way of saying aws_route_table_association.private[2], and that will be getting upgraded to the new standard address syntax. (There's a _lot_ of examples of that situation in our tests, so I'm much more confident that this part is working correctly.)

Just adding another example here where I've experienced a similar issue even though I ran through the checklist, upgrade and terraform state mv steps. My steps to reproduce:

  • Create a configuration on 0.11.x where one resource name starts with a number another resource uses the depends_on attribute referencing the resource that starts with a number.

    resource "foo" "123abc" {
    ...
    }
    
    resource "bar" "abcde" {
    ...
    depends_on = ["${foo.123abc}"]
    }
    
  • Apply configuration

  • Update the resource name to not start with a number (including the depends_on attribute)
  • Perform the terraform state mv operation according to your configuration.
  • Apply configuration and confirm all is well.
  • Run 0.12checklist, everything should be dandy.
  • Swap to running Terraform 0.12.0
  • Run 0.12upgrade. Resources should now look like:

    resource "foo" "not123abc" {
    ...
    }
    
    resource "bar" "abcde" {
    ...
    depends_on = [foo.not123abc]
    }
    
  • Run terraform init in the directory to initialise. Should receive the following error message:

    Error refreshing state: 1 problem:
    - Attribute name required: Dot must be followed by attribute name.
    

After inspecting the state file, I can see that the depends_on attribute hasn't been updated in the terraform state mv.

I'm using the S3 backend however I assume it's the same doing it locally too.

Thanks for that additional context, @jacobbednarz.

It sounds like Terraform 0.11.14's terraform apply did not correctly rewrite all of the dependencies in the state to match the latest configuration, which is very unfortunate. In that case, I think we may have to adjust the state upgrade codepath to detect when there is an unparseable depends_on address, drop it from the state during upgrade, and generate a warning that it'll be necessary to complete a terraform apply under 0.12 to correctly re-populate all of the dependencies.

This creates a small window where if you were to e.g. run terraform destroy immediately after upgrading to Terraform 0.12 you'd see these warnings and -- if you chose to ignore those warnings and proceed anyway -- Terraform may try to destroy some things in the wrong order. However, that situation seems unlikely enough that I think we can accept it to make this work properly in the more common path of upgrading to 0.12 and then applying changes.

One detail we'll need to check on is: I believe currently terraform apply will not touch the state _at all_ for a resource instance where no changes were planned, so even running terraform apply with today's logic may be insufficient to get these updated. (In fact, I suspect this is why Terraform 0.11.14 wasn't able to fix all of these automatically on apply.) Terraform very intentionally doesn't create graph nodes during apply for any instance that has no action in the plan, so we may need to do a sort of "pre-processing" or "post-processing" pass (a bit like the current "count boundary fixup") to walk over the state and make sure the instance metadata is consistent with the configuration, regardless of whether a particular instance had any changes to its actual data on the current apply.

I had a aws_s3_bucket resource named "301_redirect". Renaming the resource w/ terraform 0.11 (terraform state mv aws_s3_bucket.301_redirect aws_s3_bucket.zone_apex_redirect) completed successfully:

"aws_s3_bucket.zone_apex_redirect": {
  "type": "aws_s3_bucket",

I then upgraded to terraform 0.12 locally and ran terraform 0.12upgrade which completed successfully. I then tried to run terraform plan and received the above error (Error: Error loading state: Attribute name required: Dot must be followed by attribute name.)

Inspecting the remote state file on S3 more closely I noticed that the old resource was still referenced from within a depends_on section of another resource:

"aws_route53_record.root": {
  "type": "aws_route53_record",
  "depends_on": [
    "aws_route53_zone.main",
    "aws_s3_bucket.301_redirect"
  ],

I then manually replaced the outdated resource name in the depends_on section w/ the updated name which resolved the issue for me.

Followed the upgrade documentation meticulously, including running 0.12checklist but have had to abandon the upgrade and roll back at this point.

Immediately after Terraform runs thorough its state refresh, right before you would normally get...:

------------------------------------------------------------------------

... followed by the changes, it spits out the following errors. I'm sure the amount of times an error is displayed is probably specific to my configuration, but I included the information regardless.

The following error 17 times.

Error: Attribute name required

  on  line 1:
  (source code not available)

Dot must be followed by attribute name.

Followed immediately by this error repeated 35 times

Error: Invalid reference

  on  line 1:
  (source code not available)

A reference to a data source must be followed by at least one attribute
access, specifying the resource name.

Here's the output from debug logging, right before it starts spitting out the errors:

2019/06/03 11:11:53 [TRACE] <root>: eval: *terraform.EvalWriteState
2019/06/03 11:11:53 [TRACE] EvalWriteState: writing current state object for aws_volume_attachment.x-dev0-vol1-attachment
2019/06/03 11:11:53 [TRACE] [walkRefresh] Exiting eval tree: aws_volume_attachment.x-dev0-vol1-attachment
2019/06/03 11:11:53 [TRACE] vertex "aws_volume_attachment.x-dev0-vol1-attachment": visit complete
2019/06/03 11:11:53 [TRACE] vertex "aws_volume_attachment.x-dev0-vol1-attachment": dynamic subgraph completed

_I replaced the project name with 'x', its actually 3 letters._

This is the last resource that's checked on a normal successful 0.11.14 run.

As this error is extremely unhelpful and after digging around for a few days in my 3000 line state file haystack looking for the proverbial needle(s), I've had to give up and roll back the 0.12 upgrade progress.

Things I've tried:

  • Removed all depends_on values from the state file
  • Verified all resource names begin with a letter in HCL as well as the state file
  • Turned on TF_LOG=debug and examined the output
  • Tried removing all references to modules
  • Tried testing with local state vs http remote state
  • Tried upgrading to the latest provider
  • Tried using 0.12.1
  • Tried rolling back to 0.11.14 and running through the upgrade process again

All to no avail.

We're hit by this problem as well.

This is a helpful oneliner to identify all the depends_on entries having a dot not followed by (a letter or a star), which is a good starting point:

terraform state pull | perl -ne 'print $_ if /depends_on/ .. /\]/' | perl -ne 'print $_ if /\.[^a-z*]/i' | sort -u

(the above assumes a remote state)

If anybody can chip in with the exact rules, I can update the regex accordingly.

Thanks for sharing that, @mf-lit!

From reviewing your list, I wonder if Terraform is failing to correctly upgrade the ones that end in .*. I think I remember there being a special rule to deal with that in the state upgrade logic, but I'm not 100% sure. I too am away from things right now, so I won't be able to dig into that immediately here, but I'll take a closer look soon.

Hi @apparentlymart , I wonder if you made any progress with this? It's still blocking us from upgrading.

Thanks.

terraform state mv aws_iam_user.42_dns aws_iam_user.iam_user_42_dns

this command doesn't rename resource in depends_on section. After applied this cmd I modified state manually with push/pull.
state after mv

"depends_on": [
    "aws_iam_user.42_dns"
],

We are stuck in the middle of upgrade process too. Almost going to revert back to 0.11.14.

I do not have any depends_on relations. I made the array empty as part of trying everything else. I tried everything as @bryanuci has mentioned above.

````
Error: State format upgrade failed

Error upgrading state file format from version 3 to version 4: 2 problems:

  • Invalid character: This character is not used within the language.
  • Attribute name required: Dot must be followed by attribute name..

[DEBUG] plugin: plugin process exited: path=/terraform/live/prd/my_project/.terragrunt-cache/TgOgR0jSbtP-O8_EnfEmcpPPSrY/qemHbWzRM4hvMnetTwppbs5hqVk/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.19.0_x4 pid=157
```

We still had Attribute name required: Dot must be followed by attribute name errors, even weeks after we had updated all resource names to not start with numbers after an earlier/aborted upgrade attempt.

Tried again a few days ago, now we understood where it was coming from with the comments in this issue (and the Perl script from @saveriomiroddi - very helpful). I had to update the terraform.tfstate for the few instances that still had the old name in the auto-recorded depends_on. Running refresh isn't enough it seems (maybe it lazy updates?), had to manually edit the JSON file like @remm

So I've acually solved my original issue now and I'm fully migrated to 0.12.
Although I'm not entirely sure how I did it (but have a hunch it was merely upgrading from 0.12 to 0.12.4 that did the trick.

Anyway here's the process I went through in case it works for someone else:

  1. Install Terraform 0.11.14
  2. Run terraform apply across all environments
  3. Run terraform 0.12checklist across all environments
  4. Install Terraform 0.12.4
  5. Run terraform 0.12upgrade across all environments
    (At this point you'll know if it's working or not)
  6. Fix all flagged issues
  7. Run terraform apply across all environments, fixing any failures as we go.

@mf-lit I will try doing an update to 0.12 and then 0.12.4 then. Don't know if that will help.

I my case, I ran the perl script from @saveriomiroddi on the state file and did not highlight anything. Also, I am pretty certain, I did not have any resource names start with an integer if that seems to be a culprit here.

@venkz Please report back if 0.12.4 works for you.

I was forced to roll back to 0.11.14 and was curious whether or not the newer versions would resolve the issue.

@bryanuci will do.

It's good that I started creating a new infrastructure for testing this upgrade. I can afford to destroy all of it if I need to.

After trying out many things that did not work, I am making a final attempt

  1. Destroy all the infra I spun up using 0.11.14.
  2. Then upgrade to 0.12.4
  3. Then try to run plan

After doing all of this, I am still getting the same error.
Attaching my .tfstate file here for reference, its pretty much empty.

{ "version": 3, "terraform_version": "0.11.14", "serial": 58, "lineage": "2e6b6a59-b39c-db65-d34a-f6edde15c1d6", "modules": [ { "path": [ "root" ], "outputs": {}, "resources": {}, "depends_on": [] }, { "path": [ "root", "compute" ], "outputs": {}, "resources": {}, "depends_on": [] }, { "path": [ "root", "ecs_task" ], "outputs": {}, "resources": {}, "depends_on": [] }, { "path": [ "root", "compute", "compute" ], "outputs": {}, "resources": {}, "depends_on": [] }, { "path": [ "root", "compute", "iam" ], "outputs": {}, "resources": {}, "depends_on": []

That is not a complete json, but rest of it all the same.. empty keys and values.

It is surprising to see this upgrade fail even with complete clean state file.

My last attempt would be to get rid of this file all together, so no more remote state at all and then try.

I cannot get this to work and doing a rollback now sadly!

Setup I experimented this with:

  1. No resources in AWS
  2. No remote state file
  3. TF v0.12.4 with TG v0.18.7

I guess it fails while it tries to compute state in local machine.

* provider.aws: version = "~> 2.19"
* provider.template: version = "~> 2.1"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[terragrunt] 2019/07/18 22:28:45 Running command: /bin/terraform plan -var app_name=my-demo-project -var image_tag="2d54673"


Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


module.compute.data.terraform_remote_state.foundation: Refreshing state...
module.ecs_task.data.template_file.task_definition: Refreshing state...
module.compute.module.iam.data.template_file.task_role_policy: Refreshing state...
module.compute.data.terraform_remote_state.prd_env_shared: Refreshing state...
module.compute.data.terraform_remote_state.stg_env_shared: Refreshing state...
module.compute.data.terraform_remote_state.account_shared: Refreshing state...
module.compute.module.compute.module.autoscaling.data.aws_iam_role.autoscale-service-linked-role: Refreshing state...
module.compute.module.network.module.nlb.data.aws_acm_certificate.app: Refreshing state...
module.compute.module.iam.data.aws_iam_policy_document.api_gw_assume_role: Refreshing state...
module.compute.module.network.module.alb.data.aws_acm_certificate.app_internal: Refreshing state...
module.compute.module.network.module.nlb.data.aws_acm_certificate.app_internal: Refreshing state...
module.compute.module.network.module.alb.data.aws_acm_certificate.app: Refreshing state...

Error: State format upgrade failed

Error upgrading state file format from version 3 to version 4: 2 problems:

- Invalid character: This character is not used within the language.
- Attribute name required: Dot must be followed by attribute name..

I've created a PR (#22134) that, while it doesn't deal with potential Terraform bugs, should considerably help debugging troubled configurations, by displaying the problematic token(s) in the error message. Hopefully it will go through the acceptance cycle and it will be merged in :smile:.

@saveriomiroddi Thank you! That should be extremely helpful!

@venkz Sorry to hear you had to roll-back, hopefully @saveriomiroddi's PR will get merged into a release soon.

Hi all!

We've just merged #22136 to add some additional context to these messages about what exactly Terraform was trying to do when it encountered these problems. Once that's released, we can hopefully collect some new versions of these error messages and finally track down which cases are not being handled by the state upgrade mechanism today.

The newer error messages will appear in the next release of Terraform.

I faced this problem and it was due that backend config key was starting with a number:

$ jq '.backend.config.key' .terraform/terraform.tfstate
"NUMBER-something/terraform.tfstate"

After moving state config path in terraform block to a correct one the issue have been solved.
This key problem should be warned by terraform 0.12checklist.

@apparentlymart Do you have an estimate on when the next version with this change included might be released? Or if there's a nightly build or something similar I can use in the meantime.

I'm eager to see if this change helps pinpoint the cause of State format upgrade failed failures for us

We are also affected by this, and it's extremely hard to debug with large states. We identified one resource name that starts with a number but even though it's been removed from the state using terraform state rm, the error keeps happening when running terraform plan:

2019-07-31T10:11:17.288-0700 [DEBUG] plugin.terraform-provider-aws_v2.21.1_x4:   <ResponseMetadata>
2019-07-31T10:11:17.288-0700 [DEBUG] plugin.terraform-provider-aws_v2.21.1_x4:     <RequestId>redacted</RequestId>
2019-07-31T10:11:17.288-0700 [DEBUG] pError: Attribute name required
lugin.terraform-provider-aws_v2.21.1_x4:   </ResponseMetadata>
2019-07-31T10:11:17.288-0700 [DEBUG] plugin.terraform-provider-aws_v2.21.1_x4: </ListAttachedRolePoliciesResponse>
2019/07/31 10:11:17 [TRACE] <root>: eval: *terraform.EvalWriteState
2019/07/31 10:11:17 [TRACE] EvalWriteState: writing current state object for aws_iam_role_policy_attachment.redacted

  on  line 1:
2019/07/31 10:11:17 [TRACE] [walkRefresh] Exiting eval tree: aws_iam_role_policy_attachment.redacted
2019/07/31 10:11:17 [TRACE] vertex "aws_iam_role_policy_attachment.redacted": visit complete
2019/07/31 10:11:17 [TRACE] vertex "aws_iam_role_policy_attachment.redacted": dynamic subgraph completed successfully
2019/07/31 10:11:17 [TRACE] vertex "aws_iam_role_policy_attachment.redacted": visit complete
2019/07/31 10:11:17 [TRACE] dag/walk: upstream of "provider.aws (close)" errored, so skipping
2019/07/31 10:11:17 [TRACE] dag/walk: upstream of "root" errored, so skipping
  (source code not available)

Dot must be followed by attribute name.


Error: Attribute name required

  on  line 1:
  (source code not available)

Dot must be followed by attribute name.

@albertoal try to find the mentioned resource in the state in depends_on section, as described here

Maybe this is obvious, mentioning just in case: if there are terraform_remote_state data sources still on version 0.11 the upgrade will fail.

Here's what worked for me:

  • Fix resource(s) starting with a digit
  • Remove values coming from terraform_remote_state not upgraded yet (hard-code values instead)

@atrepca Does that include terraform_remote_state sources that have been upgraded to 0.11.14?

The upgrade docs read to me that 0.12.x Terraform should be able to read from a 0.11.14 state file:

Terraform v0.11.14 introduced support for reading outputs from the Terraform v0.12 state format, so if you upgrade all of your configurations to Terraform v0.11.14 first you can then perform v0.12 upgrades of individual configurations in any order, without breaking terraform_remote_state usage

I just had the same symptoms (error message) and I went back to terraform 0.12checklist and
got:

provider "openstack" alias "${var.region}" is not a valid identifier.

turns out I have the problem described in #21401: variables don't seem to be supported in provider alias anymore.

Good news, I just upgraded to 0.12.6 and that gave me the real source of the error. It turns out it wasn't the resource in the current project that started with a number but a resource from a remote state being used by the current project that was starting from a number!

$ terraform version
Terraform v0.12.5

$ terraform plan
Error: Attribute name required

  on  line 1:
  (source code not available)

Dot must be followed by attribute name.
#^the above error is repeated 20 more times

$ tfenv use 0.12.6
[INFO] Switching to v0.12.6
[INFO] Switching completed

terraform plan

Error: State format upgrade failed

Error upgrading state file format from version 3 to version 4: failed to
migrate primary generation of
aws_route53_record.4redacted: invalid dependency
reference "aws_route53_zone.4redacted": Attribute name required: Dot must be
followed by attribute name..
#^just the one error message

The new error message makes much more sense, and pinpoints to the specific resource which also starts with a number. Thanks everyone!

0.12.6 is giving us a more descriptive error now:

Error upgrading state file format from version 3 to version 4: invalid
provider config reference
"module.aws_cloudwatch_estimated_costs_alarm.module.admin-sns-email-topic.provider.aws.${var.stack_name}_topic_alert"
for aws_cloudformation_stack.sns-topic: 2 problems:

- Invalid character: This character is not used within the language.
- Attribute name required: Dot must be followed by attribute name..

The resource mentioned in the error is actually from a different Terraform repository (It's remote state is used to grab a couple of outputs, though no outputs to do with the resource above).

I'm unsure whether the issue is with this resource or just with the provider configuration (as pointed at by invalid provider config reference and mentioned above that variables can no longer be used in provider aliases).

It's also not clear to me now whether a 0.12.x Terraform can read from a 0.11.14 state file (Especially if that output has periods in any outputs names, which was accepted in 0.11.x but not 0.12)

@Evesy: it would help if you could share that provider configuration referenced in the error so we can help figure that out. I'm glad to see that the error is useful in 0.12.6!

The terraform_remote_state datasource should be able to read from older (0.11) states, but there are some exceptions. For example, as starting a resource name with a number was allowed in 0.11, but 0.12 cannot parse that.

This was my issue as well, remote states had resources starting with a digit, and using Terraform 0.12.6 pointed to where the problem was.

cc/ @Evesy ☝️

@teamterraform I believe this is the provider resource:

provider "aws" {
  region = "${var.topic_region}"
  alias  = "${var.stack_name}_topic_alert"
}

Edit: This provider config in another repository (linked via remote state) was the culprit.

Module names that start with a digit also gives problems, like

{
    "version": 3,
    "terraform_version": "0.11.14",
    "modules": [
        {
            "path": [
                "root",
                "192-168-all"
            ],
            "outputs": {},
            "resources": {

@teamterraform I believe this is the provider resource:

provider "aws" {
  region = "${var.topic_region}"
  alias  = "${var.stack_name}_topic_alert"
}

Is there a workaround for this?

Thanks @apparentlymart! I'm about to go away for vacation but have this lined up to be retested for my use case mentioned above upon my return 🙇‍♂️

I confirm that #23443 improved the upgrade experience a lot. I've actually succeeded now. Many thanks @apparentlymart !

I am stuck on same issue:

`Error: Error loading state: 2 problems:

  • Unsupported state file format: The state file could not be parsed as JSON: syntax error at byte offset 1.
  • Unsupported state file format: The state file does not have a "version" attribute, which is required to identify the format version.`

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

Was this page helpful?
0 / 5 - 0 ratings