Terraform: Terraform Backwards Compatibility clarification

[martinb3]

Hi there -- I'm hoping to get a clarification on:

The state is in JSON format and Terraform will promise backwards compatibility with the state file. The JSON format makes it easy to write tools around the state if you want or to modify it by hand in the case of a Terraform bug. The "version" field on the state contents allows us to transparently move the format forward if we make modifications.

source: https://www.terraform.io/docs/state/

Does this mean that, for example, v0.11.0 should be usable with a state file that was last modified by v0.11.1? The documentation seems to imply that the state file format is forward compatible, but it seems like bugfix releases may not always be able to all work safely with a state file edited by a later bugfix. I was expecting 0.11.1 to be backwards compatible with 0.11.0 in such a way that (assuming I don't encounter bugs fixed in 0.11.1) I could use both interchangeably.

Thanks in advance!

[apparentlymart]

Hi @martinb3! Sorry for how vague that documentation is. We should get that updated, since I think it's an old doc that isn't capturing some of the details of how we manage compatibility with states.

The current expectation is that newer versions of Terraform will upgrade old states (currently this is true right back to very early state formats, but this will not necessarily always be true) but the converse is not true: there is a mechanism in Terraform for upgrading an old state to a newer version, but no mechanism for _downgrading_ a newer state to an old version.

Part of the reason why things have become more complex here is that the providers are now versioned separately from Terraform Core and _their_ expectations also affect the details of how things are stored in the state file. The providers themselves have mechanisms for upgrading -- but not _downgrading_ -- the portions of the format that are defined by them.

In many cases it can work to manually edit the state to refer to an older state file version in order to downgrade Terraform, since in practice the broad format has not changed significantly in any recent version of Terraform, particularly between minor releases. Terraform does not do this automatically because it cannot _guarantee_ that this will be safe in all cases. Downgrading from 0.11.1 to 0.11.0 should be safe though, as long as all of the provider versions remain constant, since there were not state format changes between those releases.

We are expecting a significant revision to the state file format in order to introduce the forthcoming new type system, which will avoid many of the strange edge-cases that result from using complex data structures in Terraform today. Our goal eventually is a _stable_ state format that is documented, but with Terraform still under heavy development it is too soon to commit to such a thing at this time.

[martinb3]

Hi there -- thanks so much for the speedy clarifications. It sounds like while Terraform releases are backwards-compatible according to semantic versioning, the Terraform state file is currently not backwards compatible against any older release (for now, and without manual editing). That's just what I needed to know; thank you!

[robomon1]

I'm still trying to understand why there is a state file version and a terraform_version in the state file then? I figured that everything using "version: 3" of the state file knew how to read and write that version of the schema and if a version of terraform upgraded to a newer state file version then the version of terraform would use a minor version bump (ex: 0.10.2 to 0.11.0) to reflect that backwards compatibility was broken. So we shouldn't need to know if 0.10.2 or 0.10.6 wrote the state file as long as "version: 3" was the same between them.

[bitbrain]

Is there a way to avoid upgrading the statefile version? For example, a CI has an imaginary Terraform version 1.1 and locally I run with version 1.2, currently it automatically upgrades the statefile version and makes it incompatible for older versions. Is there a way to avoid that or do I always have to keep all my Terraform versions (on all CIs and locally) up-to-date?

[tdmalone]

or do I always have to keep all my Terraform versions (on all CIs and locally) up-to-date?

It’s probably wise to have a ‘supported’ version of Terraform for your organisation. When a new one comes out, check the release notes, possibly do some plans if you’re not sure, and if you’re happy upgrade CI and locally together.

[bitbrain]

I was looking but could not find anything about versioning Terraform strategies. I got a couple of questions:

• Does the fact, that statefiles are not backwards compatible only apply to 0.x versions?
• Will major versions (1.x) have statefiles which are backwards compatible? Thus, if there are breaking API changes, is the major version being incremented? (1.x->2.x)

[apparentlymart]

The common way to work with Terraform in a collaborative environment today is to add an exact version constraint to your configuration, like this:

terraform {
  required_version = "0.12.0"
}

Then if anyone tries to use the wrong version of Terraform, they will be prevented from making any changes. You can then decide to upgrade to a new version _as a team_, communicating that change via a pull-request to update the version constraint, and thus ensuring that everyone must switch over together. For a big release like Terraform 0.12.0 is, an atomic cut-over of just that version number is unlikely to be sufficient, but because this is all code in the same repository we can potentially include both updates for the new version _and_ the new version constraint together in the same PR.

---

Since there aren't any post-0.x versions yet, any statement I might make about them would be aspirational at best. The intent is for 1.0 to represent a set of compatibility promises, but exactly what those promises will apply to and what they will promise depends on what shape the product has and what additional work we see on the horizon at that time.

My expectation -- though this is not certain by any means -- is that the first set of compatibility promises in 1.0.0 will be about the configuration language and the CLI workflow. The first pass of promises is unlikely to apply to the various other artifacts Terraform generates for its own consumption, like the state snapsnots and plan files. Gradually and carefully committing to more compatibility for more parts of Terraform seems like the best approach, since otherwise we'd need to wait until everything was "perfect" (by some definition) before ever releasing a 1.0, and that doesn't seem like it would be of great benefit to anyone.

[gtmtech]

@apparentlymart - just noticed state file upgrade on 0.11.14 - on a "build number" change? That seems to violate standard practice for no-backwards-incompatible changes on "build" (and even minor numbers) used the world over?

[apparentlymart]

Hi @gtmtech!

We did not intentionally introduce any state file format changes in the 0.11.14 release. If you're seeing it upgrading your state in some way other than the usual updating of terraform_version in the file to "0.11.14" then please do open a new issue for that, because we'd consider that a bug.

[ryanm101]

This just happened to me: Was using 0.11.13 moved my local install to 0.11.14 and now it is complaining the state file is too new to be used with 0.11.13

[rudolphjacksonm]

I've noticed this and it seems to only be when you run terraform apply with a newer version against a state that had an older version previously. I was running terraform 0.12upgrade and then a plan across multiple modules and it didn't affect anything. When I tried to run an apply against one of our codebases, forgetting I was running 0.12.6 for testing purposes, I received the error mentioned above.

[shakhal]

Terraform broke for me in 0.12.14 which was working in 0.12.12 and I had to manually edit the state file to get it back to 0.12.12.

maybe the version in the state file shouldn't be updated if the deployment failed?
or save a backup of the previous file if the version changed and the deployment failed?

[shivanipremi]

I removed the .terraform folder and manually edit the state file to get it back to my previous version.

[Ak476024]

I removed the .terraform folder and manually edit the state file to get it back to my previous version.

Hi shivani,
I want to downgrade my .tfstate file from 0.12.26 to 0.12.24. I edited out terraform version in tfstate file. but still getting same error.
This snapshot was created by tf version 0.12.26
Any suggestions how to resolve it

[mildwonkey]

The state docs have been re-written since this was originally opened, and we've made sure to clarify this behavior in each upgrade guide, so I am going to close this documentation issue (there are some feature request tickets open that cover generally improving the user experience with statefile upgrades - we haven't forgotten that this can be frustrating!). Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.