Terraform: Terraform is creating a terraform.tfstate.backup file locally even when state is remote (S3 or Google Storage Bucket)

[ryan-mf]

I'm seeing a new issue, starting with (I think it was?) 0.9.4, where Terraform is writing a terraform.tfstate.backup file in the local working directory even when state is configured to be stored remotely. This is happening for me using remote state in a Google Storage Bucket, but I've confirmed on the hangops Slack group in the #hashicorp channel that others have also started noticing it when using Amazon S3 for remote state, so it appears to be a global issue for remote state storage. It means anything sensitive in the state is now being stored in the local system, and potentially even being pushed to source code repositories if these files aren't set to be ignored.

Terraform Version

v0.9.6

Terraform Configuration Files

terraform-base.tf:

terraform {
  backend "gcs" {}
}

// GCP Storage Bucket to store Terraform state
resource "google_storage_bucket" "terraform-state" {
  provider           = "google.gcp-${var.region_primary}"
  name               = "tf-state-${var.ENV_SHORT}"
  location           = "US"
}

// GCP Location for Terraform state
data "terraform_remote_state" "terraform-state-gcs" {
  backend            = "gcs"
  config {
    project          = "${var.ENVIRONMENT}"
    bucket           = "tf-state-${var.ENV_SHORT}"
    path             = "terraform/terraform.tfstate"
  }
}



md5-9f355c3fe0f46e897e6ed84d4562d5fd



terraform init -backend=true -backend-config=bucket="tf-state-<env>" -backend-config=path="terraform/terraform.tfstate" -backend-config=project="<projectprefix>-<env>"

1. Add some resource configuration blocks to the Terraform configs.
2. Run terraform apply to create those resources, and watch it write to the remote state bucket and the local file terraform.tfstate.backup

[aleybovich]

I see that as well - *.tfstate.backup is being created locally. Doesn't affect any functionality but is slightly annoying as you expect to have no tfstate files in the working directory. If that file is needed locally I would suggest moving it to .terraform - same place where local copy of *.tfstate is.

[max-lobur]

I see the same on Terraform v0.9.8, S3 backend

[kael999]

I see the same on Terraform v0.9.11, S3 backend

[jsosic]

Every version from 0.9.6 to 0.9.11 does the same thing. This is becoming frustrating.

[itsSaad]

same thing
Updated from Terraform v0.9.1 to Terraform v0.9.11 and this issue pops up.
Had to gitignore it.

[ghost]

+1

[andrey-iliyov]

+1

[nbering]

As a general point of GitHub etiquette, please don't +1 issues. Use the reactions feature GitHub has provided. You can actually sort issues by reaction counts. You can't do that with +1 comments.

[melvyndekort]

This is indeed a very annoying bug, I'm adding *.tfstate.backup to a .gitignore file at least once a week :-(

[nodesocket]

👍 using v0.10.0 and gcs backend. Is committing terraform.tfstate.backup to source control potentially dangerous? If so, I suspect many people who use a remote backend are inadvertently committing it.

[pikeas]

Just bit by this as well. Could we please get a reply from Hashicorp?

[apparentlymart]

Hi all! Sorry for the delayed response here.

This backup file is created locally to allow for it to be used to recover in the event of an erroneous update. It's placed locally rather than remotely because the recovery commands (via terraform state push, etc) want a local file to work with, and we want to be able to create the backup even if the backend write fails for some reason.

You're all correct that prior to 0.9 this file was created as a sibling of the terraform.tfstate file, which was kept in .terraform/ when remote state was active. In 0.9, the local cache of remote state was removed. It sounds like this backup path was then broken by a later change (in 0.9.6) which caused the backup path to not get overridden anymore, which in turn makes it default to a path relative to where it would be locally: directly in the working directory.

I understand the annoyance this causes, and agree that it should be written instead into the .terraform directory since we already recommend that this directory be placed in .gitignore.

One wrinkle here is that we _do_ still have a .terraform/terraform.tfstate file that _isn't_ actually a state file but is rather maintaining a few local settings for the workspace. This was a compromise to avoid weird issues when upgrading Terraform with this file already in place. If we were to restore the original behavior of writing to .terraform/terraform.tfstate.backup then that would seem to imply that the file is a backup of .terraform/terraform.tfstate, which is not the case.

Therefore I'd like to propose that we move it to .terraform/backup.tfstate. This still communicates that it's a backup but doesn't imply that it's related in any way to .terraform/terraform.tfstate. In future we will probably also rename .terraform/terraform.tfstate to something less confusing, since the compatibility bump we were trying to get over is now in the past.

I acknowledge that this doesn't address the concern of the state containing sensitive information and now being written in the local filesystem. My proposed change above doesn't address this, and focused only on preventing the backup from being inadvertently added to version control. We could consider separately providing the option to disable the local backup to address this, but since the state being on local disk isn't a _new_ problem (that's been true from day one) I'd prefer to address that separately.

Does that seem like a reasonable path here?

[melvyndekort]

@apparentlymart this sounds like some great steps in the right direction!

And you're right that renaming the backup file to .terraform/backup.tfstate leaves less room for confusion, it sounds like the best solution. Also renaming the .terraform/terraform.tfstate is a logical step to prevent further confusion.

[nodesocket]

👍 for simplicity and just moving /terraform.tfstate.backup to .terraform/backup.tfstat.

[reubit]

Wasn't the whole point of removing the local terraform.tfstate cache to prevent potentially sensitive data from getting written locally to disk? I thought this was an awesome move btw, as it meant dev's could source secrets from somewhere external, apply them somewhere else, and never risk having the secrets exposed if a laptop gets compromised.

Now these backup files are getting dropped all over the place, littered with secrets, which is a major security regression from the initial 0.9.0 release in my opinion. (Unless they were getting written to .terraform/terraform.state.backup between 0.9.0 and 0.9.6, and i just didn't notice... i've had terraform.tfstate* in my .gitignore files since we started using remote states)

My preference would be completely disabling backups by default if using remote states. If you're using S3 with versioning enabled (or something equivalent), you've got backups of all your previous states anyway. If you _really_ want to create a backup, you should be required explicitly pass -backup=foo.tfstate to whatever command you're running, perhaps also making the default behaviour configurable in the remote state configuration.

If there's concern about failing to update the remote state after applying changes, why not just keep everything in RAM, and only drop the backup to disk if you get an error writing to the remote state store? Or at the very least, delete it from disk after you confirm a successful write to the remote state store.

[ryan-mf]

I agree with @reubit, but I'm not sure if it's within the scope of this ticket. I hate worrying a dev will get their laptop stolen with all of our sensitive state on it.

[apparentlymart]

Hi all,

As noted before, I understand that there are really two issues being discussed here. Moving the file back to where it used to be (in the .terraform directory) is a straightforward change that we can make quickly, so that's why I chose to separate these two ideas for the sake of this work.

I agree that disabling the backup files altogether by default seems like a reasonable idea, but that requires more caution since it's something that could affect people's workflows. I'm open to it, but at least we'd need to wait until the next major release so we can talk more loudly about it in case anyone is relying on it and needs to make new accommodations, such as adding a new option to commands as @reubit suggested. We can move the file back into the .terraform directory in the mean time, so at least we reduce the risk of it being inadvertently placed into version control.

I also want to note that as of 0.9.6 Terraform got a new behavior where if the final state write (at the conclusion of terraform apply) fails for some reason it will, as a last resort, write a file named errored.tfstate to the current working directory and exist with an explicit error saying that it is there. This means that there is still the chance of the state being written to disk, but it does at least happen only if the remote backend is failing for some reason, and there's a very explicit error message about it. The backup tfstate is generally more for reverting in the case of human error rather than recovering from machine error, but as others have noted certain backends have remote support for versioning which serves this need.

In general I would not recommend those who have sensitive data in state files to be routinely working with those state files on arbitrary laptops -- in that case, it's better to run Terraform in a well-maintained, secure environment -- but I know that this is often easier said than done, so I'm definitely open to improving the default behavior to reduce the risk of accidental secret leakage especially since, as noted, the use-case for this backup file can be served in other ways.

[davidpellerin]

I think that moving the file to the .terraform directory is a good pragmatic first step.

[unacceptable]

This is preventing me from executing terraform via a lambda function (because AWS Lambda has read only file systems).

Why would I attempt to use Lambda in the first place you might ask. To this, I counter why not. I thought that this would be an interesting project that I might be able to create a use case for.

A great example for a use case would be if a third party developer wanted to spin up a dev/staging environment they would be able to email me and AWS Lambda with Terraform would spin up the predefined environment automagically, without releasing console credentials if there is no need for it.

Please at the very least let us disable creation of terraform.tfstate.backup files. If we have versioning enabled on our S3 buckets this 'feature' is pretty useless.

[jcrben]

@apparentlymart could you respond directly to this comment:

Wasn't the whole point of removing the local terraform.tfstate cache to prevent potentially sensitive data from getting written locally to disk?

Should we open a new issue? Or can you do that? Or is there an existing issue out there already? Pull requests welcome I imagine?

Earlier you said:

I acknowledge that this doesn't address the concern of the state containing sensitive information and now being written in the local filesystem ... since the state being on local disk isn't a new problem (that's been true from day one) I'd prefer to address that separately

Is this a problem "now" or has it always been around?

The docs seem somewhat inconsistent:

At https://www.terraform.io/docs/backends/config.html I see this:

The final, merged configuration is stored on disk in the .terraform directory, which should be ignored from version control.

Meanwhile at https://www.terraform.io/docs/backends/state.html I see this:

When using a non-local backend, Terraform will not persist the state anywhere on disk except in the case of a non-recoverable error where writing the state to the backend failed.

[jmickey]

As a work around could you give us a flag to force the backup to be placed along the tfstate file in the remote? In the event a recovery is required then downloading the backup from the remote (S3) in my case, isn't going to be a huge deal....

[et304383]

This is a huge difference from the docs explicitly stating that no state will be written to disk. This really needs to be fixed, pronto!

[jbehling]

i just ran into this and i agree: the docs are very misleading. The terraform.tfstate.backup shouldn't be created at all or that information should be added to the documentation

[snwight]

For our purposes we absolutely need to prevent writes of secrets to local disk, that's why we're using remote state to begin with. Even storing the backup state to .terraform is unacceptable by the same reasoning. I'd advocate for a flag to prevent any backup state storage.

[nodesocket]

@snwight agree ultimately remote storage should never write backups to disk, but writing into the root as terraform.tfstate.backup is exponentially worse, since the likelihood of accidentally pushing that file to source control is very high.

[snwight]

Totally agree, yes

[yosefy]

+1

[aaomoware]

@yosefy

i guess you didn't read through the thread before commenting? If you did, then you missed @nbering comment?

"As a general point of GitHub etiquette, please don't +1 issues. Use the reactions feature GitHub has provided. You can actually sort issues by reaction counts. You can't do that with +1 comments." @nbering

[yosefy]

true sorry will do reactions next time

Sent from my iPhone

On 13 Oct 2017, at 15:52, A A Omoware [e-mail] wrote:

@yosefy

i guess you didn't read through the thread before commenting? If you did, then you missed @nbering comment?

"As a general point of GitHub etiquette, please don't +1 issues. Use the reactions feature GitHub has provided. You can actually sort issues by reaction counts. You can't do that with +1 comments." @nbering

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.