Terraform: state mv -state-out=... module.source module.destination doesn't work with s3 backed remote states

[dimisjim]

Terraform Version

0.11.14

Expected Behavior

terraform state mv -state-out=../otherProject/terraform.tfstate module.source module.destination should move desired module form origin's S3 backend to the other project's statefile if it's placed locally, or ideally, even if it is stored in S3 as well.

Actual Behavior

State is removed from origin, but not moved to destination state.

Steps to Reproduce

1. Have two projects with backend in S3, or origin in S3 and destination as a local state file
2. Move a module from one project to another by for example terraform state mv -state-out=../otherProject/terraform.tfstate module.source module.destination
3. Origin state is changed correctly, destination is unaffected.

References

Current workaround, basically downloading the statefiles, apply the command on the locally, and then moving them back to the bucket: https://github.com/hashicorp/terraform/pull/15652#issuecomment-410754814

[mildwonkey]

Hi @dimisjim , I'm sorry you've come across this odd behavior!
Can you share the debug output from your terraform state mv command? I'm curious if any errors are getting buried (check for sensitive data in the output, first).

[apparentlymart]

I think we saw in another issue that the -state and -state-out arguments are arguments just to the local backend and not to Terraform itself, so whenever any other backend is used they have no effect.

In the short term it'd be good to find some way to make Terraform report an error in that case, rather than to silently do something unexpected/harmful.

I do agree that it would be better for the presence of -state and/or -state-out to override to use local state storage for only one part of the operation as expected here, to enable this sort of local-to-remote or remote-to-local refactoring. That will require some significant refactoring of how Terraform approaches state storage though -- probably to separate it more completely from the "backend" idea into a separate concept -- and thus I expect we won't be able to tackle that until we're doing a more general refactor of the backend and workspaces concepts we've been discussing, to simplify the user experience for them; that way we can get the necessary breaking changes out of the way all at once, to create less churn for those who are relying on the current behaviors.

[dimisjim]

@apparentlymart Thanks for the answer.

I think I will go ahead with the workaround linked in my initial post for the time being.

[carmo-evan]

Running into same issue today. Pretty frustrating.

[descampsk]

Yes same issue for me when I have to move state from one remote S3 backend to an other.
I do a similar workaround inspired of this https://medium.com/@lynnlin827/moving-terraform-resources-states-from-one-remote-state-to-another-c76f8b76a996.

[jfirebaugh]

There is also apparently no way to move from a local state file into remote state.

[JSteeleIR]

I encountered this bug, too, with GCS remote backends, and it's most irritating. Moving state between two remote backends is scary enough, even more so when the preferred tool to validate changes before they're actually pushed (terraform state) doesn't actually respect the local flags.

My process (also following https://medium.com/@lynnlin827/moving-terraform-resources-states-from-one-remote-state-to-another-c76f8b76a996) was:

1. Move the config for my resource from dev/ to prod/ (These are separate TF environments, that know nothing about each other, and each have their own remote state buckets.)
2. Pull remote state from each backend to local, and make local backups (To be able to validate changes)
   
   
   
   1. dev$> terraform state pull > tfstate && cp tfstate tfstate.bak
   
   
   2. prod$> terraform state pull > tfstate && cp tfstate tfstate.bak
   
   
3. Move the desired resource from dev->prod
   
   
   
   1. `dev$> terraform state mv -state=tfstate -state-out=../prod/tfstate (Terraform claims it's "Successfully moved 1 object(s).")
   
   
4. Attempt to validate that the state moved:
   
   
   
   1. dev$> terraform state list -state=tfstate | grep <resource> (No results)
   
   
   2. prod$> terraform state list -state=tfstate | grep <resource> (No results)
   
   
5. Attempt to validate the state moved, manually:
   
   
   
   1. dev$> diff tfstate tfstate.bak (shows state diff)
   
   
   2. prod$> diff tfstate tfstate.bak (shows state diff)
   
   
6. Spend significant amount of time confused as to why the state files show diffs, but the terraform state list command doesn't see the resources.
7. Finally decide to just YOLO and push to dev, run a plan:
   
   
   
   1. dev$> terraform state push tfstate
   
   
   2. dev$> terraform plan (Shows no diff, as expected. The config for the moved resource has already been removed from dev and put in prod.)
   
   
8. Be more confident in that things are actually moving, terraform state list -state=tfstate just isn't working properly, push to prod and run a plan.
   
   
   
   1. prod$> terraform state push tfstate
   
   
   2. prod$> terraform plan (No Diff, as expected)
   
   

All-in-all, the fact that terraform commands _SILENTLY_ don't respect the -state flag when remote backends are enabled leaves me feeling like remote backends need a lot of work.
Ideally, moving between remotes would be trivial, but in the absence of that, I expect that the normal terraform state commands should work normally, and not add to the already-stressful confusion that can be part of moving state.