Terraform: Error: leftover module

[arity-the-bot]

Terraform Version

Terraform v0.12

Debug Output

Error: leftover module module.high-access-isolation-segment in state that should have been removed; this is a bug in Terraform and should be reported

Crash Output

Expected Behavior

The leftover module and all of its resources should have been destroyed.

Actual Behavior

Terraform apply command failed due to the "leftover module" error, and the module was not removed.

Steps to Reproduce

terraform get -update=true ${terraformDirectory}

terraform init -backend-config='bucket=${deploy['statefileBucket']}' -backend-config='region=${deploy['statefileRegion']}' -backend-config='key=${deploy["statefileName"]}' ${terraformDirectory}

terraform plan -no-color -input=false -target module.bosh -target module.concourse -target module.opsman -var-file tf-vars.json -out terraform.tfplan

terraform apply -no-color -input=false -auto-approve -target module.bosh -target module.concourse -target module.opsman terraform.tfplan

Additional Context

References

[baurmatt]

Is there a workaround for this? Currently out CI is broken due to his issue :(

[mendhak]

Our CI has just broken as well. We could do with a workaround.

Version: 0.12.19

[sriddell]

I don't know if this will work for everyone; I just ran into this myself. I was able to manually fix it by running terraform first, and letting it delete the resources in the removed "my-module"; then downloaded the state file directly (in my case, from aws s3), edited the state file to remove all "module.my-module" entries, and copied it back into s3. After that, I wiped out .terraform, and was able to re-run terraform init and apply without the error.

Note that in my case, I'm not using -target; I was just removing the use of a module from existing terraform code, wanting all the resources the module managed to be cleanly removed.

[dghubble]

I've seen this across Terraform v0.12.19 workspaces without using -target as well. It seems that #23859 has a somewhat simpler repro example.

23822 appears to be an unreleased fix. Its a bit confusing bc many folks have started reporting this problem in different issues (e.g. https://github.com/hashicorp/terraform/issues/23821). Figured I'd save you a search.

[brucedvgw]

Running in to the same issue on v0.12.19 on a module that was removed but still exist in state. I have tried to run state rm the resources but they can not find the resources.

[ali-sattari]

@sriddell solution worked in my case (no -target in my case either)

[shanedroid]

Hit this as well using v0.12.19 and the remote backend. I can download a state file from TF Cloud but can not fix this using TF Cloud web ui 😕

I attempted to use the TF Cloud API as a workaround:

1) locked the state for the given workspace in TF Cloud UI

2) downloaded the current state file from TF CLoud for the given workspace

3) modified the local state file and convert to base64 openssl base64 -in tfcloud_old.tfstate | pbcopy

3) pushed an updated state with orphaned/removed module with serial++

{
  "data": {
    "type":"state-versions",
    "attributes": {
      "serial": $OLD_SERIAL++,
      "md5": "$md5",
      "state": `$ pbpaste`,
      "lineage": "$OLD_LINEAGE"
    }
  }
}

curl \
  --header "Authorization: Bearer $USER_TF_CLOUD_TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request POST \
  --data @payload.json \
  https://app.terraform.io/api/v2/workspaces/$TF_CLOUD_WORKSPACE_ID/state-versions

Ended up with a broken/completely empty state file most likely due to my own error (never tried this before with TF Cloud)

Fortunately was able to just recreate the workspace because this was a trivial test/internal use case. I am interested in learning what the recommended steps TF Cloud users are supposed to take to deal with this.

[isubuz]

Seeing the same issue with TF v0.12.19. Steps to reproduce

1. Create a module
2. TF apply
3. Remove the module
4. TF apply returns the "leftover module" error.

For now, I have been manually updating the state file and pushing it to the remote backend. But this is a real blocker!

[mmshaikh88]

I have also seen the same issue with TF 0.12.19.
See my ticket for the steps to reproduce. https://github.com/hashicorp/terraform/issues/23866

[brucedvgw]

This is a real issue now for Terraform Cloud users. All Applies are now throwing errors.

Any fix yet?

[brucedvgw]

Looks like v0.12.20 will address this https://github.com/hashicorp/terraform/blob/master/CHANGELOG.md#01220-unreleased

[jack1902]

Ran into this aswell, i've put a thumbs up on the initial issue to show support for it (remember this helps prioritize the fix)

Been using streamalert_release-3-0-0 and this issue hit me hard

[efernandes-dev-ops]

Same currently running into this as well, when will there be a release of v0.12.20, does anyone know?

[witchbutter]

I am having the same issue where all of my states are stored in DynamoDB so I can't edit the modules out of a tfstate file. It only happens for me during terraform apply and the apply completes despite the error. terraform state rm module.name reports nothing to clean from the state.

Terraform v0.12.19
+ provider.aws v2.42.0
+ provider.external v1.1.2
+ provider.template v2.1.1

[efernandes-dev-ops]

For anyone that can't wait for v0.12.20, to get around this you will need to download the state file in a file, find the resource that it was complaining about and remove it. Then you will need to push the updated state file up.

[jack1902]

to add to @efernandes-ANDigital you can use:

terraform state pull > terraform.tfstate to pull the state file to a local file

terraform state push terraform.tfstate once you have made any changes required (increment serial by one in order for this to push to succeed)

[mignaulo]

@jack1902's workaround worked well for us.
I'm surprised that hashicorp is not releasing 0.12.20 more quickly, or at least a hotfix that corrects this issue. This effectively breaks terraform for us, I have to assume this also affects many others negatively.

[teodor-kostadinov]

The workaround didn't work for me. TF complains the JSON file is not formatted correctly. I made sure all commas are there etc even validated the JSON file but still can't make TF to push the state file back.

I am hoping 0.12.20 will be out soon. TF is broken for us.

[martinbjorgan]

Just had and corrected the same problem @teodor-kostadinov Did you remember to +1 the serial?

[teodor-kostadinov]

@martinbjorgan - yup, upped the serial +1 as well.
No luck.
All I get is this:

[martinbjorgan]

Hm.. try pulling the state as .tfstate file, not a .json?

[teodor-kostadinov]

Good shout but still doesn't work for some reason. My state file is pretty big. It is around 6MB.
I presume that could be a problem.

I will try to download from S3, remove the leftover modules and then re-upload. Hopefully, that won't fully break it.

[bgshacklett]

@teodor-kostadinov Are you on a Windows machine? I've seen similar issues with text editors adding a Byte Order Mark (BOM) to the JSON file which caused some JSON parsers to choke.

[grigorov]

anybody know, when release terraform 0.12.20 ?

[MarvinChen003]

Similar issue happens when we trying to destroy DirectGateway, TransitGateway, Route53 modules through terraform.

Hope 12.20 helps.

[teodor-kostadinov]

@teodor-kostadinov Are you on a Windows machine? I've seen similar issues with text editors adding a Byte Order Mark (BOM) to the JSON file which caused some JSON parsers to choke.

That was it! Thanks :) @bgshacklett

[teodor-kostadinov]

Phew. 0.12.20 is now released. I cannot reproduce the problem on my side anymore.

[matschaffer]

Saw this on 0.12.19, upgraded to 0.12.20 and stopped seeing it as well.

[froch]

can confirm as well, 0.12.20 fixed this. thanks team terraform!

[nidhiben]

Agreed, This is fixed 💯

[lorengordon]

Great timing! Just updated a large config, consolidating several modules into one, moved the resources to the new state addresses, and suddenly hit this error under tf 0.12.19. Updated to tf 0.12.20 and it works!

[confiq]

can confirm, upgraded to 0.12.20 and problem disappear

[DrFunk-n-stein]

As another workaround, this worked for me:

• create a module with the same name / path with a null_resource in it which basically does nothing
• apply + destroy with -target option on this module.

[ahilsend]

0.12.20 fixed it for me

[tobypinder]

There doesn't appear to be a relevant entry in either the 0.12.20 or 0.13.0 (Unreleased) changelogs related to this issue, is this being tracked as a Bug Fix in those files?

[colin-lyman]

This issue can be closed. v0.12.20 is known to fix the issue of leftover modules.

[advissor]

After moving from 0.12.19 to v0.12.25 - the issue is gone.
Looks like since v0.12.20 it is fixed :)
Thanks!

[jbardin]

This has been fixed in master, and will be part of the 0.13 release.
The handling of module removal has been refactored so this is a case that can no longer happen. Any similar errors should be filed as a new issue.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.