Terraform: [ERROR] <<resource>>: eval: *terraform.EvalForgetResourceState, err: orphan resource <<resource>> still has a non-empty state after apply; this is a bug in Terraform

[andresvia]

Terraform Version

Terraform v0.12.0

Terraform Configuration Files

...

Debug Output

https://gist.github.com/andresvia/b4127a773f804737f85ab20c23e4aefc

Crash Output

Expected Behavior

Plan is applied.

Actual Behavior

2019/05/30 21:45:16 [ERROR] module.uw2.module.core.module.config.module.live: \
eval: *terraform.EvalForgetResourceState, err: orphan resource \
module.uw2.module.core.module.config.module.live.data.aws_caller_identity.live \
still has a non-empty state after apply; this is a bug in Terraform

Steps to Reproduce

terraform plan -out terraform.plan
terraform apply terraform.plan

Additional Context

• My remote state is S3 with DynamoDB locking
• The source for the live module is remote (git), uw2, core and config are "local" (filesystem) modules.

References

• #21515 ?

[mildwonkey]

Hi! Thank you for reporting this issue, and I am sorry that you've experienced it.

It will help if we can see your configuration files (including modules), or a simplified reproduction case, to figure out what's going on here. The error states that there is a reference to an undeclared module so it would be good to see your module layout and config:

https://gist.github.com/andresvia/b4127a773f804737f85ab20c23e4aefc#file-gistfile1-txt-L2442

The configuration contains no module.uw2.module.core.module.config.

🤔 I'd also like to see the output from your terraform plan, specifically regarding that module (as always, please remove any sensitive information).

Thanks again!

[andresvia]

@mildwonkey thanks for taking a look. I'm sorry I've been a little busy, I'll provide plan trace and some stripped down version of the configuration ASAIC.

[andresvia]

@mildwonkey I'm sorry It took me some time to re-build a repro that failed in the same way. But it was actually easier than try to clean up all the sensitive information from a trace.

To repro:

mkdir -p gist
cd gist
terraform init -from-module=git::https://gist.github.com/0713bf66554f38d2fb6efdc7ae581798.git//?ref=master .
terraform plan -out foo.plan
terraform apply foo.plan 

Error: orphan resource \
module.ue1.module.core2.module.config.module.live.module.foo.data.local_file.foo \
still has a non-empty state after apply; this is a bug in Terraform

I've highlighted the part that is weird here:

https://gist.github.com/andresvia/7ee8813d4f1df942bb21a9386b457888#file-main-tf-L20

Because just by commenting that line everything works.

BTW

terraform version
Terraform v0.12.0
+ provider.local v1.2.2
+ provider.random v2.1.2

Your version of Terraform is out of date! The latest version
is 0.12.1. You can update by downloading from www.terraform.io/downloads.html

I guess I'll try with 0.12.1

[andresvia]

I can confirm that the error persist on 0.12.1.

[unacceptable]

I have just verified that this is still an issue in 0.12.4:

Terraform v0.12.4
+ provider.aws v2.16.0
+ provider.random v2.1.2

When my team ran into this we transitioned from using three aws_subnet blocks to one with a count. For some reason, the second subnet was orphaned during this transition. ¯\_(ツ)_/¯

Here is a code snippet of how we ran into this if that helps:

Old code:

resource "aws_subnet" "lambda_private_subnet_a" {
  vpc_id            = "${data.aws_cloudformation_export.web_vpc_id.value}"
  cidr_block        = "10.0.5.0/24"
  availability_zone = "${data.aws_availability_zone.az_1.name}"
  tags  = {
    Environment = "${var.env_name}"
    Name        = "${var.env_name}-${var.env_type}-private-lambda-${data.aws_availability_zone.az_1.name}"
    ProductID   = "${var.env_name}-${var.env_type}"
  }
}

resource "aws_subnet" "lambda_private_subnet_b" {
  vpc_id            = "${data.aws_cloudformation_export.web_vpc_id.value}"
  cidr_block        = "10.0.6.0/24"
  availability_zone = "${data.aws_availability_zone.az_2.name}"
  tags  = {
    Environment = "${var.env_name}"
    Name        = "${var.env_name}-${var.env_type}-private-lambda-${data.aws_availability_zone.az_1.name}"
    ProductID   = "${var.env_name}-${var.env_type}"
  }
}

resource "aws_subnet" "lambda_private_subnet_c" {
  vpc_id            = "${data.aws_cloudformation_export.web_vpc_id.value}"
  cidr_block        = "10.0.7.0/24"
  availability_zone = "${data.aws_availability_zone.az_3.name}"
  tags  = {
    Environment = "${var.env_name}"
    Name        = "${var.env_name}-${var.env_type}-private-lambda-${data.aws_availability_zone.az_1.name}"
    ProductID   = "${var.env_name}-${var.env_type}"
  }
}

New code:

resource "aws_subnet" "lambda_private_subnet" {
    count               = "${length(data.aws_availability_zones.available.names)}"
    vpc_id              = "${data.aws_cloudformation_export.web_vpc_id.value}"
    cidr_block          = "${lookup(var.lambda_private_cidr_blocks, element(data.aws_availability_zones.available.names, count.index))}"

    availability_zone   = "${element(data.aws_availability_zones.available.names, count.index)}"

    tags  = {
        Environment = "${var.env_name}"
        Name        = "${var.env_name}-${var.env_type}-private-lambda-${element(data.aws_availability_zones.available.names, count.index)}"
        ProductID   = "${var.env_name}-${var.env_type}"
    }
}

What I did to get around this was to ensure that the resources provisioned were manually deleted then I carefully edited my remote state (after backing it up).

Super sloppy I know, but I couldn't figure out a way to gracefully reconcile this. 😕

EDIT: I also realize that the new aws_subnet block has deprecated code as well with excess characters ("${}"). I figured that I would leave it the way it was during the initial transformation as it might indicate how this odd state was achieved. Please rest assured that I did use the new syntax while updating this PR for a co-worker.

[archenroot]

Experienced such issue with: https://github.com/terraform-providers/terraform-provider-aws/issues/11024

[Gustavo-Benitez]

Error still present on 0.12.12 with Google Provider 2.19

orphan resource google_compute_instance_template.<...> still has a non-empty state after apply; this is a bug in Terraform

[eddideku]

Error: orphan resource module.myinstance.module.this_instance.aws_instance.this still has a non-empty state after apply; this is a bug in Terraform

0.12.4

Not sure how to fix this. Found this thread.

[unacceptable]

@eddideku I outlined how I was able to work around this by editing my remote state, maybe you could do something similar, but I would recommend doing the following rather than manually editing your state yourself:

terraform state rm '<resource.name>'

https://www.terraform.io/docs/commands/state/rm.html

[seh]

We just ran into this same problem with version 0.12.12, and had to use _terraform state rm_ to fix it like @unacceptable suggested above. We now upgraded to version 0.12.18, in hope of that keeping this problem from returning.

[remm]

we have the same issue on terraform v. 0.12.19 and aws provider v. 2.43.0

[chapter63]

upgraded to terraform v0.12.19 and aws provider v2.44.0

And now every apply throws this error, which fails every CI/CD pipeline we have when deploying terraform changes.

orphan resource data.aws_caller_identity.current still has a non-empty state after apply; this is a bug in Terraform

And we tried deleting that resource, but that resource persists.

[bodonald]

hello, this issue also we are experiencing with terraform v0.12.19 and provider.aws v2.41.0
the workaround is not acceptable for now, as it's a production environment

[philippmoehler0440]

Facing the same issue with aws_lambda_permission resources inside a module.

[JnMik]

Same issue here as well since I updated terraform to 12.19

[adamhathcock]

I was previously getting this issue: https://github.com/hashicorp/terraform/issues/23821

Then I updated to 0.12.20 and started getting these which I assume are more refined errors for the same issue.

provider.aws is 2.48.0

[antonbabenko]

I'd like to add that after terraform state rm is executed, you need to make sure that you run terraform apply to get terraform.tfstate updated and all orphan resources (and modules) are removed from it despite the fact that it will tell Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Also, you can run terraform apply twice and terraform.tfstate will be smaller even after the second run.

[remm]

Looks like the issue has gone in a 0.12.21 version.

[Gowiem]

Experienced this issue with Terraform v0.12.24. The 3 issues that were affected were two aws_acm_certificate resources and one aws_security_group. Using terraform state rm did the trick to fix the issue, but it would be great if we didn't need to do that obviously.

[danieldreier]

@andresilva I ran this against 0.12.0 exactly as you described, and I am able to reproduce the issue. I really appreciate the work you put into making your reproduction case.

When I try to reproduce it against 0.12.26, I run into an error:

Error: Unsupported argument

  on .terraform/modules/ue1.core1.config/main.tf line 20, in output "config":
  20:   type = list // this is weird!!

An argument named "type" is not expected here.

To get around this,I have copied your reproduction case to https://github.com/danieldreier/terraform-issue-reproductions/tree/master/21559, commented out type = list and tried to reproduce again. With that commented out, I cannot reproduce the issue on 0.12.0 or 0.12.26. 0.13.0 beta 1 crashes due to what I believe is an unrelated bug.

This issue has enough upvotes that I believe it's real - at least was in a previous version - but this particular reproduction case no longer works for 0.12.26.

If you, or anyone else on here can make a PR against https://github.com/danieldreier/terraform-issue-reproductions/tree/master/21559 to show a reproduction of this problem that works on 0.12.26 or 0.13.0, I would very much appreciate it. Without a reproduction case, there really isn't any way to fix this.

[boweeb]

For the record, I believe @danieldreier meant to tag @andresvia 😉

[boweeb]

@danieldreier -- Sorry, I didn't realize my comment would drop the waiting-response label. Reapply?

[danieldreier]

@andresvia thanks for your PR there. I appreciate the clarification. I am definitely able to reproduce this on 0.12.0 exactly as reported.

On 0.12.26 this appears to have been fixed. If nobody here can provide a reproduction that causes this issue on 0.12.26 or on an 0.13.0 beta, I think we need to consider this fixed. I'll leave this open for a bit in case someone is able to contribute a reproduction that happens on current TF versions.

[andresvia]

@danieldreier Having type in output seems to have caused some sort of weird issues, not sure why was added in the first place, when got removed and why, but it happened at some point between .0 and .26 is not an issue for me personally anymore because I moved to a more recent TF version and also removed the type that caused TF to fail, I controlled all aspects of it, so it was easy for me. This issue will happen in unlikely scenarios, and may be a problem for folks that are tied up to some external module that indicates a type in an output and they don't control it. It is just weird that type was allowed there in the first place.

Based on comments some folks are affected, but is difficult for me to figure it out why, they may have not upgraded to latest 0.12.x and at the same time they may be sourcing some external module with the type keyword.

Or maybe the issue is reflecting another deeper problem.

Thanks!

[danieldreier]

Thanks for clarifying, @andresvia (and sorry for tagging the wrong person earlier!)

I'm going to consider this resolved. I agree that it was odd to have type in output in the first place.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.