Terraform-provider-google: Refreshing state before destroy fails when gcp service account key not found.

[genevieve]

Terraform Version

v0.11.8 with google provider v1.16.2

Affected Resource(s)

• google_service_account_key

Expected Behavior

Since we are trying to destroy this resource, it should not fail when the resource does not exist.

The resource uses the handleNotFoundError function but this only does a check for a 404. For this resource, the googleapi is actually returning a 403 Forbidden for the missing resource. I'm not sure if there are scenarios where we want to validly return a 403 failure for a resource you obviously created earlier with terraform, or if, for this resource, we could do a check for that api error as well and continue along.

Actual Behavior

* google_service_account_key.opsman_service_account_key: 1 error(s) occurred:

* google_service_account_key.opsman_service_account_key: google_service_account_key.opsman_service_account_key: Error reading Service Account Key "projects/cf-releng/serviceAccounts/[email protected]/keys/x": googleapi: Error 403: Permission iam.serviceAccountKeys.get is required to perform this operation on service account key projects/cf-releng/serviceAccounts/[email protected]/keys/x., forbidden

Steps to Reproduce

1. terraform apply
2. The key gets deleted
3. terraform destroy

[Chupaka]

Well, it should, if you removed something not via Terraform :)

To fix this, use terraform state rm

[genevieve]

That’s definitely an option when you’re manually running this and only have to resolve this once.

We have multiple environments in CI that regularly show failures like this when running terraform destroy. We would have to automate searching the logs for any resources that are already gone, removing them from the state file, and then trying again.

[Chupaka]

Terraform is for full-cycle resource management. You should not 'help' it adding/changing/removing resources behind his back. If you want to remove keys manually - then create them manually, not via Terraform :)

[genevieve]

Then why do so many other resources use the handleNotFoundError to ignore 404s when refreshing state and destroying?

[genevieve]

And it’s not intentional deletion of resources “behind his back”. If a CI container is unavailable after running terraform destroy, you aren’t able to persist the tfstate at that time. When you run the job again, it sees the old tfstate and should be able to refresh, see some resources have already been deleted, be ok with that, and move on.

[ndmckinley]

I don't know about "it should" - certainly it's an understandable bug, you can see how we weren't thinking of that use case when writing it, but the best behavior is the behavior that works best for the most people. We'll fix it, for sure. PR's open and should be merged soon and then the fix will be in the next release.

[genevieve]

Thanks Nathan.

[tsadoklevi]

Hi @genevieve @ndmckinley - I'm getting the following error -

Error: Error refreshing state: 1 error(s) occurred:
* google_service_account_key.cloudsql-sa-key: 1 error(s) occurred:
* google_service_account_key.cloudsql-sa-key: google_service_account_key.cloudsql-sa-key: Error reading Service Account Key "projects/anakatech/serviceAccounts/[SA_EMAIL]@[PROJECT_ID].iam.gserviceaccount.com/keys/9ac6bddf44603244c9d0fd8d31b0f05d38a55538": googleapi: Error 403: Permission iam.serviceAccountKeys.get is required to perform this operation on service account key projects/[PROJECT_ID]/serviceAccounts/[SA_EMAIL]@[PROJECT_ID].iam.gserviceaccount.com/keys/9ac6bddf44603244c9d0fd8d31b0f05d38a55538., forbidden

Is this error related to this issue? If so, than it should be re-opened, right?

[ndmckinley]

Did you get that error while running terraform destroy?

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!