Terraform-provider-google: Error 400: Requested policy version (1) cannot be less than the existing policy version (3)

[Lirt]

[issue-type:bug-report]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.12.18

• provider.google v2.20.1
• provider.google-beta v2.20.1

Affected Resource(s)

One of

• google_project_iam_member
• google_project_iam_binding

Terraform Configuration Files

module "projects_iam_bindings" {
  source  = "terraform-google-modules/iam/google//modules/projects_iam"
  version = "~> 5.0"

  project = "XXX"
  mode    = "additive"

  bindings = {
    "roles/errorreporting.user" = [
      "group:XXX@YYY",
    ]
  }
}

Debug Output

Error: Error reading Resource "project \"XXX\"" with IAM Member: 
Role "roles/errorreporting.writer" 
Member "serviceAccount:[email protected]": 
Error retrieving IAM policy for project "XXX": 
googleapi: Error 400: Requested policy version (1) cannot be less than the existing policy version (3). 
For more information, please refer to https://cloud.google.com/iam/docs/policies#versions., badRequest

Expected Behavior

Succesful plan

Actual Behavior

Error 400: Requested policy version (1) cannot be less than the existing policy version (3).

Steps to Reproduce

1. terraform plan

Important Factoids

Since I didn't manually alter any IAM setting I think that there is a different issue.

It may be connected with the new condition feature, which uses IAM policy version 3.

[edwardmedia]

@Lirt can you post the full debug log?

[Lirt]

Hi @edwardmedia here is the relevant part
https://gist.github.com/Lirt/aacfb287edb288f10ae1de091ebbf0ec

[edwardmedia]

@Lirt terraform-provider-google might not be the best team to help you resolve this issue. But I notice that you don't have bindings={}. Is that on purpose? Here is the link where you can see a sample for using this module. https://github.com/terraform-google-modules/terraform-google-iam/blob/master/examples/project/main.tf

[Lirt]

@edwardmedia I'm sorry, I have it in code, but I accidently missed it. I updated the description now.

[edwardmedia]

@Lirt can you post the full debug log?

[danawillow]

@Lirt, do you have any IAM Conditions set on the policy? Also, are you using the google provider or the google-beta provider with that module?

[Lirt]

@danawillow We tried IAM conditions but not on policies that are bugged. Currently we are using stable provider, but I also tried beta. It didn't help.

[Lirt]

This should make sense now.

The exact error I got is listed in documentation - Scenario 2: Requesting a policy with an insufficient version (https://cloud.google.com/iam/docs/policies#specifying-version)

When I display version of bugged IAM policy, I get version: 1.

gcloud beta iam service-accounts get-iam-policy [email protected]

bindings:
- members:
  - serviceAccount:myproject.svc.id.goog[.../...]
  role: roles/iam.workloadIdentityUser
etag: ASDF=
version: 1

The error log says Requested policy version (1) cannot be less than the existing policy version (3).. That means it is requesting correct policy version (1). But fails... (if the log is OK)

I tried to use beta-provider 2.20.1 again and IT WORKS! (my bad I must have made some mistake before).

There must be a bug somewhere I guess.

[edwardmedia]

@Lirt I am glad you have found the reason. I am closing this issue now. You may reopen it if you feel a need for more help. Thanks

[pratikmallya]

I am seeing a similar issue. The strange thing is that the version of the IAM policy is still 1. Not sure why this is cropping up

[danawillow]

@Lirt and @pratikmallya (and anyone else encountering this), if you have a support contract, customer engineer, or account rep with Google Cloud can you reach out to them? I'd like to get the IAM team's perspective on this behavior, and using an official GCP support channel will help get the right information to the right people.

[pratikmallya]

@danawillow I do, will open a support ticket. Thanks!

This actually worked after using the provider mentioned by @Lirt. But yeah its rather strange that it just broke.

[danawillow]

What provider version were you using before, and which one are you using now that worked?

[pratikmallya]

The changes I had to make to fix were. Using terraform 0.11.14

[danawillow]

Yeah I think the reason it's working now is that we started reading policy version 3 around then in the beta provider.

[morgante]

@danawillow Did we start reading policy version 3 for the GA provider? Is that planned?

[danawillow]

@morgante We haven't, and I haven't really thought about it much. If a user has conditions enabled (which as far as I know is the only way to get a version 3 policy) and then tries to use TPG to make modifications to it, it would clobber the conditions, so I'm a bit hesitant to read at version 3 until conditions are GA. How would a user end up in a situation when using TPG that their policy is at version 3?

[morgante]

@danawillow It was raised on the module here, @Lirt could you provide more context?

I can definitely imagine a scenario where we'd see issues with this:

1. An IAM role is assigned with conditions (using the UI maybe?)
2. A user attempts to add a google_project_iam_member using TPG for an unrelated role/membership (no conditions) and it fails.

I think the complexity is that I believe the entire policy is versioned to v3 even if only one membership has conditions.

[danawillow]

Got it, that makes sense. I think in the interest of safety, I'd still prefer to wait until the conditions GA, though if it becomes too unusable this way we can reevaluate.

[morgante]

Hmm, it seems like a rather severe bug that a working Terraform resource (google_project_iam_member) would start failing unpredictable and forcing users to migrate to beta even when they are not using permissions.

[danawillow]

If they aren't using conditions, then their policy version should be 1, and TPG and TPGB are both viable options.

If they are using conditions, and they're using it on the binding referenced by that resource, they have to use TPGB since that's what has conditions support.

So the circumstance we're dealing with, as you said, is a binding that doesn't have a condition on it but somewhere else in the same policy does, and that other part of the policy isn't being managed in Terraform. Do you expect that to be a common setup? In this case, the options for the user are:

• Be confused, find this issue, and change to the beta provider
• Use TPG (which we update to use version 3), and run the risk of unintentionally clobbering a condition that was added out-of-band without noticing

I'm willing to be convinced that this is a common enough setup and the risk of clobbering a condition is low enough that we should go ahead and do it, but I'm not quite there yet.

[morgante]

I think the complication is that IAM for a resource (particularly an org or project) isn't necessarily managed all in a single Terraform config. If that were the case, I think it'd be easy to simply upgrade to TPGB. However, there are definitely cases where some memberships are managed elsewhere.

I agree that clobbering is an issue though. Maybe we can leave this open for a bit and see who else reports it?

Use TPG (which we update to use version 3), and run the risk of unintentionally clobbering a condition that was added out-of-band without noticing

Would it be possible to simply not touch the condition from TPG? ie. if a condition exists simply leave it as is (but don't support adding/updating conditions)?

[Lirt]

@morgante Just like you said. Conditions were tested in UI for some temporary account and all other iam policies / members in terraform got broken.

[jbouchery]

@morgante Just like you said. Conditions were tested in UI for some temporary account and all other iam policies / members in terraform got broken.

Hi @Lirt ! We encountered this problem today. We checked our UI and deleted a user with conditions and all worked !

Thx for your comment which guide us to this solution ;)

[billyfoss]

We are seeing this with:
Terraform v0.11.14

• provider.google v2.19.0
• provider.template v2.1.2

Is there a recommended version to upgrade to? I see 2.20.0 has breaking changes.

[billyfoss]

@danawillow FYI, I've opened a Google Support ticket as this appears to have migrated to the non-beta version and is blocking updates to our environments.

I see there were changes in https://github.com/GoogleCloudPlatform/magic-modules/pull/3003
to help with this, but I'm not sure if they will make it into any 2.x stream.

[danawillow]

@billyfoss this specific issue happens when there exists an IAM binding with a condition, but you're trying to read at a version of the provider that doesn't support reading those sorts of bindings. In order to fix it, you would either need to remove the conditional binding (so that the policy becomes a version 1 policy again) or use a version of the provider that reads version 3 policies.

The google provider can do that starting at v3.7.0. For the google-beta provider, it depends on which specific type of iam resource you're using (so for google_project_iam_* resources it's 2.20.0, but for google_service_account_iam_* it's 2.19.0, etc.). You can find out by seeing which version added support for IAM conditions on that specific resource in the CHANGELOG.

Does any of that work for you? I also want to bring up the Provider Versions guide in case you have hesitancies about using the beta provider, since I think that page addresses some of the concerns that people have.

[billyfoss]

@danawillow I tried moving to 2.20.x and was reminded of https://github.com/terraform-providers/terraform-provider-google/issues/4951 which is why I pinned to 2.19.0.

[danawillow]

Hmm ok. Are you using both google and google-beta versions of the provider at the moment? You could pin them to different versions and choose which one to use on a resource-by-resource basis.

[billyfoss]

We have references to both providers, but currently only using the “google” provider. I’ve used your advice and configured the iam resources to use the google beta provider and it seemed to resolve the issue in one environment. I’m debugging why it is still failing in the original environment. I’ll update tomorrow.

[billyfoss]

I was still seeing the issue because I had a resource that was going to be removed, so it was not getting updated with the google-beta provider. Terraform was still refreshing its state with the google provider.

In summary, the advice from @danawillow worked great to get us past this.
Include both google and google-beta providers.

provider "google" {
  version = "~> 2.19.0"
}

provider "google-beta" {
  version = "~> 2.20"
}

For resources that need to handle the new IAM Policy Version 3, specific the google-beta provider.

resource "google_project_iam_member" "all-compute-instanceAdmin" {
  provider = "google-beta"

Other resources will default to the google provider. This prevents our load balancer configuration from being impacted by https://github.com/terraform-providers/terraform-provider-google/issues/4951 introduced in 2.20.x.

If you have an IAM based resource that is being deleted, you may need to manually remove it and remove it from the tfstate file as Terraform will continue to use the google 2.19 provider for state refresh.

terraform state rm module.roles.google_project_iam_member.editor

Thank you for all the help in getting past this.

[danawillow]

Glad to hear that it worked! A couple additional notes:

• I'd recommend explicitly specifying the 'google' provider for the ones that you want that one to be on rather than relying on it being the one chosen by default, just as a precaution
• I believe that if you were to do a terraform refresh with the changed provider/version before removing it from your config for the delete, then it would delete with the correct version

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!