Terraform-provider-google: Error: Provider produced inconsistent result after apply at GCP project + SA creation

[mldmld68]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave _+1_ or _me too_ comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform version: 0.12.24
terraform-provider-google-beta_v3.21.0_x5
"terraform-provider-google-beta_v3.21.0_x5"
"terraform-provider-google_v3.21.0_x5"
"terraform-provider-null_v2.1.2_x4"
"terraform-provider-random_v2.2.1_x4"
"terraform-provider-template_v2.1.2_x4"

Affected Resource(s)

The error is around service account but don't know which resource exactly. The code creates 2 SA and may fail for one or other SA. Or not fail at all

• google_service_account

Terraform Configuration Files

# Création du SA ressources à utiliser par les ressources runner
resource "google_service_account" "project_runner" {
  account_id   = "sa-${var.project_name}-infra"
  display_name = "sa-${var.project_name}-infra"
  project      = google_project.main.project_id
}

# Création du SA application 
resource "google_service_account" "appli_sa" {
  account_id   = "sa-${var.project_name}-appli"
  display_name = "sa-${var.project_name}-appli"
  project      = google_project.main.project_id
}

# Récupération des informations sur le subnet auquel sera rattaché le projet
data "google_compute_subnetwork" "subnet" {
  count      = var.connection_to_host_project == "true" ? 1 : 0
  name       = "${var.subnet_prefix_name}${var.host_project_instance}${var.subnet_suffix_name}${var.subnet_index}"
  project    = var.host_project_id
  depends_on = [google_project_service.main]
}

# Donne le droit au SA ressources du projet de connecter des ressources au subnet du host project. Par exemple un GCE
resource "google_compute_subnetwork_iam_member" "default_sa" {
  count      = var.connection_to_host_project == "true" ? 1 : 0
  subnetwork = data.google_compute_subnetwork.subnet[0].name
  role       = "roles/compute.networkUser"
  member     = "serviceAccount:${google_service_account.project_runner.email}"
  project    = var.host_project_id
}

# Récupération des informations sur d'éventuels subnets supplémentaires auquel le SA devra avoir le droit networkuser
data "google_compute_subnetwork" "get_supplemental_subnets" {
  count      = (var.connection_to_host_project == "true" ? 1 : 0) * var.sa_folder_supplemental_subnets
  name       = "${var.subnet_prefix_name}${var.host_project_instance}-${var.sia}-${element(split("-", var.project_name), 0)}${element(split("-", var.project_name), 1)}${var.subnet_suffix_name}${count.index + 1}"
  project    = var.host_project_id
  depends_on = [google_project_service.main]
}

# Donne le droit au SA ressources du projet de connecter des ressources à d'éventuels autres subnets du host project dans le même VPC. Par exemple un GCE
resource "google_compute_subnetwork_iam_member" "supplemental_subnets" {
  count = (var.connection_to_host_project == "true" ? 1 : 0) * var.sa_folder_supplemental_subnets
  subnetwork = element(
    data.google_compute_subnetwork.get_supplemental_subnets.*.name,
    count.index,
  )
  role    = "roles/compute.networkUser"
  member  = "serviceAccount:${google_service_account.project_runner.email}"
  project = var.host_project_id
}

# Donne le droit au compte google_api_agent du projet de connecter des ressources au subnet du host project. Ce compte sert pour les services managés de google
resource "google_compute_subnetwork_iam_member" "google_api_agent" {
  count      = var.connection_to_host_project == "true" ? 1 : 0
  subnetwork = data.google_compute_subnetwork.subnet[0].name
  role       = "roles/compute.networkUser"
  member     = "serviceAccount:${google_project.main.number}@cloudservices.gserviceaccount.com"
  project    = var.host_project_id
}

# Donne le droit au SA ressources du projet de lire les VPC du host project : necessaire pour CloudSQL private et Load Balancer 
resource "google_project_iam_member" "network-get" {
  count   = var.connection_to_host_project == "true" ? 1 : 0
  project = var.host_project_id
  role    = "organizations/${var.org_id}/roles/svc.project.rights.on.hostprj" # Contains compute-network-get-for-hostprojects
  member  = "serviceAccount:${google_service_account.project_runner.email}"
}

# Attribue les droits au SA de création des ressources sur le service projet
resource "google_project_iam_member" "editor" {
  count   = length(var.sa_folder_iam_roles)
  project = google_project.main.project_id
  role    = var.sa_folder_iam_roles[count.index]
  member  = "serviceAccount:${google_service_account.project_runner.email}"
}

# Cre les SA décrits dans la variable sas_map du .tfvars. S'ils apparaissent plusieurs fois car ils ont plusieurs roles, les doublons sont ignores a cette etape 
resource "google_service_account" "sas" {
  count        = length(var.sas_nickname_list)
  project      = google_project.main.project_id
  account_id   = "sa-${var.project_name}-${var.sas_nickname_list[count.index]}"
  display_name = "sa-${var.project_name}-${var.sas_nickname_list[count.index]}"
}

# Attribue les droits aux SA décrits dans la variable sas_map du .tfvars
resource "google_project_iam_member" "sas" {
  count      = length(var.sas_map)
  project    = google_project.main.project_id
  role       = var.sas_map[count.index]["role"]
  member     = "serviceAccount:sa-${var.project_name}-${var.sas_map[count.index]["sa_nickname"]}@${google_project.main.project_id}.iam.gserviceaccount.com"
  depends_on = [google_service_account.sas]
}

# Autorise si besoin les "customer-managed encryption keys" pour le service pubsub (cf. https://cloud.google.com/pubsub/docs/cmek)
resource "google_project_iam_member" "pubsub_cmek" {
  count   = var.pubsub_crypt_enable ? 1 : 0
  project = google_project.main.project_id
  role    = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
  member  = "serviceAccount:service-${google_project.main.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}

### Add IAM roles to a gcp managed service account
resource "google_project_iam_member" "extra_cloud_functions_permissions" {
  count   = length(var.extra_cloud_functions_service_account_permissions)
  project = google_project.main.project_id
  role    = var.extra_cloud_functions_service_account_permissions[count.index]
  member  = "serviceAccount:service-${google_project.main.number}@gcf-admin-robot.iam.gserviceaccount.com"
  # Add a dependency on the project APIs to make sure the Cloud Functions API is enabled and the Service Account is created before it is granted extra permissions.
  depends_on = [google_project_service.main]

Debug Output

https://gist.github.com/mldmld68/c08c9db8383736e1df31a49cdbe10ab3

Panic Output

Expected Behavior

Terraform apply finish without error

Actual Behavior

I can get an error on 2 SA, but not everytime the same
I terraform plan+apply/terraform destroy/terraform plan+apply

Error: Provider produced inconsistent result after apply
When applying changes to google_service_account.appli_sa, provider
"registry.terraform.io/-/google" produced an unexpected new value for was
present, but now absent.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.
If one restart terraform apply, on get
Error: Error creating service account: googleapi: Error 409: Service account sa-xxxxx-appli already exists within project projects/xxxxxxx., alreadyExists
on service_account.tf line 9, in resource "google_service_account" "appli_sa":

Error: Provider produced inconsistent result after apply
When applying changes to google_service_account.project_runner, provider
"registry.terraform.io/-/google" produced an unexpected new value for was
present, but now absent.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.
ERROR: Job failed: exit code 1

Steps to Reproduce

terraform apply

1. terraform apply

Important Factoids

This job create a GCP projet, give roles to user groups and SAs.
If one destroy the faultly SA and restart the job, the terrform plan plays successfully

References

• #0000

[sebglon]

Hi,
I have the same case. I can reproduce with the initial TF file:

terraform {
  required_version = "= 0.12.23"
}

provider "google" {
  version = "~> 3.20.0"
}

resource "google_project" "project" {
  name            = "akecld-sgl-test-1"
  project_id      = "akecld-sgl-test-1"
  org_id          = "xxxxxxxxxxxxxx"
  billing_account = "xxxxxxxxxxxxx"


}

resource "google_service_account" "provisioner" {
  account_id   = "provisioner"
  display_name = "Provisioner"
  project      = google_project.project.project_id
}

resource "google_service_account_key" "perrybot" {
  service_account_id = google_service_account.provisioner.name
  public_key_type    = "TYPE_X509_PEM_FILE"
  depends_on= [
    google_service_account.provisioner
  ]
}

tfstate and debug log:
https://gist.github.com/sebglon/4f3111f07fca646280170d50444fcf82

[sebglon]

Only arrive on fresh project creation

[sebglon]

idem with provider 3.17 and 3.21

[benjamin-kaiser]

I encountered this issue yesterday. On first Apply I saw that the resources that depended on the service account failed. A subsequent terraform plan showed that the service account needed to be created, running apply again gave the error service account alreadyExists.

terraform version: 0.11.14
provider version: 2.20

when investigating I found:

• Google Console was not showing the Service Account
• gCloud was showing the Service Account

I resolved by:

• using gCloud to delete the Service Account
• running terraform apply again

Looks related to: https://github.com/terraform-providers/terraform-provider-google/issues/6362

[dtan4]

I met the same issue with Terraform 0.11.14/0.12.24 and Google Provider 2.20.1.

I wrote a minimal code to reproduce this behavior, which repeats create a new service account -> get it -> delete it using raw IAM API client.
https://gist.github.com/dtan4/a2081e75ff8e4737c872e80fd9e6f9e6

However, actually it's still hard to reproduce... when I started running this script, I sometimes encountered the following error which looks the same as this issue. However, after running several times, I couldn't see the error again...

2020/05/14 21:30:17 googleapi: Error 404: Service account projects/***/serviceAccounts/foobar@***.iam.gserviceaccount.com does not exist., notFound

[sebglon]

After some test it's an async data on google.
The service account API response 404 not found during ~ 30s after the create 200.
This is only available after a fresh project creation.

[bharathkkb]

I have also started seeing this same error with a TF file similar to https://github.com/terraform-providers/terraform-provider-google/issues/6377#issuecomment-628573840

provider "google" {
  version = "~> 3.20.0"
}

resource "google_project" "project" {
  name            = "foo"
  project_id      = "foo"
  billing_account = ""
  folder_id       = ""
}

resource "google_service_account" "foo" {
  account_id   = "example"
  display_name = "example"
  project      = google_project.project.project_id
}

version:
Terraform v0.12.20
provider.google v3.20.0

logs: https://gist.github.com/bharathkkb/936c43d253ffc01a353bd16a6c2be3b0

[c2thorn]

Hi all,
A longer wait time is needed in between creating a project & service account. A fix is underway, but there should be an immediate workaround provided in the resource documentation for those that could use it. Using the null_resource, you can delay the creation of the google_service_account for any amount of time.
Here's what it would look like for @bharathkkb's example:

resource "google_project" "project" {
  name            = "foo"
  project_id      = "foo"
  billing_account = ""
  folder_id       = ""
}

resource "null_resource" "delay" {
  provisioner "local-exec" {
    command = "sleep 60"
  }
  triggers = {
    "project" = "${google_project.project.id}"
  }
}

resource "google_service_account" "foo" {
  account_id   = "example"
  display_name = "example"
  project      = google_project.project.project_id
  depends_on = [null_resource.delay]
}

60 seconds seems to enough so far, but it is easily adjustable. Hopefully this helps anyone unblock themselves before waiting for the fix to be released, or using an older version.

[mldmld68]

Hi,
Tried to apply this workaround without success. Got issue on both service accounts.
Tried to add a depends_on section on the null ressource with project creation. It shows the delay do not fix the issue because it do not prevent other ressources of being created like Service Accounts.

But I noticed the sa creation begin while the project resource is still activating api.

So added depends_on section with depends_on = [google_project_service.main] in service account creation and the issue do not appear anymore. I tried only twice, but before I got the issue everytime.

[c2thorn]

Hi @mldmld68, thanks for the input, and I'm glad you were able to figure out a workaround! So you had google_project_service in your initial configuration as well? If it isn't too much hassle, do you mind sharing an example configuration for clarity?

[mldmld68]

Hi @mldmld68, thanks for the input, and I'm glad you were able to figure out a workaround! So you had google_project_service in your initial configuration as well? If it isn't too much hassle, do you mind sharing an example configuration for clarity?

Hi @c2thorn, here is an extract of the project creation :

resource "google_project" "main" {
  name       = var.project_name
  project_id = "${var.project_name}
  folder_id  = "folders/${var.folder_id}"

...
}

resource "google_project_service" "main" {
  count   = length(local.project_services)
  project = google_project.main.project_id
  service = local.project_services[count.index]
  disable_on_destroy = false
}

resource "google_service_account" "project_runner" {
  account_id   = "sa-${var.project_name}-infra"
  display_name = "sa-${var.project_name}-infra"
  project      = google_project.main.project_id
  depends_on   = [google_project_service.main]
}

[benjamin-kaiser]

Could this be related to Cloud Console and gCloud also not being in agreement as to whether the service account was created? I have concerns with this timeout workaround as I have waited 3 days for Console to show the info that gCloud has with no luck.
I have filed a ticket (https://issuetracker.google.com/issues/156578013) I believe they are related.

[c2thorn]

@benjamin-kaiser The root problem is certainly service account creation, unfortunately all the Terraform provider can do is use what the information the API returns and wait for eventual consistency.

[unclebene]

The same issue even after provider update to the version 3.22.0

[c2thorn]

Hi @unclebene
The fix just missed the timing for the v3.22.0 release, so it won't officially show until v3.23.0. If needed, you could always build from the commits generated by the pull request:
TPG
TPGB

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!