Terraform-provider-google: google_project_services apply toggling enabling and disabling services every run

[goobysnack]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.11.11
google v2.16.0

Affected Resource(s)

google_project_services

Terraform Configuration Files

locals {
  default_services = [
    "admin.googleapis.com",                 // Admin SDK
    "audit.googleapis.com",                 // Audit API
    "compute.googleapis.com",               // Compute Engine API
    "cloudresourcemanager.googleapis.com",  // Google Cloud Resource Manager
    "cloudmonitoring.googleapis.com",    // Cloud Monitoring API
    "monitoring.googleapis.com",         // Stackdriver Monitoring API
    "logging.googleapis.com",               // Stackdriver Logging API
    "oslogin.googleapis.com",               // Cloud OS Login API
    "pubsub.googleapis.com",                // Cloud Pub/Sub API
    "stackdriver.googleapis.com",           // Stackdriver API
    "storage-api.googleapis.com",           // Google Cloud Storage JSON API
    "iam.googleapis.com",                   // IAM API
    "iamcredentials.googleapis.com",        // IAM Credientials API
    "bigquery-json.googleapis.com",      // BigQuery API
    "container.googleapis.com",          // Kubernetes Engine API
    "containerregistry.googleapis.com",  // Container Registry API
    "deploymentmanager.googleapis.com",  // Google Cloud Deployment Manager V2 API
    "replicapool.googleapis.com",        // Compute Engine Instance Group Manager API
    "replicapoolupdater.googleapis.com", // Google Compute Engine Instance Group Updater API
    "resourceviews.googleapis.com",      // Google Compute Engine Instance Groups API
  ]
}
<project creation resource removed>

resource "google_project_services" "project" {
  project = "${google_project.project.id}"

  services = "${local.default_services}"
}

Debug Output

Panic Output

Expected Behavior

No changes to the list of services, and no changes to the enabled services should occur.

Actual Behavior

The enabled services toggles every single time apply is run.
Run 1 -

module.gcp_project.google_project_services.project: Modifying... (ID: <projectID>)
  services.#:          "30" => "20"

Run 2 -

module.gcp_project.google_project_services.project: Modifying... (ID: <projectID>)
  services.#:          "15" => "20"

Run 3 -

module.gcp_project.google_project_services.project: Modifying... (ID: <projectID>)
  services.#:          "21" => "20"

Run 4 -

module.gcp_project.google_project_services.project: Modifying... (ID: <projectID>)
  services.#:          "15" => "20"

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[goobysnack]

Added note - the state file is inaccurate. I verified with a show of the module list and it's definitely incorrect.

[goobysnack]

Some troubleshooting. This isn't good at all. Basically this is disruptive.
Before and after without ANY changes to the list of services (tf file).

gcloud services list --enabled | grep -c eapis
31

gcloud services list --enabled | grep -c eapis
20

The terraform plan shows going from 30 to 27, but removes 11.

  ~ module.gcp_project.google_project_services.project
      services.#:          "30" => "27"
      services.1239746899: "bigquerystorage.googleapis.com" => ""
      services.133405307:  "storage-component.googleapis.com" => ""
      services.1560437671: "iam.googleapis.com" => "iam.googleapis.com"
      services.1568433289: "oslogin.googleapis.com" => "oslogin.googleapis.com"
      services.1589981880: "vpcaccess.googleapis.com" => "vpcaccess.googleapis.com"
      services.1610229196: "bigquery-json.googleapis.com" => "bigquery-json.googleapis.com"
      services.1712537408: "containerregistry.googleapis.com" => "containerregistry.googleapis.com"
      services.1792583891: "datalabeling.googleapis.com" => "datalabeling.googleapis.com"
      services.1904024597: "runtimeconfig.googleapis.com" => "runtimeconfig.googleapis.com"
      services.2117420113: "pubsub.googleapis.com" => "pubsub.googleapis.com"
      services.2240314979: "compute.googleapis.com" => "compute.googleapis.com"
      services.2244543419: "tpu.googleapis.com" => "tpu.googleapis.com"
      services.2966512281: "deploymentmanager.googleapis.com" => "deploymentmanager.googleapis.com"
      services.2976624689: "cloudmonitoring.googleapis.com" => "cloudmonitoring.googleapis.com"
      services.3010261123: "replicapool.googleapis.com" => "replicapool.googleapis.com"
      services.3067626773: "redis.googleapis.com" => "redis.googleapis.com"
      services.3075019877: "replicapoolupdater.googleapis.com" => "replicapoolupdater.googleapis.com"
      services.3077910291: "resourceviews.googleapis.com" => "resourceviews.googleapis.com"
      services.323125032:  "cloudtrace.googleapis.com" => ""
      services.3237295688: "monitoring.googleapis.com" => "monitoring.googleapis.com"
      services.3327360159: "stackdriver.googleapis.com" => "stackdriver.googleapis.com"
      services.3355193353: "logging.googleapis.com" => "logging.googleapis.com"
      services.3604209692: "iamcredentials.googleapis.com" => "iamcredentials.googleapis.com"
      services.3612242986: "ml.googleapis.com" => "ml.googleapis.com"
      services.3644083179: "cloudresourcemanager.googleapis.com" => "cloudresourcemanager.googleapis.com"
      services.3740470850: "container.googleapis.com" => "container.googleapis.com"
      services.386687109:  "audit.googleapis.com" => "audit.googleapis.com"
      services.3875785048: "storage-api.googleapis.com" => "storage-api.googleapis.com"
      services.3899772697: "cloudfunctions.googleapis.com" => "cloudfunctions.googleapis.com"
      services.960186248:  "admin.googleapis.com" => "admin.googleapis.com"

[goobysnack]

The state file shows 30.

services.#          = 30
services.1239746899 = bigquerystorage.googleapis.com
services.133405307  = storage-component.googleapis.com
services.1560437671 = iam.googleapis.com
services.1568433289 = oslogin.googleapis.com
services.1589981880 = vpcaccess.googleapis.com
services.1610229196 = bigquery-json.googleapis.com
services.1712537408 = containerregistry.googleapis.com
services.1792583891 = datalabeling.googleapis.com
services.1904024597 = runtimeconfig.googleapis.com
services.2117420113 = pubsub.googleapis.com
services.2240314979 = compute.googleapis.com
services.2244543419 = tpu.googleapis.com
services.2966512281 = deploymentmanager.googleapis.com
services.2976624689 = cloudmonitoring.googleapis.com
services.3010261123 = replicapool.googleapis.com
services.3067626773 = redis.googleapis.com
services.3075019877 = replicapoolupdater.googleapis.com
services.3077910291 = resourceviews.googleapis.com
services.323125032  = cloudtrace.googleapis.com
services.3237295688 = monitoring.googleapis.com
services.3327360159 = stackdriver.googleapis.com
services.3355193353 = logging.googleapis.com
services.3604209692 = iamcredentials.googleapis.com
services.3612242986 = ml.googleapis.com
services.3644083179 = cloudresourcemanager.googleapis.com
services.3740470850 = container.googleapis.com
services.386687109  = audit.googleapis.com
services.3875785048 = storage-api.googleapis.com
services.3899772697 = cloudfunctions.googleapis.com
services.960186248  = admin.googleapis.com

[goobysnack]

Should the state file change? These were successive issues of the command, without any changes to the project or run of terraform apply

terraform state show module.gcp_project.google_project_services.project
...
services.#          = 19

terraform state show module.gcp_project.google_project_services.project
...
services.#          = 30

[ndmckinley]

Is this still happening for you? Could you grab some debug logs? I'll try to reproduce on my end, but our unit tests cover this case and don't show a regression.

[ndmckinley]

Okay, I can reproduce with your particular service mix. I'll look into it.

[goobysnack]

Looks like these three may be at issue:
storage-component.googleapis.com
bigquerystorage.googleapis.com
cloudtrace.googleapis.com

for this particular list. I suppose the bigger issue is that dependency services that are not configured but being enabled are not being returned as a warning (to update the config). When I added these to the config list of services, this stopped.

[ndmckinley]

Yep, that's what I'm seeing as well. :) https://github.com/GoogleCloudPlatform/magic-modules/pull/2397 will address the issue for bigquerystorage.googleapis.com - let me check on those other two.

And yes, we are definitely playing whack-a-mole on these. A warning is a good idea - but it's not easy to detect this state the way the resource is now. I can look into that.

[goobysnack]

I wish I had the Fu to help more. I bet @cblecker has the requisite Fu.

[ndmckinley]

We've looked into this some more and, boy! This resource is just absolutely a minefield for users. The issue of dependent APIs is rough. If you turn on API A, and API A turns on API B, then API B turns on API C ... this resource is always going to give you a diff. Even if we keep a mapping of API-to-API, as https://github.com/terraform-providers/terraform-provider-google/pull/4560 suggests, these API dependencies change without warning, and this resource will break regularly.

GoogleCloudPlatform/magic-modules#2397 won't help - it's actually a breaking change for users who already have those resources in their configs, and so we couldn't include it until 3.0.0.

We are seriously considering deleting this resource in 3.0.0, instead of continuing to let users run into problems like yours. Unfortunately, adding those three APIs to the config is the only workaround available right now.

We will continue to discuss.

[ndmckinley]

I encourage you and anyone else with a similar problem to use https://github.com/terraform-google-modules/terraform-google-project-factory/tree/master/modules/project_services#project-api-activation instead!

[goobysnack]

I encourage you and anyone else with a similar problem to use https://github.com/terraform-google-modules/terraform-google-project-factory/tree/master/modules/project_services#project-api-activation instead!

If I'm reading that module correctly, you're iterating on each API one by one by count in your list. How will that capture dependent services that are enabled in the state/config? How do they reconcile?

[ndmckinley]

That module does not try to enforce that _only_ the listed services are active. It tried to enforce that _at least_ the listed services are active, which is the only approach truly compatible with GCP's dependency system.

[goobysnack]

OIC - this is the important bit I understand now
When a non-authoritative resource becomes authoritative, any services not in the list are disabled.

So, the singular resource here never disables running services, unless it's explicitly in the TF configuration and is removed.

[ndmckinley]

Exactly!

[ndmckinley]

Can you report back on your experience with that module? I'm strongly inclined to deprecate this resource and remove it in 3.0.0, but I want to make sure that the replacement is a good one.

[goobysnack]

yup, gimme an hour :D

[ndmckinley]

Great! That'll be https://github.com/GoogleCloudPlatform/magic-modules/pull/2402 and https://github.com/GoogleCloudPlatform/magic-modules/pull/2403 if you find it serviceable and I don't hear any other objections.

[goobysnack]

Not sure I want to muddy those pages just yet. FYI, one issue I found. It worked fine to create the new state entries, however, if you remove a service, it moves the entry in the state file. I removed redis as a test and this happened:

Terraform will perform the following actions:

-/+ module.gcp_project.google_project_service.project_services[18] (new resource required)
      id:                         "testproj/redis.googleapis.com" => <computed> (forces new resource)
      disable_dependent_services: "false" => "false"
      disable_on_destroy:         "true" => "true"
      project:                    "testproj" => "testproj"
      service:                    "redis.googleapis.com" => "replicapool.googleapis.com" (forces new resource)

-/+ module.gcp_project.google_project_service.project_services[19] (new resource required)
      id:                         "testproj/replicapool.googleapis.com" => <computed> (forces new resource)
      disable_dependent_services: "false" => "false"
      disable_on_destroy:         "true" => "true"
      project:                    "testproj" => "testproj"
      service:                    "replicapool.googleapis.com" => "replicapoolupdater.googleapis.com" (forces new resource)

-/+ module.gcp_project.google_project_service.project_services[20] (new resource required)
      id:                         "testproj/replicapoolupdater.googleapis.com" => <computed> (forces new resource)
      disable_dependent_services: "false" => "false"
      disable_on_destroy:         "true" => "true"
      project:                    "testproj" => "testproj"
      service:                    "replicapoolupdater.googleapis.com" => "resourceviews.googleapis.com" (forces new resource)

-/+ module.gcp_project.google_project_service.project_services[21] (new resource required)
      id:                         "testproj/resourceviews.googleapis.com" => <computed> (forces new resource)
      disable_dependent_services: "false" => "false"
      disable_on_destroy:         "true" => "true"
      project:                    "testproj" => "testproj"
      service:                    "resourceviews.googleapis.com" => "runtimeconfig.googleapis.com" (forces new resource)

-/+ module.gcp_project.google_project_service.project_services[22] (new resource required)
      id:                         "testproj/runtimeconfig.googleapis.com" => <computed> (forces new resource)
      disable_dependent_services: "false" => "false"
      disable_on_destroy:         "true" => "true"
      project:                    "testproj" => "testproj"
      service:                    "runtimeconfig.googleapis.com" => "stackdriver.googleapis.com" (forces new resource)

-/+ module.gcp_project.google_project_service.project_services[23] (new resource required)
      id:                         "testproj/stackdriver.googleapis.com" => <computed> (forces new resource)
      disable_dependent_services: "false" => "false"
      disable_on_destroy:         "true" => "true"
      project:                    "testproj" => "testproj"
      service:                    "stackdriver.googleapis.com" => "storage-api.googleapis.com" (forces new resource)

-/+ module.gcp_project.google_project_service.project_services[24] (new resource required)
      id:                         "testproj/storage-api.googleapis.com" => <computed> (forces new resource)
      disable_dependent_services: "false" => "false"
      disable_on_destroy:         "true" => "true"
      project:                    "testproj" => "testproj"
      service:                    "storage-api.googleapis.com" => "tpu.googleapis.com" (forces new resource)

-/+ module.gcp_project.google_project_service.project_services[25] (new resource required)
      id:                         "testproj/tpu.googleapis.com" => <computed> (forces new resource)
      disable_dependent_services: "false" => "false"
      disable_on_destroy:         "true" => "true"
      project:                    "testproj" => "testproj"
      service:                    "tpu.googleapis.com" => "vpcaccess.googleapis.com" (forces new resource)

  - module.gcp_project.google_project_service.project_services[26]

Plan: 8 to add, 0 to change, 9 to destroy.

Basically the enumeration module.gcp_project.google_project_service.project_services[x] is re-calculated everytime. If this is expected and nonimpactful (except for the removed service) I'll move on and make it so.

[goobysnack]

Also, when you do apply it errors out. I had to delete the module in the state file to get it to run...but it does work, meaning it doesn't try to manage dependencies and it fails when there's an attempt to do so.

Error: Error applying plan:

2 error(s) occurred:

* module.gcp_project.google_project_service.project_services[18] (destroy): 1 error(s) occurred:

* google_project_service.project_services.18: Error reading Project Service testproj/replicapool.googleapis.com: Error disabling service "replicapool.googleapis.com" for project "testproj": googleapi: Error 400: The service replicapool.googleapis.com is depended on by the following active services: resourceviews.googleapis.com,replicapoolupdater.googleapis.com; Please specify disable_dependent_services=true if you want to proceed with disabling all services., failedPrecondition
* module.gcp_project.google_project_service.project_services[23] (destroy): 1 error(s) occurred:

* google_project_service.project_services.23: Error reading Project Service testproj/storage-api.googleapis.com: Error disabling service "storage-api.googleapis.com" for project "testproj": googleapi: Error 400: The service storage-api.googleapis.com is depended on by the following active services: cloudfunctions.googleapis.com,resourceviews.googleapis.com,replicapoolupdater.googleapis.com,container.googleapis.com,containerregistry.googleapis.com,replicapool.googleapis.com; Please specify disable_dependent_services=true if you want to proceed with disabling all services., failedPrecondition

[ndmckinley]

Hm, that does seem like an issue! I'll check on that with the team that created that module.

[sebglon]

Hi,

I have au bug with the same bigquery service:

terraform version:

terraform version
Terraform v0.11.14
+ provider.aws v2.43.0
+ provider.external v1.2.0
+ provider.google v2.20.1
+ provider.local v1.4.0
+ provider.null v2.1.2
+ provider.random v2.2.1

and the definition:

resource "google_project_service" "bigquerystorage" {
  project                    = "${google_project.project.number}"
  service                    = "bigquerystorage.googleapis.com"
  disable_dependent_services = true
  depends_on                 = ["google_project_service.bigquerystorage"]
}

resource "google_project_service" "bigquery" {
  project                    = "${google_project.project.number}"
  service                    = "bigquery.googleapis.com"
  disable_dependent_services = true
}

The terraform destroy not work i have this error:

Error reading Project Service 90037394926/bigquery.googleapis.com: Error disabling service "bigquery.googleapis.com" for project "90037394926": googleapi: Error 400: The service bigquery.googleapis.com is depended on by the following active service: bigquerystorage.googleapis.com; Please specify disable_dependent_services=true if you want to proceed with disabling all services., failedPrecondition

I think it's linked

[danawillow]

Hey @sebglon, thanks for opening a new issue about the problem you're currently facing. Since this one is about a resource that has been removed in the later versions of the provider, I'm going to close it.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!