Terraform-provider-google: If an instance group managed by a GKE node pool is manually removed, terraform plan fails to refresh

[misha-ridge]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• If an issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to "hashibot", a community member has claimed the issue already.

Terraform Version

Terraform v0.12.6
+ provider.google v2.13.0
+ provider.ignition v1.1.0
+ provider.null v2.1.2

Affected Resource(s)

• google_container_node_pool

Terraform Configuration Files

Basic GKE cluster, such as described here: https://www.terraform.io/docs/providers/google/r/container_cluster.html

Expected Behavior

If an instance group managed by node pool is removed, node pool should be replaced or healed in some other way.

Actual Behavior

terraform plan displays Error: Error reading instance group manager returned as an instance group URL: "googleapi: Error 404: The resource 'projects/$proj/zones/europe-west1-b/instanceGroupManagers/gke-$cluster-default-af15db45-grp' was not found, notFound" and exits with non-zero code.

Steps to Reproduce

1. Create a GKE cluster using Terraform.
2. Delete one of instance groups backing a node pool
3. terraform plan

The error does not go away until the node pool is removed manually.

[jacobstr]

Is there a workaround available for this? I've tried to remove the google_container_cluster state as well as the google_container_node_pool state with no success. _Something_ is causing a lookup of an old instanceGroupManager.

I've looked at my terraform state as well and cannot find evidence of e.g. gke-$cluster-default-af15db45-grp.

[danawillow]

The IGM URL that it's looking up comes directly from the cluster or node pool response from the GCP API, so it's surprising that the groups are still there if that's removed. Do either of you have debug logs (env var TF_LOG=DEBUG) that you could share from a run that had this error? Those will have the api requests/responses sent to/from GCP, and should have more data on where those URLs are coming from.

[misha-ridge]

@jacobstr This seems to be a different issue.

[danawillow]

Agreed, that one seems to be regarding node taints. If it's causing issues still then I'd recommend filing a new issue with the bug template fully filled out (config, plan output, debug logs), keeping in mind that any fixes that we make on our side (if it's determined to be a bug in the provider) will likely occur in the 3.X series of the provider.

For this one specifically, I'm going to go ahead and put the waiting-response label back on until we're able to see debug logs from a repro.

[jacobstr]

Deleted my earlier comment - indeed a different issue.

[jacobstr]

So - I was able to reproduce this by modifying the node locations on an existing google_container_cluster.

The output below is fairly butchered from the original output due to attempts at obscuring/redacting details. I hope I've left the substantiative points intact.

• It looks like the _missing_ InstanceGroupManager is showing up in some cluster-level API response within the nodePools attribute.
• I had tried (unsuccessfully) to surgically modify/push the remote terraform state to remove any references to *-13553402-grp. The API response continued to show the destroyed group.
• Deleting affected instances groups does indeed seem to unwedge things. That'd be a sad place to land in a production cluster. You'd have to clone the node pool out-of-band from terraform (because terraform crashes on the affected workspace), non-disruptively shift pods around, then possibly re-import state.

<Aside>: If anyone has advice on:

• Redacting sensitive data with TF_LOG enabled.
• De-interleaving std/stdout from the various terraform goroutines / plugins.

I'm all 👂.

<SNIP>

2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: 2020/04/16 00:41:23 [DEBUG] Google API Response Details:
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: ---[ RESPONSE ]--------------------------------------
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: HTTP/1.1 200 OK
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Connection: close
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Transfer-Encoding: chunked
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Cache-Control: private
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Content-Type: application/json; charset=UTF-8
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Date: Thu, 16 Apr 2020 00:41:23 GMT
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Server: ESF
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Vary: Origin
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Vary: X-Origin
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Vary: Referer
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: X-Content-Type-Options: nosniff
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: X-Frame-Options: SAMEORIGIN
2020-04-16T00:41:23.736Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: X-Xss-Protection: 0
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: 3522
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: {
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:  "name": "bob-gk02",
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:  "description": "bobs test cluster",
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:  "initialNodeCount": 1,
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:  "nodeConfig": {
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "machineType": "n1-standard-4",
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "diskSizeGb": 32,
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "oauthScopes": [
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "https://www.googleapis.com/auth/cloud-platform"
2020-04-16T00:41:23.737Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   ],
2020-04-16T00:41:23.740Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:  "masterAuth": {<redacted>},
2020-04-16T00:41:23.741Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:  "subnetwork": "bob-backbone-prod",
2020-04-16T00:41:23.741Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:  "nodePools": [
2020-04-16T00:41:23.744Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   {
2020-04-16T00:41:23.744Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "name": "bob-gk02-standardv2-pool",
2020-04-16T00:41:23.744Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "config": {
2020-04-16T00:41:23.744Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "machineType": "n1-standard-16",
2020-04-16T00:41:23.744Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "diskSizeGb": 64,
2020-04-16T00:41:23.744Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "oauthScopes": [
2020-04-16T00:41:23.744Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "https://www.googleapis.com/auth/cloud-platform"
2020-04-16T00:41:23.744Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     ],
2020-04-16T00:41:23.744Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "metadata": {
2020-04-16T00:41:23.744Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "disable-legacy-endpoints": "true",
2020-04-16T00:41:23.744Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "managed_by": "geko",
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "cluster_id": "bob-gk02"
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     },
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "imageType": "COS",
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "labels": {
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "gkeo.bob.com/cidr_size": "26",
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "gkeo.bob.com/pool_type": "worker",
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "gkeo.bob.com/worker_type": "standard",
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "node_pool": "standard",
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "cluster_id": "bob-gk02"
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     },
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "tags": [
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "bob-gk02",
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "standard",
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "bob-gk02-standard"
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     ],
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "serviceAccount": "[email protected]",
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "diskType": "pd-ssd",
rm-provider-google-beta_v3.17.0_x5:     "shieldedInstanceConfig": {
2020-04-16T00:41:23.745Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:      "enableIntegrityMonitoring": true
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     }
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    },
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "initialNodeCount": 1,
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "autoscaling": {
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "enabled": true,
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "maxNodeCount": 100
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    },
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "management": {
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "autoUpgrade": true,
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "autoRepair": true
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    },
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "maxPodsConstraint": {
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "maxPodsPerNode": "32"
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    },
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "podIpv4CidrSize": 26,
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "locations": [
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "us-central1-b",
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "us-central1-f",
in.terraform-provider-google-beta_v3.17.0_x5:     "us-central1-c",
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "us-central1-a"
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    ],
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "selfLink": "https://container.googleapis.com/v1beta1/projects/bob-gketest-prod/locations/us-central1/clusters/bob-gk02/nodePools/bob-gk02-standardv2-pool",
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "version": "1.14.10-gke.32",
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "instanceGroupUrls": [
2020-04-16T00:41:23.746Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-c/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-81dd898a-grp",
2020-04-16T00:41:23.748Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-f/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-13553402-grp",
2020-04-16T00:41:23.748Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-a/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-6ce6878b-grp",
2020-04-16T00:41:23.748Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-b/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-571015c6-grp"
2020-04-16T00:41:23.748Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    ],
2020-04-16T00:41:23.748Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:    "status": "RUNNING"
2020-04-16T00:41:23.748Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   }
G] plugin.terraform-provider-google-beta_v3.17.0_x5:  ],
2020-04-16T00:41:23.748Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:  "locations": [
2020-04-16T00:41:23.748Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "us-central1-b",
2020-04-16T00:41:23.748Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "us-central1-f",
2020-04-16T00:41:23.748Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "us-central1-c",
2020-04-16T00:41:23.748Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "us-central1-a"
2020-04-16T00:41:23.749Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:  ],
2020-04-16T00:41:23.757Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:  "privateCluster": true,
2020-04-16T00:41:23.760Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:  "instanceGroupUrls": [
2020-04-16T00:41:23.761Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-c/instanceGroupManagers/gke-bob-gk02-bob-gk02-ingress-82280b62-grp",
2020-04-16T00:41:23.761Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-f/instanceGroupManagers/gke-bob-gk02-bob-gk02-ingress-774299dd-grp",
2020-04-16T00:41:23.761Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-a/instanceGroupManagers/gke-bob-gk02-bob-gk02-ingress-b6d05a17-grp",
2020-04-16T00:41:23.761Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-b/instanceGroupManagers/gke-bob-gk02-bob-gk02-ingress-3048d276-grp",
2020-04-16T00:41:23.761Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-c/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-81dd898a-grp",
2020-04-16T00:41:23.761Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-f/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-13553402-grp",
2020-04-16T00:41:23.761Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-a/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-6ce6878b-grp",
2020-04-16T00:41:23.761Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "https://www.googleapis.com/compute/v1/projects/bob-gketest-prod/zones/us-central1-b/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-571015c6-grp"


<SNIP>


2020-04-16T00:41:26.055Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5: 2020/04/16 00:41:26 [DEBUG] Google API Response Details:
2020-04-16T00:41:26.056Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5: ---[ RESPONSE ]--------------------------------------
2020-04-16T00:41:26.056Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5: HTTP/1.1 200 OK
2020-04-16T00:41:26.056Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5: Connection: close
2020-04-16T00:41:26.056Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5: Transfer-Encoding: chunked
2020-04-16T00:41:26.057Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5: Cache-Control: private
2020-04-16T00:41:26.057Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5: Content-Type: application/json; charset=UTF-8
2020-04-16T00:41:26.057Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5: Date: Thu, 16 Apr 2020 00:41:26 GMT
2020-04-16T00:41:26.057Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5: Server: ESF
2020-04-16T00:41:26.057Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5: Server-Timing: gfet4t7; dur=259
2020-04-16T00:41:26.057Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5: Vary: Origin
2020-04-16T00:41:26.055Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: 2020/04/16 00:41:26 [DEBUG] Google API Response Details:
2020-04-16T00:41:26.057Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: ---[ RESPONSE ]--------------------------------------
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: HTTP/1.1 404 Not Found
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Connection: close
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Transfer-Encoding: chunked
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Cache-Control: private
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Content-Type: application/json; charset=UTF-8
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Date: Thu, 16 Apr 2020 00:41:26 GMT
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Server: ESF
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Vary: Origin
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Vary: X-Origin
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: Vary: Referer
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: X-Content-Type-Options: nosniff
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: X-Frame-Options: SAMEORIGIN
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: X-Xss-Protection: 0
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: 1db
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: {
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   "error": {
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "code": 404,
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "message": "The resource 'projects/bob-gketest-prod/zones/us-central1-f/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-13553402-grp' was not found",
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     "errors": [
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:       {
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:         "message": "The resource 'projects/bob-gketest-prod/zones/us-central1-f/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-13553402-grp' was not found",
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google_v3.17.0_x5:    ]
2020-04-16T00:41:26.058Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:         "domain": "global",
2020-04-16T00:41:26.059Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:         "reason": "notFound"
2020-04-16T00:41:26.061Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:       }
2020-04-16T00:41:26.063Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:     ]
2020-04-16T00:41:26.064Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5:   }
2020-04-16T00:41:26.065Z [DEBUG] plugin.terraform-provider-google-beta_v3.17.0_x5: }

2020/04/16 00:41:26 [ERROR] <root>: eval: *terraform.EvalRefresh, err: Error reading instance group manager returned as an instance group URL: "googleapi: Error 404: The resource 'projects/bob-gketest-prod/zones/us-central1-f/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-13553402-grp' was not found, notFound"

Error: Error reading instance group manager returned as an instance group URL: "googleapi: Error 404: The resource 'projects/bob-gketest-prod/zones/us-central1-f/instanceGroupManagers/gke-bob-gk02-bob-gk02-standar-13553402-grp' was not found, notFound"


2020-04-16T00:41:26.691Z [DEBUG] plugin: plugin process exited: path=/src/.terraform/bob-gk02/plugins/linux_amd64/terraform-provider-google_v3.17.0_x5 pid=261
2020-04-16T00:41:26.691Z [DEBUG] plugin: plugin exited
2020-04-16T00:41:26.691Z [DEBUG] plugin: plugin process exited: path=/src/.terraform/bob-gk02/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4 pid=231
2020-04-16T00:41:26.691Z [DEBUG] plugin: plugin exited
2020-04-16T00:41:26.693Z [DEBUG] plugin: plugin process exited: path=/src/.terraform/bob-gk02/plugins/linux_amd64/terraform-provider-aws_v2.11.0_x4 pid=252
2020-04-16T00:41:26.693Z [DEBUG] plugin: plugin exited
2020-04-16T00:41:26.697Z [DEBUG] plugin: plugin process exited: path=/src/.terraform/bob-gk02/plugins/linux_amd64/terraform-provider-google-beta_v3.17.0_x5 pid=271
2020-04-16T00:41:26.697Z [DEBUG] plugin: plugin exited

[shields-fn]

We're seeing similar behavior on a cluster where we enabled Batch on GKE, which dynamically creates and destroys instance groups. That's a lot more concerning, since it implies that we can't use Terraform to manage the node pools of a cluster with Batch enabled.

[danawillow]

Thanks everyone for your patience; coming back to this now.

One way I could fix this is to just not include any instance group url that 404s in the instance_group_urls field, and stop throwing that error. I'll prepare that PR now.

If I did that, does that sound like something that would be sufficient?

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!