Terraform-provider-google: google-beta v3.10.0 fail to modify gke node pool status

Created on 26 Feb 2020 · 8Comments · Source: hashicorp/terraform-provider-google

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave _+1_ or _me too_ comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.20

Affected Resource(s)

google_container_node_pool

Terraform Configuration Files

provider "google-beta" {
  project = var.google_project
  region  = var.google_region
  zone    = var.google_zone
}

resource "google_container_node_pool" "main_pool" {
  provider   = google-beta
  name       = "main-pool"
  cluster    = google_container_cluster.primary.name
  node_count = var.main_pool_min_size

  autoscaling {
    max_node_count = var.main_pool_max_size
    min_node_count = var.main_pool_min_size
  }

  node_config {
    machine_type = var.main_pool_machine_type
    preemptible  = var.main_pool_with_preemptible_nodes

    oauth_scopes = ["compute-rw", "storage-ro", "logging-write", "monitoring"]

    workload_metadata_config {
      node_metadata = "GKE_METADATA_SERVER"
    }
  }

  lifecycle {
    prevent_destroy = true
    ignore_changes  = [node_count]
  }

  management {
    auto_upgrade = true
    auto_repair  = true
  }
}

Expected Behavior

module.platform.google_container_node_pool.main_pool: Modifying... [id=europe-west1-b/idp-datastore-cluster/main-pool]
module.platform.google_container_node_pool.main_pool: Modifications complete after 2s [id=europe-west1-b/idp-datastore-cluster/main-pool]

Actual Behavior

 module.platform.google_container_node_pool.main_pool: Modifying... [id=europe-west1-b/idp-datastore-cluster/main-pool]
 Error: googleapi: got HTTP response code 404 with body: <!DOCTYPE html>
 <html lang=en>
   <meta charset=utf-8>
   <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
   <title>Error 404 (Not Found)!!1</title>
   <style>
     *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
   </style>
   <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
   <p><b>404.</b> <ins>That’s an error.</ins>
   <p>The requested URL <code>/v1beta1/europe-west1-b/idp-datastore-cluster/main-pool?alt=json&amp;prettyPrint=false</code> was not found on this server.  <ins>That’s all we know.</ins>
   on platform/cluster.tf line 71, in resource "google_container_node_pool" "main_pool":
   71: resource "google_container_node_pool" "main_pool" {

Steps to Reproduce

terraform apply

Important Factoids

References

#0000

bug

Source

aubm

👍11

Most helpful comment

Ah, got it- if it's happening for users upgrading from 2.X, it's very likely that it's a change related to the id. We used to store the id as {{location}}/{{name}} but now store projects/{{project}}/locations/{{location}}/clusters/{{cluster}}/nodePools/{{name}}. Code relying directly on the id won't work as expected.

@davidxia: Thanks for the release range! And whoops, the change to node pools that landed in 3.10 was mine. I'd added code that relied directly on the id, which worked in tests / locally for me, since the pools were all created with the new id format.

@sdif: I'd be surprised if 3.9 and prior failed for the same reason. Can you get debug logs?

I'm going to assign this to our bug onduty to fix the regression in 3.10.

rileykarson on 10 Mar 2020

👍3

All 8 comments

Currently trying to reproduce the error outside of the context of my project. Closing this issue in the meantime, as the elements in the current description are not enough to reproduce.

aubm on 26 Feb 2020

I just hit this issue today when trying to upgrade a node pool for the first time with provider 3.11.0. It worked for me on 3.9.0. My error was the same as above.

Looks like some undocumented change or regression in the code between 3.9 and 3.11 that creates the GKE API URL differently?

This might be a related issue.

davidxia on 9 Mar 2020

Hi there,
We have the same issue but with every version of the provider since the 3.X.
Our node pools are fine with the v2.20.2 when we change autoscaling.max_node_count for example but as soon as we try to do the same thing with the v3 of the provider we have this issue:

google_container_node_pool.k8s-cluster-np-002-regular: Modifying... [id=europe-west1-b/integration-v1/np-002-regular]

Error: googleapi: got HTTP response code 404 with body: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/v1beta1/europe-west1-b/integration-v1/np-002-regular?alt=json&amp;prettyPrint=false</code> was not found on this server.  <ins>That’s all we know.</ins>

We tried many versions of the provider: 3.2, 3.5, 3.9 and 3.12.

sdif on 10 Mar 2020

Observed with 3.12.0 on a state created by a 2.x provider. Will try downgrades.

rtoma on 10 Mar 2020

@sdif: I'd be surprised if 3.9 and prior failed for the same reason. Can you get debug logs?

I'm going to assign this to our bug onduty to fix the regression in 3.10.

rileykarson on 10 Mar 2020

👍3

Acknowledged, I see how to fix this. :)

ndmckinley on 10 Mar 2020

We ran into the same issue when trying to set upgrade_settings on a node pool created in 2.x. The only fix was to destroy the node pool and create a new one with 3.11 google-beta provider.

jravetch on 10 Mar 2020

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!