Terraform-provider-google: google_container_cluster node_version inconsistent

[rweverwijk]

I'm using terraform to create a project with a google container engine.

Last week google updated the version numbers. They added a suffix -gke.1.

For example: from 1.7.5 to 1.7.5-gke.1

This works fine while creating a cluster, but after a successful terraform apply a terraform plan indicates that the version number is changed:

node_version: "1.7.5" => "1.7.5-gke.1"

Terraform Version

terraform 0.10.6 on mac.

Terraform Configuration Files

resource "google_container_cluster" "foo" {
  project = "${google_project.foo.project_id}"
  name = "${google_project.foo.project_id}"
  zone = "europe-west1-c"
  node_version = "1.7.5-gke.1"
  initial_node_count = 3

  node_config {
    machine_type = "n1-standard-2"
    oauth_scopes = [
      "https://www.googleapis.com/auth/compute",
      "https://www.googleapis.com/auth/devstorage.read_only",
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
    ]
  }
}

output

...
google_container_cluster.foo: Still modifying... (ID: gdd-foo-8, 3m30s elapsed)
google_container_cluster.foo: Still modifying... (ID: gdd-foo-8, 3m40s elapsed)
Error applying plan:

1 error(s) occurred:

* google_container_cluster.foo: 1 error(s) occurred:

* google_container_cluster.foo: googleapi: Error 400: bad desired node version (1.7.5-gke.1)., badRequest

[smeruelo]

Same problem here.

I was able to create a cluster using node_version = 1.7.5 yesterday, but today I got this error:

Error applying plan:

1 error(s) occurred:

* module.cluster.module.k8s_cluster.google_container_cluster.primary: 1 error(s) occurred:

* google_container_cluster.primary: googleapi: Error 400: Version is invalid., badRequest

Changing it to 1.7.5-gke.1, creation succeeds but then it seems like it tries to update it and it fails:

module.cluster.k8s_cluster.google_container_cluster.primary: Creation complete (ID: staging)
module.cluster.k8s_cluster.google_container_cluster.primary: Modifying... (ID: staging)
  node_version: "1.7.5" => "1.7.5-gke.1"
module.cluster.k8s_cluster.google_container_cluster.primary: Still modifying... (ID: staging, 10s elapsed)
module.cluster.k8s_cluster.google_container_cluster.primary: Still modifying... (ID: staging, 20s elapsed)
module.cluster.k8s_cluster.google_container_cluster.primary: Still modifying... (ID: staging, 30s elapsed)
...
Error applying plan:

1 error(s) occurred:

* module.cluster.module.k8s_cluster.google_container_cluster.primary: 1 error(s) occurred:

* google_container_cluster.primary: googleapi: Error 400: bad desired node version (1.7.5-gke.1)., badRequest

After getting the cluster created, however, I can change it back to 1.7.5 and continue applying the rest of the stuff.

getServerconfig returns this:

{
  "defaultClusterVersion": "1.7.5-gke.1",
  "validNodeVersions": [
    "1.7.6",
    "1.7.5",
    "1.7.4",
    "1.7.3",
    "1.7.2",
    "1.6.10",
    "1.6.9",
    "1.6.8",
    "1.6.7",
    "1.6.4",
    "1.5.7"
  ],
  "defaultImageType": "COS",
  "validImageTypes": [
    "UBUNTU",
    "COS"
  ],
  "validMasterVersions": [
    "1.7.6-gke.1",
    "1.7.5-gke.1",
    "1.6.10-gke.1"
  ]
}

If I understand it correctly, when creating the cluster InitialClusterVersion is used (which needs to match with validMasterVersions), and when updating it it uses DesiredNodeVersion (which has to match validNodeVersions). I guess node_version can't be used for both from now on.

[rochdev]

Adding a master_version field would solve this problem and make upgrades safer by changing the version from the master separately from the nodes.

[danawillow]

Yup, we discovered this last week while trying to get all the tests passing for the release. It's not going to be a trivial solution but it's definitely something I plan on fixing asap.

[Stono]

Yes, we're now experiencing this problem too and are unable to do cluster changes, our pipelines are broke so would love an ETA to know if we should invest effort in some sort of temporary workaround.

I really see this as an opportunity to split out master version and node version as @rochdev suggested as they are separate concerns, trying to merge them into one is what has caused this issue!

[Stelminator]

👍 would also like this change. Glad to see it in progress.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!