Terraform-provider-kubernetes: GKE destroy fails on namespace resource

[hashibot[bot]]

_This issue was originally opened by @peter-fe as hashicorp/terraform#20855. It was migrated here as a result of the provider split. The original body of the issue is below._

---

Hi,

When using this minimal setup, a destroy fails because of the namespace resource, what leaves the deployment in an inconsistent/broken state.
Parts of the cluster remain on the project but in an unusable state as well as the namespace itself.

If you remove the namespace in advance by editing the main.tf => plan & apply and run a subsequent destroy after that, everything works fine.

It seems that the node pool resources are destroyed before the namespace ones and that causes this problem.

Terraform Version

Terraform v0.11.13
+ provider.google v2.3.0
+ provider.kubernetes v1.5.2

Terraform Configuration Files

provider "google" {
  credentials = "${file("../../credentials/gcp-sa-terraform.json")}"
  project     = "<replace-me>"
}

resource "google_container_cluster" "container_cluster" {
  name           = "my-cluster"
  location       = "europe-west4"
  node_locations = "europe-west4-a"

  remove_default_node_pool = true

  initial_node_count = 1

  master_auth {
    username = ""
    password = ""
  }
}

resource "google_container_node_pool" "container_cluster_node_pool" {
  name       = "node-pool-1"
  location   = "europe-west4"
  cluster    = "${google_container_cluster.container_cluster.name}"
  node_count = "1"

  management {
    auto_repair = true
  }

  node_config {
    machine_type = "n1-standard-1"

    oauth_scopes = [
      "https://www.googleapis.com/auth/compute",
      "https://www.googleapis.com/auth/devstorage.read_only",
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
    ]
  }
}

provider "kubernetes" {
  host                   = "${google_container_cluster.container_cluster.endpoint}"
  token                  = "${data.google_client_config.current.access_token}"
  cluster_ca_certificate = "${base64decode(google_container_cluster.container_cluster.cluster_ca_certificate)}"
}

resource "kubernetes_namespace" "ns_my-namespace" {
  metadata {
    name = "my-namespace"
  }
}

Crash Output

google_container_cluster.container_cluster: Refreshing state... (ID: my-cluster)
data.google_client_config.current: Refreshing state...
google_container_node_pool.container_cluster_node_pool: Refreshing state... (ID: europe-west4-a/my-cluster/node-pool-1)
kubernetes_namespace.ns_my-namespace: Refreshing state... (ID: my-namespace)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - google_container_cluster.container_cluster

  - google_container_node_pool.container_cluster_node_pool

  - kubernetes_namespace.ns_my-namespace


Plan: 0 to add, 0 to change, 3 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

google_container_node_pool.container_cluster_node_pool: Destroying... (ID: europe-west4-a/my-cluster/node-pool-1)
kubernetes_namespace.ns_my-namespace: Destroying... (ID: my-namespace)
google_container_node_pool.container_cluster_node_pool: Still destroying... (ID: europe-west4-a/my-cluster/node-pool-1, 10s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 10s elapsed)
google_container_node_pool.container_cluster_node_pool: Still destroying... (ID: europe-west4-a/my-cluster/node-pool-1, 20s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 20s elapsed)
google_container_node_pool.container_cluster_node_pool: Still destroying... (ID: europe-west4-a/my-cluster/node-pool-1, 30s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 30s elapsed)
google_container_node_pool.container_cluster_node_pool: Still destroying... (ID: europe-west4-a/my-cluster/node-pool-1, 40s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 40s elapsed)
google_container_node_pool.container_cluster_node_pool: Still destroying... (ID: europe-west4-a/my-cluster/node-pool-1, 50s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 50s elapsed)
google_container_node_pool.container_cluster_node_pool: Still destroying... (ID: europe-west4-a/my-cluster/node-pool-1, 1m0s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 1m0s elapsed)
google_container_node_pool.container_cluster_node_pool: Still destroying... (ID: europe-west4-a/my-cluster/node-pool-1, 1m10s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 1m10s elapsed)
google_container_node_pool.container_cluster_node_pool: Still destroying... (ID: europe-west4-a/my-cluster/node-pool-1, 1m20s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 1m20s elapsed)
google_container_node_pool.container_cluster_node_pool: Destruction complete after 1m25s
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 1m30s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 1m40s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 1m50s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 2m0s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 2m10s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 2m20s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 2m30s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 2m40s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 2m50s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 3m0s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 3m10s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 3m20s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 3m30s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 3m40s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 3m50s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 4m0s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 4m10s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 4m20s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 4m30s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 4m40s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 4m50s elapsed)
kubernetes_namespace.ns_my-namespace: Still destroying... (ID: my-namespace, 5m0s elapsed)

Error: Error applying plan:

1 error(s) occurred:

* kubernetes_namespace.ns_my-namespace (destroy): 1 error(s) occurred:

* kubernetes_namespace.ns_my-namespace: timeout while waiting for resource to be gone (last state: 'Terminating', timeout: 5m0s)

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Expected Behavior

The destroy should succeed and every part of the deployment should get removed leaving a clean state.

Actual Behavior

When running a destroy it fails after the 5 minutes timeout, leaving the cluster (without the node-pool) and the namespace as fragments in the GCP project.
It seems that destroy is removing the node-pools prior the namespaces and this is causing this behavior.
Every subsequent run of destroy or plan fails after that and the terraform state as well as the remaining resource have to get cleaned up by hand.

Steps to Reproduce

terraform init
terraform apply
terraform destroy

Additional Context

GKE version: 1.11.7-gke.12

[alexsomesan]

@peter-fe This is not a Kubernetes provider issue, nor is it easy to attribute to Terraform itself.

In Terraform's view of resource dependencies the namespace resource does not depend on either of the GKE resource, because attributes of those resources are not involved in feeding inputs of the namespace resource (this is how Terraform establishes dependencies).
The easiest workaround (but not the cleanest) is to define these dependencies explicitly by using the depends_on meta-attribute. In this particular case, you need to tell Terraform that the kubernetes_namespace resource depends on the google_container_node_pool so that it orders the destroy operations accordingly. One other option is to add labels to the Kubernetes object who's values interpolate attributes of the dependent resource.

Although there is nothing the Kubernetes provider itself can do to enforce these dependencies, I'm happy to explore options of expressing these in template configuration.

[peter-fe]

@alexsomesan, thank you for the hint regarding depends_on. We tried that before and it was not respected, so we created the testcase and at the same time updated our tf version, but only checked versus the synthetic testcase, so never tried depends_on in the whole template again (depends_on a Module) with the current release.
It works now as expected.

[alexsomesan]

@peter-fe Happy to hear it works.
For what it's worth, here's the configuration I used to confirm the depends_on solution. This also works as expected.

provider "google" {
  project = "test-project"
}

data "google_client_config" "current" {}

resource "google_container_cluster" "container_cluster" {
  name           = "my-cluster"
  location       = "europe-west4"
  node_locations = ["europe-west4-a"]

  remove_default_node_pool = true

  initial_node_count = 1

  master_auth {
    username = "hello"
    password = "world"
  }
}

resource "google_container_node_pool" "container_cluster_node_pool" {
  name       = "node-pool-1"
  location   = "europe-west4"
  cluster    = "${google_container_cluster.container_cluster.name}"
  node_count = "1"

  management {
    auto_repair = true
  }

  node_config {
    machine_type = "n1-standard-1"

    oauth_scopes = [
      "https://www.googleapis.com/auth/compute",
      "https://www.googleapis.com/auth/devstorage.read_only",
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
    ]
  }
}

provider "kubernetes" {
  host                   = "${google_container_cluster.container_cluster.endpoint}"
  token                  = "${data.google_client_config.current.access_token}"
  cluster_ca_certificate = "${base64decode(google_container_cluster.container_cluster.master_auth.0.cluster_ca_certificate)}"
}

resource "kubernetes_namespace" "test-namespace" {

  depends_on = ["google_container_node_pool.container_cluster_node_pool"]

  metadata {
    name = "test-namespace"
  }
}

[alexsomesan]

Feel free to close the issue if you don't require any more assistance.

[peter-fe]

@alexsomesan, my issue is solved, but as the ticket has been cloned by the bot I'm not aware how I can close it.
Can I?

[alexsomesan]

I'll close it. Happy to hear it's working for you.

[jodersky]

I'm in a situation where the cluster is created in a different module than the namespace, hence there's no way to use the depends_on trick. Any suggestions on how to work around this?

[eddy-curv]

I'm in a situation where the cluster is created in a different module than the namespace, hence there's no way to use the depends_on trick. Any suggestions on how to work around this?

https://discuss.hashicorp.com/t/tips-howto-implement-module-depends-on-emulation/2305