Terraform-provider-google: Error: Error when reading or editing Dataset: googleapi: Error 400: Dataset gcp-project-name:test_cluster_dataset is still in use, resourceInUse

[jsoref]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave _+1_ or _me too_ comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

terraform version
Terraform v0.12.26

terraform-provider-google-beta_v3.27.0_x5
terraform-provider-google_v3.27.0_x5

Affected Resource(s)

• google_bigquery_dataset

Terraform Configuration Files

old:

{
  "provider": {
    "google": {
      "version": "~> 3.0",
      "credentials": "/Users/jsoref/key.json",
      "project": "gcp-project-name",
      "region": "us-central1"
    },
    "google-beta": {
      "version": "~> 3.0",
      "credentials": "/Users/jsoref/key.json",
      "project": "gcp-project-name",
      "region": "us-central1"
    }
  },
  "terraform": {
    "required_version": "~> 0.12.13",
    "backend": {
      "gcs": {
        "bucket": "gcp-project-name-tfstate",
        "prefix": "terraform/state/k8s_cluster"
      }
    }
  },
  "resource": {
    "google_service_account": {
      "kubernetes_test": {
        "account_id": "kubernetes-test",
        "display_name": "kubernetes-test"
      }
    },
    "google_project_iam_member": {
      "kubernetes_test_iam_serviceaccountuser": {
        "project": "gcp-project-name",
        "member": "serviceAccount:${google_service_account.kubernetes_test.email}",
        "role": "roles/iam.serviceAccountUser"
      },
      "kubernetes_test_logging_logwriter": {
        "project": "gcp-project-name",
        "member": "serviceAccount:${google_service_account.kubernetes_test.email}",
        "role": "roles/logging.logWriter"
      },
      "kubernetes_test_monitoring_metricwriter": {
        "project": "gcp-project-name",
        "member": "serviceAccount:${google_service_account.kubernetes_test.email}",
        "role": "roles/monitoring.metricWriter"
      },
      "kubernetes_test_monitoring_viewer": {
        "project": "gcp-project-name",
        "member": "serviceAccount:${google_service_account.kubernetes_test.email}",
        "role": "roles/monitoring.viewer"
      }
    },
    "google_bigquery_dataset": {
      "test_cluster_dataset": {
        "dataset_id": "test_cluster_dataset"
      }
    },
    "google_container_cluster": {
      "test-cluster": {
        "provider": "google-beta",
        "name": "test-cluster",
        "addons_config": {
          "istio_config": {
            "disabled": true
          }
        },
        "enable_shielded_nodes": true,
        "initial_node_count": 1,
        "ip_allocation_policy": [
          {
            "cluster_ipv4_cidr_block": "",
            "services_ipv4_cidr_block": ""
          }
        ],
        "location": "us-central1",
        "maintenance_policy": {
          "daily_maintenance_window": {
            "start_time": "08:00"
          }
        },
        "master_auth": {
          "username": "",
          "password": "",
          "client_certificate_config": {
            "issue_client_certificate": false
          }
        },
        "master_authorized_networks_config": {
          "cidr_blocks": [
          ]
        },
        "network": "default",
        "private_cluster_config": {
          "enable_private_nodes": true,
          "enable_private_endpoint": false,
          "master_ipv4_cidr_block": "172.16.0.16/28"
        },
        "release_channel": {
          "channel": "UNSPECIFIED"
        },
        "remove_default_node_pool": true,
        "resource_usage_export_config": {
          "enable_network_egress_metering": false,
          "bigquery_destination": {
            "dataset_id": "${google_bigquery_dataset.test_cluster_dataset.dataset_id}"
          }
        },
        "subnetwork": "default",
        "workload_identity_config": {
          "identity_namespace": "gcp-project-name.svc.id.goog"
        }
      }
    },
    "google_container_node_pool": {
      "test-pool": {
        "name": "test-pool",
        "location": "us-central1",
        "cluster": "${google_container_cluster.test-cluster.name}",
        "initial_node_count": 1,
        "management": {
          "auto_repair": true,
          "auto_upgrade": true
        },
        "node_config": {
          "service_account": "${google_service_account.kubernetes_test.email}",
          "oauth_scopes": [
            "cloud-platform"
          ],
          "machine_type": "e2-standard-4",
          "disk_size_gb": "50",
          "shielded_instance_config": {
            "enable_secure_boot": true
          },
          "tags": [
            "test-pool"
          ]
        },
        "autoscaling": {
          "min_node_count": 1,
          "max_node_count": 1
        }
      }
    },
    "google_compute_firewall": {
      "test-cluster-to-gce": {
        "network": "default",
        "name": "test-cluster-to-gce",
        "allow": [
          {
            "protocol": "tcp"
          },
          {
            "protocol": "udp"
          },
          {
            "protocol": "icmp"
          },
          {
            "protocol": "esp"
          },
          {
            "protocol": "ah"
          }
        ],
        "source_ranges": [
          "${google_container_cluster.test-cluster.cluster_ipv4_cidr}"
        ]
      },
      "master-to-gke": {
        "network": "default",
        "name": "master-to-gke",
        "allow": [
          {
            "protocol": "tcp",
            "ports": [
              "8443",
              "8080"
            ]
          }
        ],
        "source_ranges": [
          "172.16.0.16/28"
        ]
      },
      "allow-egress-gke-test-pool": {
        "name": "allow-egress-gke-test-pool",
        "network": "default",
        "direction": "EGRESS",
        "target_tags": [
          "test-pool"
        ],
        "priority": 1500,
        "allow": [
          {
            "protocol": "tcp"
          },
          {
            "protocol": "udp"
          }
        ]
      }
    }
  }
}

new:

{
  "provider": {
    "google": {
      "version": "~> 3.0",
      "credentials": "/Users/jsoref/key.json",
      "project": "gcp-project-name",
      "region": "us-central1"
    },
    "google-beta": {
      "version": "~> 3.0",
      "credentials": "/Users/jsoref/key.json",
      "project": "gcp-project-name",
      "region": "us-central1"
    }
  },
  "terraform": {
    "required_version": "~> 0.12.13",
    "backend": {
      "gcs": {
        "bucket": "gcp-project-name-tfstate",
        "prefix": "terraform/state/k8s_cluster"
      }
    }
  },
  "resource": {
    "google_service_account": {
      "kubernetes_default": {
        "account_id": "kubernetes-default",
        "display_name": "kubernetes-default"
      }
    },
    "google_project_iam_member": {
      "kubernetes_default_iam_serviceaccountuser": {
        "project": "gcp-project-name",
        "member": "serviceAccount:${google_service_account.kubernetes_default.email}",
        "role": "roles/iam.serviceAccountUser"
      },
      "kubernetes_default_logging_logwriter": {
        "project": "gcp-project-name",
        "member": "serviceAccount:${google_service_account.kubernetes_default.email}",
        "role": "roles/logging.logWriter"
      },
      "kubernetes_default_monitoring_metricwriter": {
        "project": "gcp-project-name",
        "member": "serviceAccount:${google_service_account.kubernetes_default.email}",
        "role": "roles/monitoring.metricWriter"
      },
      "kubernetes_default_monitoring_viewer": {
        "project": "gcp-project-name",
        "member": "serviceAccount:${google_service_account.kubernetes_default.email}",
        "role": "roles/monitoring.viewer"
      }
    },
    "google_bigquery_dataset": {
      "default_cluster_dataset": {
        "dataset_id": "default_cluster_dataset"
      }
    },
    "google_container_cluster": {
      "default-cluster": {
        "provider": "google-beta",
        "name": "default-cluster",
        "addons_config": {
          "istio_config": {
            "disabled": true
          }
        },
        "enable_shielded_nodes": true,
        "initial_node_count": 1,
        "ip_allocation_policy": [
          {
            "cluster_ipv4_cidr_block": "",
            "services_ipv4_cidr_block": ""
          }
        ],
        "location": "us-central1",
        "maintenance_policy": {
          "daily_maintenance_window": {
            "start_time": "08:00"
          }
        },
        "master_auth": {
          "username": "",
          "password": "",
          "client_certificate_config": {
            "issue_client_certificate": false
          }
        },
        "master_authorized_networks_config": {
          "cidr_blocks": [
          ]
        },
        "network": "default",
        "private_cluster_config": {
          "enable_private_nodes": true,
          "enable_private_endpoint": false,
          "master_ipv4_cidr_block": "172.16.0.16/28"
        },
        "release_channel": {
          "channel": "UNSPECIFIED"
        },
        "remove_default_node_pool": true,
        "resource_usage_export_config": {
          "enable_network_egress_metering": false,
          "bigquery_destination": {
            "dataset_id": "${google_bigquery_dataset.default_cluster_dataset.dataset_id}"
          }
        },
        "subnetwork": "default",
        "workload_identity_config": {
          "identity_namespace": "gcp-project-name.svc.id.goog"
        }
      }
    },
    "google_container_node_pool": {
      "default-pool": {
        "name": "default-pool",
        "location": "us-central1",
        "cluster": "${google_container_cluster.default-cluster.name}",
        "initial_node_count": 1,
        "management": {
          "auto_repair": true,
          "auto_upgrade": true
        },
        "node_config": {
          "service_account": "${google_service_account.kubernetes_default.email}",
          "oauth_scopes": [
            "cloud-platform"
          ],
          "machine_type": "e2-standard-4",
          "disk_size_gb": "50",
          "shielded_instance_config": {
            "enable_secure_boot": true
          },
          "tags": [
            "default-pool"
          ]
        },
        "autoscaling": {
          "min_node_count": 1,
          "max_node_count": 1
        }
      }
    },
    "google_compute_firewall": {
      "default-cluster-to-gce": {
        "network": "default",
        "name": "default-cluster-to-gce",
        "allow": [
          {
            "protocol": "tcp"
          },
          {
            "protocol": "udp"
          },
          {
            "protocol": "icmp"
          },
          {
            "protocol": "esp"
          },
          {
            "protocol": "ah"
          }
        ],
        "source_ranges": [
          "${google_container_cluster.default-cluster.cluster_ipv4_cidr}"
        ]
      },
      "master-to-gke": {
        "network": "default",
        "name": "master-to-gke",
        "allow": [
          {
            "protocol": "tcp",
            "ports": [
              "8443",
              "8080"
            ]
          }
        ],
        "source_ranges": [
          "172.16.0.16/28"
        ]
      },
      "allow-egress-gke-default-pool": {
        "name": "allow-egress-gke-default-pool",
        "network": "default",
        "direction": "EGRESS",
        "target_tags": [
          "default-pool"
        ],
        "priority": 1500,
        "allow": [
          {
            "protocol": "tcp"
          },
          {
            "protocol": "udp"
          }
        ]
      }
    }
  }
}

Debug Output

https://gist.github.com/jsoref/da74214afe233a4fa80ed958e84a1140

Panic Output

Expected Behavior

delete the object

Actual Behavior

google_bigquery_dataset.test_cluster_dataset: Destroying... [id= gcp-project-name:test_cluster_dataset]

Error: Error when reading or editing Dataset: googleapi: Error 400: Dataset gcp-project-name:test_cluster_dataset is still in use, resourceInUse

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #5726

[venkykuberan]

@jsoref did the retry of terraform destroy worked ?

[jsoref]

Sorry, I don't understand what you mean.

running terraform apply and then terraform apply again doesn't improve things. I've been trying to delete this thing for days.

[venkykuberan]

The reason for asking to rerun the destroy command to check if retry mechanism helps.

My understanding is that you are trying to rename the cluster, if so you don't have to run the destroy command, just change the name of the cluster and run the terraform apply. It will do the force destroy and recreate the cluster with the new name.

P.S. - Please just share the problematic config, It would be easier to troubleshoot the prob.

[jsoref]

What we've done is rename the cluster (in terraform), it successfully performs all the other steps (deleting the cluster, creating the cluster, creating the new dataset), it just fails with the delete.

[venkykuberan]

I am able to repro it, however i don't think its a provider/terraform issue. Its more of an upstream issue as i get the same error when i hit API directly - https://cloud.google.com/bigquery/docs/reference/rest/v2/datasets/delete. I am not sure its a cache issue on the API side.

{
  "error": {
    "code": 400,
    "message": "Dataset xxx-xxx:test_cluster_dataset is still in use",
    "errors": [
      {
        "message": "Dataset xxx-xxx:test_cluster_dataset is still in use",
        "domain": "global",
        "reason": "resourceInUse",
      }
    ],
    "status": "INVALID_ARGUMENT"
  }

If this is one time activity it can be mitigated by deleting the dataset on the console and run terraform destroy -target=google_bigquery_dataset.test_cluster_dataset will clean the state file.

@c2thorn what are your thoughts on this ?

[jsoref]

I think the problem is that delete_contents_on_destroy isn't set.

This is technically documented:

delete_contents_on_destroy - (Optional) If set to true, delete all the tables in the dataset when destroying the resource; otherwise, destroying the resource will fail if tables are present.

Offhand, I think the error should either suggest adding delete_contents_on_destroy (in order to use terraform to then destroy it) or suggest going to bigquery (preferably by offering a link) to do the delete (either of the contents or the object). I claim the current user experience is user hostile.

[c2thorn]

Hi @jsoref
Thanks for the feedback and sorry about your experience. Usually we just pass the error through, but this may be a good candidate for wrapping the error message with a suggestion. I'll close the issue for now, since adding delete_contents_on_destroy does solve the issue when reproducing.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!