Terraform: 500 Errror when deploying GKE Cluster to GCP

[cofonseca]

Hello,

I'm trying to use Terraform to deploy a very basic Kubernetes cluster to GCP. I've created one project in GCP. I have a single service account with the "organization administrator" IAM role. I've downloaded the credentials json file to allow Terraform's Google provider to connect, which appears to work.

When I try to use Terraform to deploy a cluster, a "500 Internal Server Error" is returned, with a reason of "backendError". Terraform retries over and over until it eventually times out and fails.

I'm able to use Terraform to deploy a new Project in the same GCP org using the same service account and provider information, and this works just fine. I have not been able to find any information on this error. I've ensured that the container engine API is enabled as initially requested by Terraform.

Terraform Version

v0.11.11

Terraform Configuration Files

resource "google_container_cluster" "k8s" {
    name = "testcluster"
    zone = "us-east1-c"
    initial_node_count = 3

    master_auth {
        username = "k8sadmin"
        password = "password"
    }

    node_config {
        oauth_scopes = [
            "https://www.googleapis.com/auth/compute",
            "https://www.googleapis.com/auth/devstorage.read_only",
            "https://www.googleapis.com/auth/logging.write",
            "https://www.googleapis.com/auth/monitoring"
        ]
    }
}

Debug Output

2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: 2019/01/16 13:48:40 [DEBUG] Google API Request Details:
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: ---[ REQUEST ]---------------------------------------
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: POST /v1beta1/projects/My%20First%20Project/locations/us-east1-c/clusters?alt=json&prettyPrint=false HTTP/1.1
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Host: container.googleapis.com
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: User-Agent: google-api-go-client/0.5 Terraform/0.11.7 (+https://www.terraform.io) terraform-provider-google/1.20.0
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Content-Length: 477
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Content-Type: application/json
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Accept-Encoding: gzip
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: {
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:  "cluster": {
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "binaryAuthorization": {
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    "enabled": false
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   },
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "initialNodeCount": 3,
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "legacyAbac": {
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    "enabled": false
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   },
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "masterAuth": {
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    "password": "NEwGa643JSRwEcUQ",
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    "username": "k8sadmin"
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   },
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "name": "testcluster",
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "network": "projects/My First Project/global/networks/default",
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "nodeConfig": {
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    "oauthScopes": [
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "https://www.googleapis.com/auth/monitoring",
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "https://www.googleapis.com/auth/devstorage.read_only",
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "https://www.googleapis.com/auth/logging.write",
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "https://www.googleapis.com/auth/compute"
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:    ]
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   }
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:  }
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: }
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:
2019-01-16T13:48:40.647-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: -----------------------------------------------------
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: 2019/01/16 13:48:40 [DEBUG] Google API Response Details:
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: ---[ RESPONSE ]--------------------------------------
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: HTTP/2.0 500 Internal Server Error
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Alt-Svc: quic=":443"; ma=2592000; v="44,43,39,35"
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Cache-Control: private
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Content-Type: application/json; charset=UTF-8
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Date: Wed, 16 Jan 2019 18:48:40 GMT
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Server: ESF
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Vary: Origin
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Vary: X-Origin
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: Vary: Referer
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: X-Content-Type-Options: nosniff
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: X-Frame-Options: SAMEORIGIN
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: X-Xss-Protection: 1; mode=block
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: {
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   "error": {
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "code": 500,
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "message": "Internal error encountered.",
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "errors": [
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:       {
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:         "message": "Internal error encountered.",
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:         "domain": "global",
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:         "reason": "backendError"
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:       }
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     ],
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:     "status": "INTERNAL"
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:   }
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: }
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe:
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: -----------------------------------------------------
2019-01-16T13:48:40.840-0500 [DEBUG] plugin.terraform-provider-google_v1.20.0_x4.exe: 2019/01/16 13:48:40 [TRACE] Waiting 1s before next try

Expected Behavior

I expected that Terraform would have deployed a very basic Kubernetes cluster with 3 nodes and default settings.

Actual Behavior

Terraform fails to create the cluster. The program keeps trying over and over, but receives a 500 error every time. After each failure, the retry period increases a little bit, but the same backendError is returned every time. It eventually times out and fails altogether.

Steps to Reproduce

Additional Context

I know my GCP provider is configured correctly, because I am able to use Terraform to create other resources, such as a new project, using the same service account. The container engine API has been enabled as requested by Terraform.

[cofonseca]

The issue was due to my project name being incorrect. It would be nice to have some more specific error messaging around this in the future.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.