Terraform-provider-kubernetes: config_context ignore

[Djabx]

Terraform Version and Provider Version

Terraform: 0.12.29

• provider.google v3.40.0
• provider.google-beta v3.40.0
• provider.kubernetes v1.13.2

Affected Resource(s)

• provider

Terraform Configuration Files

resource "null_resource" "cluster" {
  provisioner "local-exec" {
    command = "gcloud container clusters get-credentials ${local.cluster_name} --project=${local.project_id} --zone=${local.zones[0]}"
  }
}

provider "kubernetes" {
  config_context  = "gke_${local.project_id}_${local.zones[0]}_${local.cluster_name}"
  load_config_file       = false

  alias = "cluster"
}

Debug Output

Panic Output

Expected Behavior

I should connect to my cluster

Actual Behavior

The context file is created like this:

- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://1.1.1.1
  name: gke_project_europe-west1-b_cluster-name
[...]
- name: gke_project_europe-west1-b_cluster-name
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: /usr/lib64/google-cloud-sdk/bin/gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

But when I do my terraform apply I've got this output:

Error: Post "http://localhost/api/v1/namespaces/default/serviceaccounts": dial tcp [::1]:80: connect: connection refused

The URL does not seem to be changed.

Steps to Reproduce

Important Factoids

References

• #813

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[liuyangc3]

I am using a statically defined provider got same error

provider "kubernetes" {
  load_config_file = "false"
  alias = "cluster"
  host = "https://..."

  client_certificate     = "${file("~/.kube/client-cert.pem")}"
  client_key             = "${file("~/.kube/client-key.pem")}"
  cluster_ca_certificate = "${file("~/.kube/cluster-ca-cert.pem")}"
}

module "mymodule" {
  providers = {
    kubernetes = kubernetes.cluster
  }
 ...
}

Error: Get "http://localhost/apis/rbac.authorization.k8s.io/v1/clusterroles/cs:restricted": dial tcp [::1]:80: connect: connection refused

[aareet]

@Djabx I see you've specified load_config_file = false in your config, along with config_context. If you see the docs, config_context is a way to specify a context when using the config file. Please try removing the load_config_file = false and retrying.

[aareet]

@liuyangc3 your issue seems to be a different one - can you run a trace and post the output?

[liuyangc3]

@aareet sorry the code is just an example, actually, I use alicloud k8s resource to provision the cluster, so after the cluster created, the cert and key are in the state instead of files. Also, I didn't set the context and the KUBE_CONFIG for the kubectl yet, I need to apply some RBAC roles in my module before I save the config to files.

I can show you the state, it basically looks like below, I have to remove some sensitive information.

terraform state show data.terraform_remote_state.k8s
data "terraform_remote_state" "k8s" {
    backend   = "s3"
    config    = {...}
    outputs   = {
        client_cert          = <<~EOT
                -----BEGIN CERTIFICATE-----
                ...
         client_key          = <<~EOT
              ...
         cluster_ca_cert          = <<~EOT
             ...
}

I can ensure that the api_server_internet is correct. and I create the provider like this

locals {
  k8s = data.terraform_remote_state.k8s.outputs
}
provider "kubernetes" {
  alias                  = "cluster"
  host                   = local.k8s.api_server_internet
  client_certificate     = local.k8s.client_cert
  client_key             = local.k8s.client_key
  cluster_ca_certificate = local.k8s.cluster_ca_cert
}

[liuyangc3]

trace log doesn't help much

2020/09/25 22:54:07 [TRACE] EvalReadState: reading state for module.mymodule.kubernetes_service_account.this
2020/09/25 22:54:07 [TRACE] UpgradeResourceState: schema version of module.mymodule.kubernetes_service_account.this is still 0; calling provider "kubernetes" for any other minor fixups
2020/09/25 22:54:07 [TRACE] GRPCProvider: UpgradeResourceState
2020/09/25 22:54:07 [TRACE] GRPCProvider: ReadResource
2020/09/25 22:54:07 [TRACE] GRPCProvider: ReadResource
2020/09/25 22:54:07 [TRACE] eval: *terraform.EvalRefreshDependencies
2020/09/25 22:54:07 [TRACE] eval: *terraform.EvalRefresh
2020/09/25 22:54:07 [TRACE] GRPCProvider: ReadResource
2020-09-25T22:54:07.965+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2020/09/25 22:54:07 [INFO] Checking cluster role cs:admin
2020-09-25T22:54:07.965+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2020/09/25 22:54:07 [INFO] Checking RoleBinding gitlab-role-binding
2020-09-25T22:54:07.965+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2020/09/25 22:54:07 [DEBUG] Kubernetes API Request Details:
2020-09-25T22:54:07.965+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: ---[ REQUEST ]---------------------------------------
2020-09-25T22:54:07.965+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: GET /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings/gitlab-role-binding HTTP/1.1
2020-09-25T22:54:07.965+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Host: localhost
2020-09-25T22:54:07.965+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: User-Agent: HashiCorp/1.0 Terraform/0.13.3
2020-09-25T22:54:07.965+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Accept: application/json, */*
2020-09-25T22:54:07.965+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Accept-Encoding: gzip
2020-09-25T22:54:07.966+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 
2020-09-25T22:54:07.966+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 
2020-09-25T22:54:07.966+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: -----------------------------------------------------
2020-09-25T22:54:07.966+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2020/09/25 22:54:07 [DEBUG] Received error: &url.Error{Op:"Get", URL:"http://localhost/apis/rbac.authorization.k8s.io/v1/clusterroles/cs:admin", Err:(*net.OpError)(0xc000b64230)}
2020-09-25T22:54:07.966+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2020/09/25 22:54:07 [DEBUG] Received error: &url.Error{Op:"Get", URL:"http://localhost/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings/gitlab-role-binding", Err:(*net.OpError)(0xc000a85a40)}
2020-09-25T22:54:07.966+0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2020/09/25 22:54:07 [DEBUG] Received error: &url.Error{Op:"Get", URL:"http://localhost/api/v1/namespaces/kube-system/serviceaccounts/gitlab", Err:(*net.OpError)(0xc000b64370)}
2020/09/25 22:54:07 [ERROR] eval: *terraform.EvalRefresh, err: Get "http://localhost/apis/rbac.authorization.k8s.io/v1/clusterroles/cs:admin": dial tcp [::1]:80: connect: connection refused
2020/09/25 22:54:07 [ERROR] eval: *terraform.EvalSequence, err: Get "http://localhost/apis/rbac.authorization.k8s.io/v1/clusterroles/cs:admin": dial tcp [::1]:80: connect: connection refused
2020/09/25 22:54:07 [TRACE] [walkRefresh] Exiting eval tree: module.czjk.kubernetes_cluster_role.admin

[liuyangc3]

Hi @aareet, If I defined provider inside my module, it works

[Djabx]

@aareet well... thanks you it work :)

Sorry ... RTFM

[aareet]

@Djabx great :)

@liuyangc3 are you provisioning your provider in the same apply as your resources when it's not in the module?

[liuyangc3]

right, apply when provider outside module not works

provider kubernetes {
   ...
}

module {
  providers = { kubernetes = the prodiver above}
}

works

# provider defined in module
module {
  ...
}

[liuyangc3]

@aareet do you mind checking this?

[aareet]

@liuyangc3 it isn't clear to me from your snippet what the issue could be. Can you please post a working configuration that reproduces the issue for you as a separate issue, with all the issue information requested? I'm going to close this bug since @Djabx's issue is resolved. I don't think your problem is related based on the information so far.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!