Terraform-provider-kubernetes: Provider is creating incorrect ImagePullSecret for Docker Registry

[eyeezzi]

An ImagePullSecret created with the Kubernetes Terraform Provider should be exactly the same as if the kubectl command was used. However, this is not the case. The result is that using the exact same data, secrets created with kubectl work, while those created with KTP result in ImagePullBackoff error.

Terraform Version

• Terraform v0.12.7
• provider.kubernetes v1.9.0
• provider.google v2.13.0

Affected Resource(s)

• kubernetes_secret

Terraform Configuration Files

locals {
  dockerconfigjson = {
    "https://gcr.io" = {
      email    = "[email protected]" 
      username = "_json_key"       
      password = "test"      
      # password = base64decode(google_service_account_key.digital-ocean-cluster.private_key)
    }
  }
}
resource "kubernetes_secret" "gcr-imagepullsecret-staging" {
  metadata {
    name      = "gcr-imagepullsecret-staging"
    namespace = kubernetes_namespace.staging.metadata.0.name
  }
  data = {
    ".dockerconfigjson" = jsonencode(local.dockerconfigjson)
  }
  type = "kubernetes.io/dockerconfigjson"
}

Debug Output

https://gist.github.com/eyeezzi/b01cfb365a372b4a6ae4a22f1d4bf295
Shows terraform output AND kubectl output

Panic Output

N/A

Expected Behavior

The attribute data.dockerconfigjson of the resulting secret should exactly match that produced from regular kubectl command.

Actual Behavior

The secret produced by Terraform has a data.dockerconfigjson attribute which does not match that produced by kubectl command.

Steps to Reproduce

1. Create an imagePullSecret for Google Container Registry (GCR) using kubectl

kubectl --namespace=staging create secret docker-registry gcr-json-key \
--docker-server=https://gcr.io \
--docker-username=_json_key \
 --docker-password="test" \
[email protected] \
--dry-run -o yaml  

1. Now do the exact same thing using the Kubernetes Terraform provider...apply the Terraform HCL snippet above.

2. The output of both methods should be the same...but it is not. The Terraform-created secret results in pulling error.

Important Factoids

• I am providing a minimal example here. In my actual implementation, the dockerconfig password is the GCP service account json key (shown commented)...it is still affected by this issue.
• I suspect that either the provider is not mapping correctly to the format expected by docker, or some weird json-encoding issue is happening here.

References

https://github.com/terraform-providers/terraform-provider-kubernetes/issues/81

[PipTurner]

Well documented. I have exactly the same problem the secret produced is missing the auth section in the encrypted output. Having to use a pipeline to use a kubectl command instead. There is a version of kubectl that didn't create valid dockerconfig secrets -- v1.13.0 or v1.13.1 I think. It's plausible that terraform has a similar problem.

[ghost]

+1

[kasunt-nixdev]

This is working as intended !

@eyeezzi, @PipTurner, @AndreasKnapp1812 it appears you have not read the https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#inspecting-the-secret-regcred correctly.

Your auth would have to be something like,

locals {
    dockerconfigjson = {
        "auths": {
            "https://gcr.io" = {
                email    = "[email protected]"
                username = "_json_key"
                password = trimspace(file(pathexpand("gcr.json")))
                auth     = base64encode(join(":",["_json_key", trimspace(file(pathexpand("gcr.json")))]))
            }
        }
    }
}
resource "kubernetes_secret" "gcr-imagepullsecret-staging" {
    metadata {
        name      = "gcr-imagepullsecret-staging"
        namespace = kubernetes_namespace.staging.metadata.0.name
    }
    data = {
        ".dockerconfigjson" = jsonencode(local.dockerconfigjson)
    }
    type = "kubernetes.io/dockerconfigjson"
}

[PipTurner]

Hi nixgadget, ok understood so you have to manually add the auths section into the dockerconfigjson. I couldn't find anywhere where this is documented in the terraform documentation. Kubernetes automatically produces the auth section when manually creating a secret so it behaves a little differently. Thank you for the clarification.

[kasunt-nixdev]

Unfortunately this isnt documented anywhere afaik. I had to find it the hardway by digging into the k8, kubectl api code.
This is perhaps something the provider library should offer but the above work around works just fine for me.
FYI this is the code i discovered https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kubectl/pkg/generate/versioned/secret_for_docker_registry.go. Came across it while looking at https://github.com/kubernetes/kubernetes/pull/72344

[dpkirchner]

Thanks for that @nixgadget . I'm adding a "complete" example based on your work, showing how to create the service account and pull secret in terraform (excluding the google_storage_default_object_acl and google_storage_bucket_acl stuff), feel free to add it to the docs:

resource "google_service_account" "image-puller" {
  account_id = "image-puller"
  display_name = "Image Puller"
}

resource "google_service_account_key" "image-puller" {
  service_account_id = google_service_account.image-puller.name
  public_key_type    = "TYPE_X509_PEM_FILE"
}

resource "kubernetes_secret" "image-puller" {
  metadata {
    name = "image-puller"
  }

  data = {
    ".dockerconfigjson" = jsonencode({
      "auths" : {
        "https://gcr.io" : {
          email    = data.google_service_account.image-puller.email
          username = "_json_key"
          password = trimspace(base64decode(google_service_account_key.image-puller.private_key))
          auth     = base64encode(join(":", ["_json_key", base64decode(google_service_account_key.image-puller.private_key)]))
        }
      }
    })
  }

  type = "kubernetes.io/dockerconfigjson"
}

[kasunt-nixdev]

@dpkirchner bang on. i like service account approach better than what i had done. Thanks.

[hashibot]

This issue has been open 180 days with no activity. If this issue is reproducible with the latest version of the provider and with Terraform 0.12, please comment. Otherwise this issue will be closed in 30 days.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!