Terraform-provider-kubernetes: New resource: GKE Managed Certificate

[brendanator]

Affected Resource(s)

Please list the resources as a list, for example:

• kubernetes_ingress

Expected Behaviour

Now the ingress resource is available it is nice and easy to setup ingress to load balance a backend service over http. Ideally it would only be available over https.

According to https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs a managed certificate is easy to create

apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
  name: example-certificate
spec:
  domains:
    - example.com

and using a certificate created this way with terraform ingress works correctly.

As this resource is not available I tried using google_compute_managed_ssl_certificate

resource "google_compute_managed_ssl_certificate" "example_certificate" {
  provider = "google-beta"

  name = "example-certificate"

  managed {
    domains = ["example.com."]
  }
}

but this never provisions and ingress doesn't switch over to https.

In both cases the ingress looks like this

resource "kubernetes_ingress" "example" {
  metadata {
    name = "example"

    annotations {
      "kubernetes.io/ingress.allow-http"       = "false"
      "networking.gke.io/managed-certificates" = "example-certificate"
    }
  }

  spec {
    rule {
      host = "example.com"

      http {
        path {
          backend {
            service_name = "example"
            service_port = 80
          }
        }
      }
    }
  }
}

Is there any scope in this provider to cloud specific extensions? If not, are there any suggestions as to how to provision this with terraform?

[sarneaud]

Try making sure the ingress is dependent on the cert in Terraform by interpolating the example-certificate. That way the ingress should be created after the cert.

GKE's ingress integration is a PITA when it comes to correctly reflecting config updates, but it should work cleanly if the cert exists first. I can confirm that the ingress works with certs that are created directly with gcloud and not with ManagedCertificate.

[caquino]

I've been trying to achieve the same, but for some reason the certificate is never attached to my load balancer resource and it stays on FAILED_NOT_VISIBLE.

Did any of you succeed without the ManagedCertificate resource? should the annotation for pre-shared certificate should be used instead?

For me the certificate only managed to be provisioned when I switched to the pre-shared annotation, but the load balancer accepts connections on 443 without negotiating SSL, which is really weird.

[brendanator]

I can confirm that I tried @sarneaud's suggestion and it didn't work. I get FAILED_NOT_VISIBLE like you @caquino

In the end I did this but it's a bit of a messy solution

resource "null_resource" "example_cert" {
  provisioner "local-exec" {
    command = <<EOT
kubectl create -f - -- <<EOF
apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
  name: example-certificate
spec:
  domains:
    - example.com
EOF
EOT
  }
}

[caquino]

@brendanator I managed to get it working with the following solution:
First by creating a certificate:

resource "google_compute_managed_ssl_certificate" "default" {
  provider = "google-beta"

  name = "${var.name}-${var.environment}-ssl"

  managed {
    domains = ["www.${var.domain_name}"]
  }
}

The service should be created with type NodePort:

resource "kubernetes_service" "proxy_svc" {
  metadata {
    namespace = "default"
    name      = kubernetes_deployment.proxy_dep.metadata.0.name
  }

  spec {
    type             = "NodePort"
    session_affinity = "ClientIP"

    port {
      name        = "http"
      protocol    = "TCP"
      port        = 80
      target_port = 80
    }

    selector = {
      app = kubernetes_deployment.proxy_dep.metadata.0.labels.app
    }
  }
  depends_on = ["google_container_cluster.cluster"]
}

And the ingress configuration:

resource "kubernetes_ingress" "default" {
  metadata {
    name = "${var.name}-${var.environment}-ingress"

    annotations = {
      "ingress.gcp.kubernetes.io/pre-shared-cert"   = google_compute_managed_ssl_certificate.default.name
      "kubernetes.io/ingress.global-static-ip-name" = google_compute_global_address.address.name
    }
  }

  spec {
    rule {
      http {
        path {
          backend {
            service_name = kubernetes_service.proxy_svc.metadata.0.name
            service_port = 80
          }
        }
      }
    }
  }
}

The trick was to use ingress.gcp.kubernetes.io/pre-shared-cert instead of networking.gke.io/managed-certificates

Hope it helps

[brendanator]

Thanks @caquino, I can confirm that using "ingress.gcp.kubernetes.io/pre-shared-cert" works

For anyone else doing this, I had to wait about 15 minutes for the cert to provision and another few minutes for ingress to start using it

[orkenstein]

I'd still like to have ManagedCertificate resource available in kubernetes provider.

[orkenstein]

@caquino thanks for you post. How to force load balancer to use HTTPS in that case? In my setup it still goes over HTTP all the time.

[marcmarcet-sage]

I agree with @orkenstein that the solution to this should be a ManagedCertificate resource for kubernetes.

[essjayhch]

The problem is that from what I can tell, by interacting with this from the kubectl side as well as the terraform google_compute_managed_ssl_certificate is that when an ingress is created, it will only provision an HTTP target proxy for the load balancer that is configured. The only way that the HTTPS target proxy, which is where the certificate gets terminated on, is by way of ManagedCertificate resources for these specific instances. This is probably a side effect of the way that certificates are a prerequisite of provisioning the proxy in the first place.

[griffinator76]

Not sure why this issue has been closed? It would simplify deploying secured apps on GKE using this custom K8s resource instead of the workaround described above if there was an associated Terraform resource:
https://github.com/GoogleCloudPlatform/gke-managed-certs
https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs