Terraform-provider-kubernetes: Can not set internal annotations to specify load balancer ssl certs

[udangel-r7]

With the recent change of disallowing internal annotations it is no longer possible to setup internal annotations.

I want to use these internal annotations to provide SSL certificates/arn configuration for the load balancers. Any idea for a workaround?

Affected Resource(s)

• service

Terraform Configuration Files

"aws_elb_hosted_zone_id" "main" {}

data "aws_acm_certificate" "hello" {
  domain   = "*.hello.com"
  statuses = ["ISSUED"]
}

resource "kubernetes_service" "hello" {
  metadata {
    name = "terraform-hello-integrated-example"

    annotations {
      "service.beta.kubernetes.io/aws-load-balancer-ssl-cert" = "${data.aws_acm_certificate.hello.arn}"
    }
  }

  spec {
    selector {
      app  = "hello"
    }

    port {
      port             = 80
      target_port = 80
    }

    type = "LoadBalancer"
  }
}

resource "aws_route53_record" "hello" {
  zone_id = "HELLO"
  name    = "hello"
  type    = "A"

  alias {
    name                   = "${kubernetes_service.hello.load_balancer_ingress.0.hostname}"
    zone_id                = "${data.aws_elb_hosted_zone_id.main.id}"
    evaluate_target_health = true
  }
}

[radeksimko]

Hi @udangel-r7
I'm sorry for any inconvenience caused by this.

Allowing users to specify internal (kubernetes.io) annotations is something I plan to look into eventually, but it's non trivial for us to support. See full explanation at https://github.com/terraform-providers/terraform-provider-kubernetes/pull/50#issue-251016641

[fbsb]

Hi, I am also affected by this change. Is there any known workaround to configure SSL for the load balancers on AWS?

[frankthelen]

The same problem exists when your LB shall not have an external IP, e.g., in development environments with only internal IPs. I'd be happy if this issue can be solved. Thank you.

annotations {
   "service.beta.kubernetes.io/aws-load-balancer-internal" = "0.0.0.0/0"
}

[jsullivanLI]

Is there a previous version of Terraform where internal annotations are allowed?

[pdecat]

Hi, same issue for me, as I need to specify the well known pv.beta.kubernetes.io/gid annotation on my persistent volumes.

I've read https://github.com/terraform-providers/terraform-provider-kubernetes/pull/50#issue-251016641 and understand that the root issue from the terraform point of view is more with _dynamic_ kubernetes annotations that potentially change all the time like those listed at https://kubernetes.io/docs/reference/labels-annotations-taints/ :

• beta.kubernetes.io/arch
• beta.kubernetes.io/os
• kubernetes.io/hostname
• beta.kubernetes.io/instance-type
• failure-domain.beta.kubernetes.io/region
• failure-domain.beta.kubernetes.io/zone

As maintaining the list of such annotations exhaustively can seem overwhelming if not impossible as time goes, what about allowing to whitelist annotations by configuration?

I guess doing so at the provider level would be practical:

provider "kubernetes" {
  internal_annotations_whitelist = [
    "pv.beta.kubernetes.io/gid",
    "service.beta.kubernetes.io/aws-load-balancer-internal",
    "service.beta.kubernetes.io/aws-load-balancer-ssl-cert",
 ]
}

What do you think?

PS: I'll use the pod security context's fs_group field to resolve for my current FS GID issue, but I believe whitelisting will come in handy anyway.

Edit: #325 is a better alternative as it does not require the user to actively maintain the whitelist.

[bobbytables]

This is pretty annoying, why enforce this at all? Doesn't kubernetes API prevent this?

[Phylu]

Actually there is a good list regarding the aws specific annotations here: https://gist.github.com/mgoodness/1a2926f3b02d8e8149c224d25cc57dc1

I think a whitelist would be the best solution currently.

[bitglue]

Further broken use cases include assigning an ELB to a service: https://gist.github.com/mgoodness/1a2926f3b02d8e8149c224d25cc57dc1

And allowing syscalls in pods: https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#setting-sysctls-for-a-pod

[offlinehacker]

By my opinion this check is unneeded, it's up to user to not use these annotations, and use them when he knows what he is doing.

[r-moiseev]

Same here, I am trying to make storage class to be default. And I need storageclass.kubernetes.io/is-default-class annotation.

Using of this annotation is not restricted by Kubernetes and documented there https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

[kattelus]

Same problem with azure internal load balancer:
https://docs.microsoft.com/en-us/azure/aks/internal-lb
service.beta.kubernetes.io/azure-load-balancer-internal: "true"

[kattelus]

Any possibility allow annotation usage e.g. change this check to just output warning or something? Now this means that everyone using e.g. internal load balancers in azure or aws or otherwise defining additional parameters to those fail? Or is there any easy way to make those work in other ways?

[bitglue]

Please read https://github.com/terraform-providers/terraform-provider-kubernetes/pull/50#issue-251016641, which explains why this check exists.

[Phylu]

I think that especially because of the reasoning behind #50 this is a big issue for many of us.

Unfortunately, the usage after this change showed that some tasks that were possible before are not possible anymore. For those of us who rely on these features (internal load balancers, aws certificates, ...) this makes those things really hard do achieve.

Therefore, I would be really interested in feasible solution such as some whitelisting of certain annotations or similar.

[chreichert]

Same problem here, when having the external IP for a loadbalancer inside a different resource group on azure:
The annotation needed for this is not allowed:
service.beta.kubernetes.io/azure-load-balancer-resource-group: RG_SANDBOX_006_QA_PROD_PublicIP_Pool

Whitelisting these necessary annotations would be highly appreciated.
The disallowing renders Terraform useless for us atm.

[mb290]

I'm also now hit with the bug and having the list of approved annotations would seriously help. I'm not a developer and had never even ventured into Go before, but managed to piece together the following extension to the existing function validateAnnotations in https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/kubernetes/validators.go.

func validateAnnotations(value interface{}, key string) (ws []string, es []error) {
    m := value.(map[string]interface{})
    for k, _ := range m {
        errors := utilValidation.IsQualifiedName(strings.ToLower(k))
        if len(errors) > 0 {
            for _, e := range errors {
                es = append(es, fmt.Errorf("%s (%q) %s", key, k, e))
            }
        }

        permittedInternalKubAnnotations := map[string]bool {
            "service.beta.kubernetes.io/azure-load-balancer-internal": true,
            "service.beta.kubernetes.io/azure-load-balancer-resource-group": true,
            "service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval": true,
            "service.beta.kubernetes.io/aws-load-balancer-access-log-enabled": true,
            "service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name": true,
            "service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix": true,
            "service.beta.kubernetes.io/aws-load-balancer-backend-protocol": true,
            "service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled": true,
            "service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout": true,
            "service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout": true,
            "service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled": true,
            "service.beta.kubernetes.io/aws-load-balancer-internal": true,
            "service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": true,
            "service.beta.kubernetes.io/aws-load-balancer-ssl-cert": true,
            "service.beta.kubernetes.io/aws-load-balancer-ssl-ports": true,
        }

        val := permittedInternalKubAnnotations[k];

        if isInternalKey(k) && val != true {
            es = append(es, fmt.Errorf("%s: %q is internal Kubernetes annotation", key, k))
        }
    }
    return
}

I've tested the above and it works with my specific use case of needing an internal load balancer in Azure for provisioning a POC. Whilst i understand why the check for internal providers was built in the first place, this is severely impacting the usefulness of the Kubernetes provider without the ability for this exception.

If there is appetite for me to submit a pull request i'm more than happy to but I suspect there is a significantly more efficient way of the above check :). I'll also have to figure out the part of that stops the attempt to re-enable the load balancer every time as this is what happens when I re-run:

  ~ kubernetes_service.nginx
      metadata.0.annotations.%:                                                       "0" => "1"
      metadata.0.annotations.service.beta.kubernetes.io/azure-load-balancer-internal: "" => "true"

[juhan88]

Are there plan to support annotations? Specifically creating kubernetes service's with internal private elbs

[mbarrien]

Here's another set of "internal" kubernetes.io annotations that are disallowed: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/master/docs/ingress-resources.md#annotations. This ingress controller is absolutely unusable without whitelisting the large list.

C'mon guys, the examples where this is blocking people for legitimate usage are far outweighing and far outnumbering the original reasons for blocking it, and severely degrades the usability of this provider. Please revert #50 or reduce its scope.

[andyvanosdale]

Now that development is starting back up on this, it would be great to have this revisited. I would love to use this in some different projects, but kube2iam used the internal annotations to allow pods to assume an IAM role.

[dh-harald]

I'm not a go developer, so I won't open a pull request, but here's my solution for this problem:
https://github.com/dh-harald/terraform-provider-kubernetes/commit/d0b48e8ec7e991f702aa6e0ea1dfdfef6380c509

Idea: @pdecat / Solution: @mb290

Here's an example:

provider "kubernetes" {
  internal_annotations_whitelist = [
    "storageclass.kubernetes.io/is-default-class",
  ]
}

resource "kubernetes_storage_class" "gp2" {
  metadata {
    name = "gp2"

    annotations {
      "storageclass.kubernetes.io/is-default-class" = "true"
    }
  }
  storage_provisioner = "kubernetes.io/aws-ebs"
  parameters {
    type      = "gp2"
    encrypted = "true"
  }
}

I don't know is it known, but if you're using true instead of "true", terraform translates it into 1, and it won't work

[jakubigla]

Haha, lack of support of internal annotations just made this and couple of other Kubernetes resources useless. Especially in enterprise. Thanks for removing this. Now automation of this just got much complicated.

[dh-harald]

@jakubigla For similar reason, we've started using https://github.com/sl1pm4t/terraform-provider-kubernetes instead of the official... It has no problem with internal annotations, there's a lot of beta APIs are supported, which we're using (ingress, clusterrole, etc)... Worth a try instead of using (for example) dumb hacks with null_provider
I though, that the official provider could be usable if we're helping them, but it looks nothing has changed... So I'm very disappointed

[rpatrick00]

Are we going to integrate my PR #244 to address this problem that makes the Kubernetes provider unusable in real-world scenarios?

[rpatrick00]

@dh-harald, the main issue I had with using https://github.com/sl1pm4t/terraform-provider-kubernetes was it's lack of support for exec authentication (because it is using an older k8s client) which is needed to integrate with AWS (at least without hacks). I took your suggested changes and filed PR #244. I really don't understand why #Hashicorp is allowing this provider to languish and become useless by both making changes that prevent its use in real-world scenarios and not adding support for current k8s objects like Ingress, ClusterRole, etc.

[andyvanosdale]

Alex has been doing some great work on this provider, and there are active PRs for different Kubernetes resources. If there is a need for a resource, please file an issue so that people know and feel free to work on it.

[rpatrick00]

@enozero, I agree that his version of the provider is nice. Unfortunately, https://github.com/sl1pm4t/terraform-provider-kubernetes/issues/36 seems to be a bigger change than I am able to take on right now.

[dh-harald]

@rpatrick00 for me, aws-iam-authenticator is working fine. They merged a newer go library, far as I know

[rpatrick00]

@dh-harald I was able to get it to work with AWS authentication using the workaround but this forced me to do some hacking of my kubeconfig file from my terraform scripts that I didn't like. If this is working without the need for the workaround now, he should close sl1pm4t#36 so that everyone knows.

[dh-harald]

No, I don't hack anything... I just configure the provider:

provider "kubernetes" {
  host                   = "${aws_eks_cluster.main.endpoint}"
  token                  = "${data.external.aws_iam_authenticator.result["token"]}"             # https://github.com/hashicorp/terraform/issues/17173
  load_config_file       = false
  cluster_ca_certificate = "${base64decode(aws_eks_cluster.main.certificate_authority.0.data)}"
}

data "external" "aws_iam_authenticator" {
  program = ["sh", "-c", "AWS_PROFILE=${var.aws_account} aws-iam-authenticator token -i ${aws_eks_cluster.main.name} -r ${aws_iam_role.admin.arn} | jq -r -c .status"]
}

When you created the cluster first, you need to skip the -r (role) part, until you didn't add the configmap for the role as well

[dh-harald]

@rpatrick00 it looks, something weird happend, it just stops working, and I'm facing the same problem like https://github.com/sl1pm4t/terraform-provider-kubernetes/issues/36 describes... I don't know, why it worked before, and why it stops working.

[rikkuness]

Also just run into this trying to set storage classes on persistent volume claims in OpenShift, without the ability to set set the annotations to specify a storage tier the claims will sit in pending and Terraform times out with "Still creating..."

[bitglue]

Alex has been doing some great work on this provider, and there are active PRs for different Kubernetes resources. If there is a need for a resource, please file an issue so that people know and feel free to work on it.

Does this mean the policy of refusing to support beta resources is no longer in effect? I think most (if not all) the use cases for annotations brought up here are beta.

[tdmalone]

Has anyone come across a workaround for this (either internal annotations in general, or HTTPS load balancer support on AWS, without shelling out to kubectl)?

I'm new to Kubernetes and learning as I go, but it sounds like https://github.com/terraform-providers/terraform-provider-kubernetes/pull/50 was a bit of a sledgehammer to apply, given use of _certain_ internal annotations seems to be quite widespread. It would seem to me to be much more user friendly to allow ignore_changes and/or a (user-configurable, if necessary) whitelist to be used instead.

(As a general comment, it appears it might get a bit hard to use the Terraform Kubernetes provider, when documentation elsewhere points to a particular way to do it with kubectl/YAML that just isn't supported with Terraform - I had assumed after some initial basic experimentation that I'd be able to convert almost anything from YAML to HCL).

[tdmalone]

List of related issues/PRs:

• https://github.com/terraform-providers/terraform-provider-kubernetes/pull/50 (also https://github.com/terraform-providers/terraform-provider-kubernetes/issues/37 and https://github.com/terraform-providers/terraform-provider-kubernetes/pull/40)
• https://github.com/terraform-providers/terraform-provider-kubernetes/pull/199
• https://github.com/terraform-providers/terraform-provider-kubernetes/issues/200
• https://github.com/terraform-providers/terraform-provider-kubernetes/pull/244
• https://github.com/terraform-providers/terraform-provider-kubernetes/pull/325

[tdmalone]

And here's a dodgy workaround (partial implementation - it provisions, but not updates or destroys). YMMV.

resource "null_resource" "eks_service_metadata_annotations" {
  provisioner "local-exec" {
    command = <<EOF
kubectl annotate --overwrite --namespace ${local.eks_namespace} \
  service ${kubernetes_service.sample.metadata.0.name} \
  'service.beta.kubernetes.io/aws-load-balancer-backend-protocol=http' \
  'service.beta.kubernetes.io/aws-load-balancer-ssl-cert=${local.acm_certificate_arn}'
EOF
  }
}

[holms]

So is there's any progress with this? I mean it's 2019 and April already hello? @pdecat @AlexBoulyga Anyone? Help? Devops can't live without terraform!

[Xyrodileas]

And here's a dodgy workaround (partial implementation - it provisions, but not updates or destroys). YMMV.

resource "null_resource" "eks_service_metadata_annotations" {
  provisioner "local-exec" {
    command = <<EOF
kubectl annotate --overwrite --namespace ${local.eks_namespace} \
  service ${kubernetes_service.sample.metadata.0.name} \
  'service.beta.kubernetes.io/aws-load-balancer-backend-protocol=http' \
  'service.beta.kubernetes.io/aws-load-balancer-ssl-cert=${local.acm_certificate_arn}'
EOF
  }
}

Thanks a lot, I would prefer that the official provider support k8s annotation, but ¯_(ツ)_/¯

[rpatrick00]

@Xyrodileas, most of us have gone off the reservation and built our own version of the provider to get around this problem. I created PR #244 for the main provider to address the problem in the way suggested by one of the contributors. Others, like sl1pm4t / terraform-provider-kubernetes, have forked the code since Hashicorp seems uninterested in making this provider capable of real-world deployments. Either way, the only real workaround at this point is to either build your own custom provider or use the local-exec hack to run kubectl.

[erick-thompson]

Is this still stalled? I'm trying to use External DNS, and I've run into this issue. It's very annoying. I don't like this idea of terraform making decisions about what I can and can't do in k8s. I've run into other issues like this (not being able to set automountServiceAccountToken on deployments).

Is there a workaround for this particular issue?

[tdmalone]

@erick-thompson Workaround is currently to shell out to kubectl using a null_resource and local-exec provisioner (see a few comments up for a simple example). Fix appears to not be coming anytime soon, I think likely not to happen until well after Terraform 0.12 launches.

Regarding automountServiceAccountToken, there is a potential workaround here.

Alternatively there's also a number of forks of the provider, such as this one, which appear to be a good option (I haven't gone down this path just yet). Or of course you could also build a provider locally based on any of the PRs that have been submitted to solve this issue.

[erick-thompson]

I ended up using that workaround for the service account token, but I wish I had those hours back, when I was trying to debug what was going on and how to fix it.

For this issue, I'll likely just use terraform to create the yaml file and apply it later. The problem is that I provision the cluster (in my case AKS) from the same terraform as I wanted to configure the service for, so I can't use local-exec unless I do something to ensure that the kubectl config is set up correctly.

[tdmalone]

but I wish I had those hours back, when I was trying to debug what was going on and how to fix it.

Yeah, I hear you. It's one thing to make questionable design decisions; it's another thing entirely to not document them!

I can't use local-exec unless I do something to ensure that the kubectl config is set up correctly.

I'm on EKS so I'm not sure what's available for AKS, but I'm using aws eks update-kubeconfig --name "my-cluster-name" && kubectl apply ... in my local-exec. I'm not sure if Azure has a similar tool or automated way to configure kubeconfig.

[erick-thompson]

I took a similar approach and used "az aks get-credential". It works well enough, but I find I have to be a lot more explicit about my dependencies.

[dayglojesus]

Plan works, apply produces this...

Error: kubernetes_service.ingress_nginx: metadata.0.annotations: "service.beta.kubernetes.io/aws-load-balancer-type" is internal Kubernetes annotation

[tdmalone]

This should now be fixed by the merge of #325

[selslack]

Can we have a release tagged with the fix included, please?

[alexsomesan]

Hi. Yes, a release is coming in the next couple of days.
I’m aiming for it to include PodAffinity as well, so I'm looking into
getting that ready as well.

On Fri 17. May 2019 at 19:46, Pavel Eremeev [e-mail]
wrote:

Can we have a release tagged with the fix included, please?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/terraform-providers/terraform-provider-kubernetes/issues/60?email_source=notifications&email_token=AAIL5G3552LIY2B2JR7P2R3PV3VQLA5CNFSM4DYLP6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVVNG7I#issuecomment-493540221,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAIL5G7ACG2BGC4GUM6OMI3PV3VQLANCNFSM4DYLP6HQ
.

>

— Sent from my phone.

[tdmalone]

This has now been released in 1.7.0: https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/CHANGELOG.md#170-may-22-2019
🎉 🚀

[bobbytables]

Rock on 🤘

On Thu, May 23, 2019 at 12:22 AM Tim Malone [e-mail]
wrote:

This has now been released in 1.7.0:
https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/CHANGELOG.md#170-may-22-2019
🎉 🚀

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/terraform-providers/terraform-provider-kubernetes/issues/60?email_source=notifications&email_token=AAD7LGY46QBNL2PYYDJNRXDPWYLYHA5CNFSM4DYLP6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWBAW3Y#issuecomment-495061871,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAD7LG3B6DRAAGEAKPQZPTDPWYLYHANCNFSM4DYLP6HQ
.

>

• Robert Ross

[andyvanosdale]

Thank you @tdmalone for taking the torch on this provider!

[laurawadden]

I am still getting an error - maybe my syntax is incorrect?

* provider.kubernetes: version = "~> 1.7"

resource "kubernetes_secret" "my-secret" {
  metadata {
    name = "my-secret"
    annotations {
      kubernetes.io/service-account.name = "my-account"
    }
  }
  type = "kubernetes.io/service-account-token"
}

Error:

Error: Argument or block definition required

  on x.tf line x, in resource "kubernetes_secret" "my-secret":
  x:       kubernetes.io/service-account = "my-account"

An argument or block definition is required here. To set an argument, use the
equals sign "=" to introduce the argument value.

In the error message kubernetes is underlined, as if it parses it as a keyword.

[dh-harald]

Can you try with quotes?

resource "kubernetes_secret" "my-secret" {
  metadata {
    name = "my-secret"
    annotations {
      "kubernetes.io/service-account" = "my-account"
    }
  }
  type = "kubernetes.io/service-account-token"
}

[laurawadden]

I tried before with quotes and I got:

Error: Invalid argument name

  on xx.tf line 17, in resource "kubernetes_secret" "my-token":
  x:       "kubernetes.io/service-account" = "my-account"

Argument names must not be quoted.

[pdecat]

With terraform 0.12+, you need to specify annotations as a map and with an equal sign:

resource "kubernetes_secret" "my-secret" {
  metadata {
    name = "my-secret"
    annotations = {
      "kubernetes.io/service-account.name" = "my-account"
    }
  }
  type = "kubernetes.io/service-account-token"
}

[laurawadden]

With terraform 0.12+, you need to specify annotations as a map and with an equal sign:

resource "kubernetes_secret" "my-secret" {
  metadata {
    name = "my-secret"
    annotations = {
      "kubernetes.io/service-account" = "my-account"
    }
  }
  type = "kubernetes.io/service-account-token"
}

This worked! Thank you. I was losing my mind. I'm new to Terraform - where could I have found this in reference docs? For this specific issue, neither https://www.terraform.io/docs/providers/kubernetes/r/secret.html#annotations nor https://www.terraform.io/docs/configuration/syntax.html were that helpful.

p.s. for posterity's sake, the correct annotation is "kubernetes.io/service-account.name" - just in case someone ends up reading this comment in more detail. Changed in my comment above.

[pdecat]

where could I have found this in reference docs? For this specific issue, https://www.terraform.io/docs/providers/kubernetes/r/secret.html#annotations nor https://www.terraform.io/docs/configuration/syntax.html were that helpful.

That specific use case is missing from the docs.
The most extensive source for examples are the acceptance test cases, e.g. https://github.com/terraform-providers/terraform-provider-kubernetes/blob/v1.7.0/kubernetes/resource_kubernetes_secret_test.go#L342
Maybe they could be extracted and put somewhere more easily discoverable.

Edit: I've noticed a few examples were outdated for terraform 0.12, working on a PR.

[tdmalone]

This ticket should probably be closed - as the original issue is now resolved :)
Might need a new ticket if the docs aren't right for 0.12 though.

[pdecat]

+1 to close this issue.

The docs should be fine now: https://github.com/terraform-providers/terraform-provider-kubernetes/pull/501

[alexsomesan]

Closed

[NZenitram]

This still isn't working for me with an AWS NLB. I've checked the versions as well.