Terraform-provider-kubernetes: kubernetes_deployment: changing env vars causes re-creation

[dasch]

Terraform Version

Terraform v0.11.8
+ provider.kubernetes v1.3.0

Affected Resource(s)

• kubernetes_deployment

Terraform Configuration Files

variable "kubernetes_host" {}

provider "kubernetes" {}

locals {
  nginx_labels = {
    app = "nginx"
  } 
}

resource "kubernetes_deployment" "nginx" {
  metadata {
    name = "nginx-deployment"
    labels = "${local.nginx_labels}"
  }

  spec {
    replicas = 3

    selector {
      match_labels = "${local.nginx_labels}"
    }

    template {
      metadata {
        labels = "${local.nginx_labels}"
      }

      spec {
        container {
          name = "nginx"
          image = "nginx:1.15.4"

          port {
            container_port = 80
          }

          env {
            name = "MYSQL_HOST"
            value = "1.2.3.4"
          }
        }
      }
    }
  }
}

Expected Behavior

When changing the value of an environment variable, e.g. MYSQL_HOST, the Deployment is updated in-place.

Actual Behavior

When changing the value of MYSQL_HOST, the Deployment needs to be recreated. When using kubectl, this is not the case – the Deployment is updated in-place (or rather, a new version is created behind the scenes.)

[jordan-keyes]

+1 to this. We have a kubernetes_deployment that has a few environment variables starting with "INFO_APP_" for the /info Spring Boot Actuator, and since those update with every deployment, it was causing our deployments to completely re-create without updating the revision.
Removing those has allowed it to be a modification instead of a re-creation.

[pasali]

Since there is no restart in k8s, a lot of k8s user use env variable -which is usually a timestamp- to refresh configMaps. It would be very nice to make this issue disappear.

[campbel]

This is a result of https://github.com/terraform-providers/terraform-provider-kubernetes/blob/3c56c5554be98497160f93d098d64b1fd976edb8/kubernetes/schema_container.go#L225 and was added intentionally https://github.com/terraform-providers/terraform-provider-kubernetes/commit/e5b83644bf4c8346698c80b0fb1106adfc111909

Requires more digging to understand why this was done.

[campbel]

Maybe @appilon or @alexsomesan could help us understand why ForceNew is needed for container env vars?

[campbel]

Here is the PR https://github.com/terraform-providers/terraform-provider-kubernetes/pull/73 it was part of adding Cluster-Role-Binding support... though I'm not sure why. Probably need to go look at the discussions on the fork.

[coryflucas]

I believe the changes in #73 were introduced because they are correct when dealing with a pod directly. Looks like it should actually just be set to !isUpdatable rather than explicitly true

[ryanking]

@coryflucas fwiw I just tried that and I didn't appear to help. Of course this is my first time in this code base, so I may have done something wrong...

[ryanking]

I take that back - I was running the wrong version. The change worked!

[pdecat]

FYI, there's already https://github.com/terraform-providers/terraform-provider-kubernetes/pull/389

[heidemn]

I would appreciate if the fix could be merged. This can cause unexpected downtime.

We used env vars with config file hashes to trigger re-deployments when configuration changes.
As a workaround, we're now instead using annotations for this purpose. Labels could also be used.
See also: https://github.com/kubernetes/kubernetes/issues/27081

[dak1n1]

It looks like there were some fixes relating to this issue that merged a while back. I tested with the latest version of the provider and saw the behavior I was expecting (that is, a rolling deployment when the environment variable is updated). Here is an example to illustrate what I mean:

My main.tf was copy/pasted from the first comment of this issue. I created the deployment using terraform apply, and then edited the environment variable and ran the apply again. This was the result:

[dakini@dax 20200414]$ terraform apply
var.kubernetes_host
  Enter a value: testhost

kubernetes_deployment.nginx: Refreshing state... [id=default/nginx-deployment]
[...]
kubernetes_deployment.nginx: Modifying... [id=default/nginx-deployment]
kubernetes_deployment.nginx: Modifications complete after 0s [id=default/nginx-deployment]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

[dakini@dax 20200414]$ kubectl get deployment
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           6m34s

[dakini@dax 20200414]$ kubectl get pods
NAME                                READY   STATUS        RESTARTS   AGE
nginx-deployment-774457d84c-98xg6   1/1     Running       0          4m56s
nginx-deployment-774457d84c-cnxdc   0/1     Terminating   0          4m55s
nginx-deployment-774457d84c-w95gq   1/1     Running       0          4m57s
nginx-deployment-78c8b65dbc-prndq   1/1     Running       0          1s
nginx-deployment-78c8b65dbc-r4pnr   1/1     Running       0          1s

[dakini@dax 20200414]$ kubectl get pods
NAME                                READY   STATUS        RESTARTS   AGE
nginx-deployment-774457d84c-98xg6   0/1     Terminating   0          4m58s
nginx-deployment-774457d84c-w95gq   1/1     Terminating   0          4m59s
nginx-deployment-78c8b65dbc-jgs4v   1/1     Running       0          2s
nginx-deployment-78c8b65dbc-prndq   1/1     Running       0          3s
nginx-deployment-78c8b65dbc-r4pnr   1/1     Running       0          3s

[dakini@dax 20200414]$ kubectl get pods
NAME                                READY   STATUS        RESTARTS   AGE
nginx-deployment-774457d84c-98xg6   0/1     Terminating   0          4m59s
nginx-deployment-774457d84c-w95gq   0/1     Terminating   0          5m
nginx-deployment-78c8b65dbc-jgs4v   1/1     Running       0          3s
nginx-deployment-78c8b65dbc-prndq   1/1     Running       0          4s
nginx-deployment-78c8b65dbc-r4pnr   1/1     Running       0          4s

[dakini@dax 20200414]$ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-78c8b65dbc-jgs4v   1/1     Running   0          4s
nginx-deployment-78c8b65dbc-prndq   1/1     Running   0          5s
nginx-deployment-78c8b65dbc-r4pnr   1/1     Running   0          5s

And here is the same behavior when using kubectl:

[dakini@dax 20200414]$ kubectl set env deployment/nginx-deployment MYSQL_HOST="1.5.5.5"
deployment.apps/nginx-deployment env updated
[dakini@dax 20200414]$ kubectl get pods
NAME                                READY   STATUS              RESTARTS   AGE
nginx-deployment-5b458986d6-4dn5j   1/1     Running             0          2s
nginx-deployment-5b458986d6-87mdv   1/1     Running             0          2s
nginx-deployment-5b458986d6-p4qwh   0/1     ContainerCreating   0          1s
nginx-deployment-78c8b65dbc-jgs4v   0/1     Terminating         0          3m43s
nginx-deployment-78c8b65dbc-prndq   1/1     Terminating         0          3m44s
nginx-deployment-78c8b65dbc-r4pnr   1/1     Running             0          3m44s

[dakini@dax 20200414]$ kubectl get pods
NAME                                READY   STATUS        RESTARTS   AGE
nginx-deployment-5b458986d6-4dn5j   1/1     Running       0          6s
nginx-deployment-5b458986d6-87mdv   1/1     Running       0          6s
nginx-deployment-5b458986d6-p4qwh   1/1     Running       0          5s
nginx-deployment-78c8b65dbc-prndq   0/1     Terminating   0          3m48s

So it seems to be working correctly. Please let me know if anyone still has an issue. Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!