Terraform-provider-kubernetes: Restart pods when configmap/secret is changed

[flmmartins]

Hi there,

I have a code that creates configmap & secret. I noticed that when I change a variable the pods are not restarted. This is a know issue: https://stackoverflow.com/questions/37317003/restart-pods-when-configmap-updates-in-kubernetes

I wonder if we could force a restart using null_resource (or if someone has another idea, I'll be glad to hear it). At first I thought about using resource_version in metadata as trigger but that don't restart in the same execution (you woud have to run apply 2x).

Here is an conceptual example with daemonset but I think with deployment would be similar:

resource "null_resource" "datadog_restart_when_variables_change" {
  depends_on  = ["kubernetes_daemonset.datadog-agent"]
  triggers {
    configmap = "${kubernetes_config_map.datadog-agent.metadata.0.resource_version}"
    secret = "${kubernetes_secret.datadog-agent.metadata.0.resource_version}"
  }
   provisioner "local-exec" {
    command = <<EOT
      kubectl delete pod $(kubectl get pods -n ${local.namespace} | grep ${local.app_name} | cut -d ' ' -f 1) -n ${local.namespace}
  EOT
  }
}

resource_version doesn't work here cause I would have to run terraform 2x. We could add a label but we would have to change everytime I change the code and that can be quite annoying.

Someone has an idea? Tks!

[jrhouston]

A simple thing you can do here, which avoids the resource_version issue of having to run apply twice, is to add an annotation to the deployment or daemonset that is a hash of the data in your config_map and secret. Here is an example:

annotations = {
  config_change = sha1(jsonencode(merge(
    kubernetes_config_map.test_config.data,
    kubernetes_secret.test_secret.data
  )))
}

The downside of this approach is that you will have to duplicate this code for every deployment that uses a particular configmap/secret. It also creates some noise in the diff that may not be immediately intuitive to someone making a config change who didn't set this up.

A more complex alternative is to add an operator to your cluster whose job is to update pods whose configmaps and secrets have changed, such as reloader.

[essjayhch]

The problem is you have an anti-k8s-pattern here because config-maps are mounted at pod creation time. An alternative to the annotation approach to be a little closer to the K8s idiomatic way to do this is to "create a new config-map" and then "modify the deployment to reference that config-map".

This sounds more complicated than it is - Terraform can do soemthing like this for you:

locals {
  config_data = {}
}
resource random_id config_map_name_tail {
    length = 8
    keepers {
       data = local.config_data
   } 
}
resource kubernetes_config_map config {
  metadata {
    name = "name-${random_id.config_map_name_tail.hex}
  }
  data = local.config_data
}

resource kubernetes_deployment {
  spec {
    volume {
       name = config
       config_map {
          name = kubernetes_config_map.config.metadata.0.name
       }
    }
  }
}

Your mileage may vary

[dak1n1]

There is also a Kubernetes native way to do this, using a side-car container that watches for file changes. I noticed this pattern is used by Prometheus, and found an example of how a watch container works.

Basically the side-car container runs in the same pod as your application, and will send a restart command to your application when a change occurs to the file mounted via the configmap. That way, the pod restart isn't necessary, since the application will be notified to re-read the updated config file.

This is the example from the link above. It might be possible to add a reload endpoint to your app and make a container similar to this:

spec:
  containers:
  - name: prometheus
    ...
    volumeMounts:
    - name: config-volume
      mountPath: /etc/prometheus
  - name: watch
    image: weaveworks/watch:master-5b2a6e5
    imagePullPolicy: IfNotPresent
    args: ["-v", "-t", "-p=/etc/prometheus", "curl", "-X", "POST", "--fail", "-o", "-", "-sS", "http://localhost:80/-/reload"]
    volumeMounts:
    - name: config-volume
      mountPath: /etc/prometheus
  volumes:
    - name: config-volume
      configMap:
        name: prometheus-config

[jrhouston]

Just to follow up on this issue and give a more precise answer. We are not going to support this pattern natively in the Kubernetes provider - we feel that it is a brittle anti-pattern to do this at the provisioning stage.

The thinking here is that if you have Secrets and ConfigMaps that resources depend upon, the problem of keeping those resources up to date should be solved at either:

1. The application level by mounting the Secret or ConfigMap as a volume and using a hot-reloading mechanism in the application code (e.g viper) or a sidecar container like weaveworks/watch.

2. The cluster level by running a controller that ensures deployments are using the latest configuration, such as stakater/reloader or pusher/wave

That being said, if you do want to go ahead and do this anyway then you will have to use a workaround like the one described in my earlier comment or by triggering a kubectl rollout-restart using an exec provisioner.

There is an open issue on the Kubernetes repo that tracks native support for this functionality here.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!