Terraform-provider-kubernetes: Partial security_context blocks are explicitly setting runAsUser/runAsGroup/fsGroup to 0 instead of leaving them unset

[jhoos]

Recently while setting up our EKS cluster to use "IAM roles for Service Accounts", we ran into a problem with the security_context block, as we are using it to set fs_group in order to allow the token files injected by EKS to be readable by processes that are not running as root. We discovered that when doing this, the Kubernetes provider was also setting run_as_user and run_as_group in the deployment to 0 without telling us.

The various fields in the security_context block of container specifications are documented as being optional in both the Kubernetes documentation and the Terraform Kubernetes provider documentation. When the fields are not set in the block, the provider is defaulting their values to 0/false and sending them to Kubernetes, instead of just leaving the fields unset entirely.

This behavior is incorrect, because in Kubernetes, the default values for these fields are not 0. For the run_as_user and run_as_group fields, the defaults are in fact declared by the container images, via the Docker USER command.

For the fs_group field, I'm not completely certain at this point whether Kubernete's behavior if the field is not set is basically akin to setting it to 0, but I suspect it's not entirely, since according to the Kubernetes documentation (https://kubernetes.io/docs/tasks/configure-pod-container/security-context/), "Since fsGroup field is specified, all processes of the container are also part of the supplementary group ID ...", which would presumably have undesired effects if fsGroup were defaulted to 0 and runAsGroup (or the group set in the container) were set to something other than 0.

Terraform Version

Terraform v0.12.15
Kubernetes provider version 1.10.0

Affected Resource(s)

Please list the resources as a list, for example:

• kubernetes_deployment
• probably anything with a container/pod spec in it that can have a security_context block

Terraform Configuration Files

This is an example to demonstrate the problem; I used the memcached image because it contains a Docker USER command, similar to the in-house containers we ran into the problem with.

provider "kubernetes" {
    version = "~> 1.10"
    config_context_auth_info="docker-desktop"
    config_context_cluster="docker-desktop"
}

resource "kubernetes_deployment" "test" {
  metadata {
    name = "test"

    labels = {
      name = "test"
    }
  }

  spec {
    selector {
      match_labels = {
        name = "test"
      }
    }

    template {
      metadata {
        labels = {
          name = "test"
        }
      }

      spec {
        security_context {
          fs_group = 100
        }

        container {
          image = "memcached:1.5.20-alpine"
          name  = "test"
        }
      }
    }
  }
  timeouts {
    create = "30s"
  }
}

Debug Output

https://gist.github.com/jhoos/fe4ae247dfbbba5912efe87b1dfac832

Expected Behavior

The Kubernetes provider should have created a deployment containing a spec.template.spec that contained: ```
securityContext:
fsGroup: 100

### Actual Behavior
It actually created a securityContext containing: ```
      securityContext:
        fsGroup: 100
        runAsGroup: 0
        runAsNonRoot: false
        runAsUser: 0

and the apply failed because the container refused to start, due to the fact that memcached refuses to run as root (under normal circumstances, without the runAsUser: 0, the container would start with the default user set by its Dockerfile USER command).

Steps to Reproduce

1. terraform apply

Important Factoids

Memcached container's Dockerfile (https://github.com/docker-library/memcached/blob/master/alpine/Dockerfile) contains a USER memcached command, which Kubernetes respects if runAsUser is not set in securityContext

[jhoos]

I've done some digging in the provider code, and while obviously I don't know all the ins and outs after just one day of digging, I don't see a simple way to fix this. From what I can tell, by the time the code gets into the expandPodSecurityContext function that populates the data going out to Kubernetes, the data structure it's receiving has been converted from Terraform's ResourceData structure into a basic map, and the data in that map has been defaulted to 0 already at that point, and information about whether the original config contained the key is lost.

It would be fairly simple to change the code in that function (and its sister expandContainerSecurityContext function) to not send the fields to Kubernetes if they're 0, but that just doesn't feel right as there may be legitimate reasons why someone would want to explicitly set it to 0...

[pdecat]

by the time the code gets into the expandPodSecurityContext function that populates the data going out to Kubernetes, the data structure it's receiving has been converted from Terraform's ResourceData structure into a basic map, and the data in that map has been defaulted to 0 already at that point, and information about whether the original config contained the key is lost.

That's indeed the issue as it prevents using the GetOkExists() function on the resource data where it is needed in order to determine if the value is actually set in the configuration.

It would be fairly simple to change the code in that function (and its sister expandContainerSecurityContext function) to not send the fields to Kubernetes if they're 0, but that just doesn't feel right as there may be legitimate reasons why someone would want to explicitly set it to 0...

Agreed, not an option as that it's a legitimate use case to define runAsUser as 0.
Also the defaulting also applies to the other fields: runAsNonRoot / runAsGroup.

The alternative I'm experimenting with is to defer the type conversion in the expansion code and use GetOkExists().
It works but requires a major overhaul of all the expanders functions, e.g. with the following configuration:

resource "kubernetes_deployment" "test" {
  metadata {
    name = "%s"

    labels = {
      Test = "TfAcceptanceTest"
    }
  }

  spec {
    selector {
      match_labels = {
        Test = "TfAcceptanceTest"
      }
    }

    template {
      metadata {
        labels = {
          Test = "TfAcceptanceTest"
        }
      }

      spec {
        security_context {
          fs_group            = 100
          supplemental_groups = [101]
        }

        container {
          image = "%s"
          name  = "containername"
        }
      }
    }
  }
}

the following is sent/received to/from the API:

    "securityContext": {                                                                                                                                                                      
     "supplementalGroups": [                                                                                                                                                                  
      101                                                                                                                                                                                     
     ],                                                                                                                                                                                       
     "fsGroup": 100                                                                                                                                                                           
    },                                                                                                                                                                                        

[larkintuckerllc]

Confirmed that this continues to be an issue in:

Terraform: 0.13.2
Kubernetes provider version: 1.13.2x4

The only work-around that I can think of is to manually set the user and group until this is resolved. The problem, however, with this approach is that now the k8s container configuration is tightly coupled with the container image definition, i.e., need to make sure that the container can indeed run as user/group 1000.

security_context {
  allow_privilege_escalation = false
  read_only_root_filesystem  = true
  run_as_non_root            = true
  run_as_group               = 1000 # ISSUE: https://github.com/hashicorp/terraform-provider-kubernetes/issues/695
  run_as_user                = 1000 # ISSUE: https://github.com/hashicorp/terraform-provider-kubernetes/issues/695
}

[gws]

Was this fixed in the v2 release? I didn't see anything in the change log but I am able to omit run_as_user and run_as_group now in conjunction with run_as_non_root = true and the provider appears to be doing something much closer to what kubectl apply would do.

[pdecat]

Certainly fixed by https://github.com/hashicorp/terraform-provider-kubernetes/pull/1093

[aareet]

Thank you for confirming @pdecat, I'll close this, in that case.

[shane-kearns]

runAsNonRoot has the same issue (being explicitly set to false), and is not included in #1093

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!