Splunk-connect-for-kubernetes: daemonset splunk-kubernetes-logging not started on openshift

[schose]

What happened:

When deploying SCK v1.4.3 using helm the daemonset for logging is not starting due to privileged container security settings.

What you expected to happen:

daemonset need to be startet to collect container logsfiles.

How to reproduce it (as minimally and precisely as possible):

start openshift playground: https://learn.openshift.com/playgrounds/openshift44/

oc create ns splunk-logging
oc config set-context --current --namespace=splunk-logging

helm install -f config.yml my-splunk-connect https://github.com/splunk/splunk-connect-for-kubernetes/releases/download/1.4.3/splunk-connect-for-kubernetes-1.4.3.tgz --namespace=splunk-logging

oc describe daemonset/my-splunk-connect-splunk-kubernetes-logging --namespace=splunk-logging

Errorevents:

 Warning  FailedCreate  27s (x14 over 68s)  daemonset-controller  Error creating: pods "my-splunk-connect-splunk-kubernetes-logging-" is forbidden: unable to validate against any security context constraint: [spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[2]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000160000, 1000169999] spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed]

Anything else we need to know?:

To get this work you need to create a SecurityContextConstraints for the SA User. Example: https://github.com/schose/splunk-connect-for-kubernetes/blob/openshift/helm-chart/splunk-connect-for-kubernetes/charts/splunk-kubernetes-logging/templates/ssc.yaml

unfortunately this breaks "helm install" functionality. "helm template" + "oc apply" is working..

Environment:

• Kubernetes version (use kubectl version):
  ```
  $ oc version
  Client Version: 4.4.0-202005290638-1960dd7
  Server Version: 4.4.8
  Kubernetes Version: v1.17.1+3f6f40d
  ````
• Ruby version (use ruby --version):
• OS (e.g: cat /etc/os-release):
• Splunk version:
• Others:

[mwang2016]

Thank you @schose, I've created ticket for this issue. Our team will look into it.

[matthewmodestino]

I have been able to edit the templates directly to remove this:

https://github.com/splunk/splunk-connect-for-kubernetes/blob/53d5b0c1e333ad04c16a556aedd137f15c95a630/helm-chart/splunk-connect-for-kubernetes/charts/splunk-kubernetes-logging/templates/daemonset.yaml#L49

Looks like in a few clusters ( OpenShift, Gravity, etc) all we need is:

https://github.com/splunk/splunk-connect-for-kubernetes/blob/53d5b0c1e333ad04c16a556aedd137f15c95a630/helm-chart/splunk-connect-for-kubernetes/charts/splunk-kubernetes-logging/templates/daemonset.yaml#L50

[rockb1017]

Thank you Matthew. so does it mean we need to remove line number 49 for openshift and gravity?

[matthewmodestino]

I believe we may be able to get rid of it for all...need to test...

[rockb1017]

okay thank you!

[shane-emery]

I have tested removing the privledged line in the deamonset.yaml file and just leaving the runAsUser: 0 . In OKD 4.5 with SCK helm chart version 1.4.3 is works. I have a new problem with the /var/log mount permission being denied :D

[jj1yang]

Shane, Is the problem with the /var/log mount permission being denied on the OKD 4.5/SCK 1.4.3 platform? Thanks. -Jim

[shane-emery]

I believe so.  The /var/log from the node can be mounted, but if I look at the fluentd logs or manually login and run an ls command on the folder…I get a permission denied.  I am told it is an issue with SCC policy maybe not in place for the splunk-logging role?

Is there documentation somewhere on creating/attaching the required SCC to the SCK service account?

[shane-emery]

This got it working for me.

1. readd line 49
2. oc adm policy add-scc-to-user privileged system:serviceaccount:splunk-connect:splunk-kubernetes-logging

[matthewmodestino]

bingo...this works because it allows the service account the permission to runAsUser 0 which is what is required. we do not need privilege!! so you can be sure to set that to false (the openshift helm chart is currently default true, pending confirmation it can be removed)....all you need is "runAsUser 0" because the container logs are stored as "root:root" by default.

This is the key section in the deployment.

https://github.com/splunk/splunk-connect-for-kubernetes/blob/8ba455e7f988d129c9bfa13fdcf2025c6c4ae724/helm-chart/splunk-connect-for-kubernetes/charts/splunk-kubernetes-logging/templates/daemonset.yaml#L52-L54

and as you said the other item is then giving the service account the permission to run the pod as user 0 aka root, but NOT necessarily a "privileged" workload.