Serving: Does privileged container supported?

[cometta]

My service.yaml look like below

apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: applicationX
  namespace: default
spec:
  runLatest:
    configuration:
      revisionTemplate:
        spec:
          serviceAccountName: build-robot
          container:
            image: custom_image/1.0
            securityContext:
              privileged: true

I able to apply this yaml file successfully. However, when i try to mount using blobfuse for azure files, it just hangs. I able to run blobfuse using privileged on a docker on my development machine.

[cometta]

after apply below config, it works in 0.5.2.

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-svc-redirect
spec:
  hosts:
  - "*.windows.net"
  location: MESH_EXTERNAL
  ports:
  - number: 443
    name: https
    protocol: TLS
  resolution: NONE

However on 0.6 , i still get Error from server (InternalError): error when creating "service.yaml": Internal error occurred: admission webhook "webhook.serving.knative.dev" denied the request: mutation failed: must not set the field(s): spec.runLatest.configuration.revisionTemplate.spec.container.securityContext.privileged

[dgerd]

securityContext was allowed in Knative to allow users to configure the runAsUser within the container. From 0.5 -> 0.6 we went from an incomplete blacklist of disallowed fields in Knative to a whitelist of fields we allow, have tests for, and believe that we can support. As part of this one time migration some fields were disabled and privileged was one of them. Given the permissiveness of the 0.5 webhook this was unavoidable, but should not happen again.

priviledged is definitely a field worth some discussion before we decide to re-enable it as it brings with it some challenges in supportability. While Kubernetes for the most part treats the contents within the container as a black box we have a specific set of requirements we expect of Knative containers ( https://github.com/knative/serving/blob/master/docs/runtime-contract.md ) in order to provide higher-level functionality. priviledged offers a backdoor to the developer to break many of the assumptions made in the runtime contract which can result in Knative not working properly.

[amrueda76]

@dgerd Can you verify if metadata.name was one of the decommissioned fields as a result of this change?

[dgerd]

If you are referring to metadata.name within the fieldRef key then yes. This would also be one of those fields.

I have created an issue here ( https://github.com/knative/serving/issues/4190 ) for that, and hope to have a discussion about it at an upcoming API Working Group meeting. There are clearly many good use-cases for the downward API, but we need to figure out the best way to add this back.

[dgerd]

I would like to discuss this (priviledged support) in the next API Working group meeting. It is 10:30am-11:00am Pacific on Wednesday 6/12.

@cometta if you are able to attend and talk more about your use-case that would be awesome. If the date doesn't work for you and you would like to attend we can bump to a future WG meeting.

[dgerd]

Adding some content here prior to today's discussion in the working group.

Where I currently stand on this is that I believe we should keep privileged support disabled in Knative.

The strongest concerns I have for enabling this feature is that:

1. Enabling privileged mode allows a developer to bypass/override any and all constraints listed in the Knative Runtime contract ( https://github.com/knative/serving/blob/master/docs/runtime-contract.md ) that are currently API or operator enforced. As the container is less of a blackbox for Knative this brings in a strong likelihood for Knative functionality to degrade or break because of actions taken in privileged mode.
2. Privileged mode does not work for all Kubernetes container runtimes ( https://kubernetes.io/docs/concepts/workloads/pods/pod/#privileged-mode-for-pod-containers
   ) and may be disabled by a platform-provider/operator for security purposes. This means that a container that uses privileged mode cannot easily be moved between Knative installations without knowledge of the underlying runtime or operator-enabled constraints. This hurts our portability story.
3. As privileged mode is allowing access to the underlying host WHICH underlying host you are scheduled upon becomes more important. The need to know about the underlying host is inherently less serverless, and pushes us more towards supporting nodeSelector and tolerations which we have previously punted upon.

Concerns aside I do want to look at this from a use-case enabling perspective. A lot has happened since Kubernetes added the privileged flag in 2014. I managed to find the original issue adding the privileged flag ( https://github.com/kubernetes/kubernetes/issues/391 ) as I was curious as to the use-case discussion back then and if it is still relevant. Looking through the comments the core use-case that seemed to drive support for privileged back then was doing on-cluster Docker builds which had no good workarounds. Since this point in time there has been an increase in tooling to do rootless builds such as Kaniko ( https://github.com/GoogleContainerTools/kaniko ).

Looking at the specific issue here of FUSE volume support, the addition of volume drivers into Docker and the CSI interface into Kubernetes has enabled API supported ways to mount custom volumes into containers without the need for privileged. Azure looks to offer a FUSE Flex Volume driver for this use-case today (note: I have not tried this): https://github.com/Azure/kubernetes-volume-drivers/tree/master/flexvolume/blobfuse. While mounting volumes into the container will likely slow down container creation and add to cold-start latency it has much lower impact to Knative operating correctly and improved security over privileged.

This ends up looking like this in the PodSpec (taken from example in the documentation):

 volumes:
  - name: test
    flexVolume:
      driver: "azure/blobfuse"
      readOnly: false
      secretRef:
        name: blobfusecreds
      options:
        container: CONTAINER-NAME
        tmppath: /tmp/blobfuse
        mountoptions: "--file-cache-timeout-in-seconds=120"

We don't currently allow non-configmap/secret volumes to be attached to Knative due to state persistence and scaling concerns, but volumes types that inherently support multiple readers/writers seem worthwhile to look into. This still has some concerns with portability across Knative installs, but an error can be exposed much earlier in the lifecycle as the request for the volume is now structured.

[duglin]

@dgerd a lot of your concerns seem more focused on Kube's use of privileged containers rather than Knative directly. I tend to look at Knative (first) as a nicer UX for Kube and as such unless there's a specific Kn-specific technical reason for why we can't expose some Kube features via Knative, I think we should. I do think making Kn's UX ugly is a good reason, but I don't think this one attribute does that when it's already part of the PodSpec. I'd like to try to get as many workloads from Kube to run on Kn as possible.

And +1 to generic PV support :-)

[duglin]

Since @dgerd asked on today's call I just got this feedback w.r.t. using privileged containers:

---

We're using them in order to allow us to generate core dumps for some Java workloads we’re deploying to our clusters…

We’re using a daemon set to apply sysctl -w kernel.core_pattern=core settings on the worker nodes…

---

[duglin]

Also, I misspoke on the call today, we DO allow privileged containers on our cloud - learn something new every day :-)

[duglin]

Couple more uses:

• We use it to run daemon sets that install Spectrum Scale (https://en.wikipedia.org/wiki/IBM_Spectrum_Scale) on our nodes.
• We also use them to run Spectrum Protect (https://en.wikipedia.org/wiki/IBM_Tivoli_Storage_Manager) backup servers

[dgerd]

As discussed in the API WG last week I am going to close this issue as we do not plan to allow privileged containers in Knative at this time.

The pattern that we would like to take is to instead enable particular use-cases through more well-formed API semantics. For this particular use case we should explore relaxing volume restrictions to allow volumes types that are known to work and scale well (multi-read-write or multi-reader read-only). See #4417.

@duglin Thanks for the examples of other use-cases here. I don't see the two daemon sets being particularly valuable workloads in Knative given their host-centric nature. I am not sure I understand the Spectrum Protect workload. If this is something you believe can and should be able to be ran on Knative can you open an issue?