Serving: /var/log mount breaks some containers by default

[greghaynes]

Expected Behavior

I expect the following service definition to result in a working nginx deployment

apiVersion: serving.knative.dev/v1alpha1 # Current version of Knative
kind: Service
metadata:
  name: hello-nginx # The name of the app
  namespace: default # The namespace the app will use
spec:
  runLatest:
    configuration:
      revisionTemplate:
        spec:
          container:
            image: nginx:stable
            env:
              NGINX_PORT: 80

Actual Behavior

The user-container pod fails to start with the following error:

nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (2: No such file or directory)
2018/10/03 07:42:46 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (2: No such file or directory)

Steps to Reproduce the Problem

1. Define the above service
2. Observe the failure in the user-container pod

Additional Info

This is caused due to our mounting of /var/log for fluentd overriding the image /var/log and nginx expecting the /var/log/nginx dir to be present from the image build. Although this can be worked around by explicitly setting the error log path to be in /var/log this is not a great experience for new users.

[simplytunde]

Yes, its breaking my test. now I have to play around customizing my nginx

[mattmoor]

This isn't really the control plane so much as the runtime contract, but still a spec thing.

I suspect that this is WAI, but I'll defer to Evan to make the call, since he drove that spec.

/assign @evankanderson

[evankanderson]

(sorry for the delay)

Unfortunately, I think this is the best we can do right now with Kubernetes.

Ideally, we could bind-mount and share an emptyDir for /var/log and have it overlay with the existing docker image, but it doesn't appear that there are any kubernetes-level controls to support this type of overlay. This might be reasonably requested in the fluentd-kubernetes-plugin upstream repo, though it will probably require kubernetes changes.

[ReggieCarey]

Greetings, I'm new to Knative but I'm running into this issue as well. If Knative is attempting to enforce some structure to a container in order to run, then it has stepped beyond its area of concern. Why should my service be limited to writing logs to /var/log and why is this not spelled out somewhere in the docs. If this has something to do with fluentd as suggested in the comments, then how do I disable it?

Is there any solution to this problem? I'm stopped cold.

[evankanderson]

I think #6147 suggests that we should stop doing this (mounting an emptyDir to /var/log). This was based on a reading of https://github.com/knative/serving/blob/master/docs/runtime-contract.md#default-filesystems which is not quite what was intended. (i.e. the difference between mkdir -p -m 1777 /var/log and replacing /var/log with an empty directory.

[evankanderson]

Do you want to go add a +1 vote to #6147 ?