Origin: Permission problem with pod's volumeMounts

[AgainstWind]

when i run a container with external volume on the openshift, the application process doesn't run as root
user (it is different with docker) which cause the problem: application process has no permission to create file in the volumeMounts.
i have tried to modify the scc configuration, but has nouse.

[php-coder]

In order to help you, we need more information:

when i run a container with external volume on the openshift

• How do you run a container? Could you provide a manifest?
• Which exactly external volume? Do you mean hostPath?

the application process doesn't run as root user (it is different with docker)

Yes, it's on purpose to make it more secure by default.

which cause the problem: application process has no permission to create file in the volumeMounts.

What are exactly application requirements? Does it really need to be run under root?

i have tried to modify the scc configuration, but has nouse.

How did you try? What commands have you used?

Have you read SCC docs https://docs.openshift.org/latest/admin_guide/manage_scc.html? It would be great if you will help us to improve this doc by providing a feedback (what is unclear? what are we missed?)

[AgainstWind]

Sorry to reply late， i want to create a mysql instance based on openshift mariadb(persistent) template, just do it in the web console, and the deployment yaml file is:

apiVersion: v1
kind: DeploymentConfig
metadata:
  creationTimestamp: '2017-08-18T04:01:32Z'
  generation: 31
  labels:
    app: mysql-persistent
    template: mysql-persistent-template
  name: mysql3
  namespace: mariadb
  resourceVersion: '661118'
  selfLink: /oapi/v1/namespaces/mariadb/deploymentconfigs/mysql3
  uid: eb7bbfa9-83c9-11e7-8bda-0cda411d0456
spec:
  replicas: 1
  selector:
    name: mysql3
  strategy:
    activeDeadlineSeconds: 21600
    recreateParams:
      timeoutSeconds: 600
    resources: {}
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: mysql3
    spec:
      containers:
        - env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: database-root-password
                  name: mysql3
          image: >-
            centos/mysql-57-centos7@sha256:ac910747810d350bac61ebfd1aa54d75626e194a4c0549edf4ca59d49c35b910
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3306
            timeoutSeconds: 1
          name: mysql
          ports:
            - containerPort: 3306
              protocol: TCP
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - '-i'
                - '-c'
                - >-
                  MYSQL_PWD="$MYSQL_PASSWORD" mysql -h 127.0.0.1 -u $MYSQL_USER
                  -D $MYSQL_DATABASE -e 'SELECT 1'
            failureThreshold: 3
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              memory: 512Mi
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /var/lib/mysql/data
              name: host-test
      dnsPolicy: ClusterFirst
      nodeSelector:
        mysql-test: 'true'
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - name: mysql3-data
          persistentVolumeClaim:
            claimName: mysql3
        - hostPath:
            path: /home/mnt
          name: host-test
  test: false
  triggers:
    - imageChangeParams:
        automatic: true
        containerNames:
          - mysql
        from:
          kind: ImageStreamTag
          name: 'mysql:5.7'
          namespace: openshift
        lastTriggeredImage: >-
          centos/mysql-57-centos7@sha256:ac910747810d350bac61ebfd1aa54d75626e194a4c0549edf4ca59d49c35b910
      type: ImageChange
    - type: ConfigChange
status:
  availableReplicas: 0
  conditions:
    - lastTransitionTime: '2017-08-18T04:01:32Z'
      lastUpdateTime: '2017-08-18T04:01:32Z'
      message: Deployment config does not have minimum availability.
      status: 'False'
      type: Available
    - lastTransitionTime: '2017-08-20T13:02:22Z'
      lastUpdateTime: '2017-08-20T13:02:22Z'
      message: replication controller "mysql3-20" has failed progressing
      reason: ProgressDeadlineExceeded
      status: 'False'
      type: Progressing
  details:
    causes:
      - type: Manual
    message: manual change
  latestVersion: 20
  observedGeneration: 31
  replicas: 0
  unavailableReplicas: 0
  updatedReplicas: 0

It has two external storage volumes:

      volumes:
        - name: mysql3-data
          persistentVolumeClaim:
            claimName: mysql3
        - hostPath:
            path: /home/mnt
          name: host-test

Practically, i just use one volume, just make a comparison, and the application container error logs are identical.
The mysql3-data pvc and pv configuration is:

[root@master1 nfstest]# oc describe pv mysql3 
Name:       mysql3
Labels:     <none>
Annotations:    pv.kubernetes.io/bound-by-controller=yes
StorageClass:   
Status:     Bound
Claim:      mariadb/mysql3
Reclaim Policy: Retain
Access Modes:   RWO
Capacity:   1Gi
Message:    
Source:
    Type:   NFS (an NFS mount that lasts the lifetime of a pod)
    Server: nfs.openshifth3c.com
    Path:   /home/nfs/mysql3
    ReadOnly:   false
Events:     <none>

[root@master1 nfstest]# oc describe pvc mysql3 
Name:       mysql3
Namespace:  mariadb
StorageClass:   
Status:     Bound
Volume:     mysql3
Labels:     app=mysql-persistent
        template=mysql-persistent-template
Annotations:    pv.kubernetes.io/bind-completed=yes
        pv.kubernetes.io/bound-by-controller=yes
Capacity:   1Gi
Access Modes:   RWO
Events:     <none>



md5-0f8b591c0210b7a803822e597ef34fd7



---> 01:34:18     Processing MySQL configuration files ...
--
  | ---> 01:34:18     Initializing database ...
  | ---> 01:34:18     Running mysqld --initialize-insecure ...
  | mysqld: Can't create/write to file '/var/lib/mysql/data/is_writable' (Errcode: 13 - Permission denied)
  | 2017-08-21T01:34:19.658234Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
  | 2017-08-21T01:34:19.699794Z 0 [ERROR] --initialize specified but the data directory exists and is not writable. Aborting.
  | 2017-08-21T01:34:19.699816Z 0 [ERROR] Aborting



md5-0d3f2e8e76bca43d8323069187c6e529



[root@master1 nfstest]# oc get scc -o yaml
apiVersion: v1
items:
- allowHostDirVolumePlugin: false
  allowHostIPC: false
  allowHostNetwork: false
  allowHostPID: false
  allowHostPorts: false
  allowPrivilegedContainer: false
  allowedCapabilities: []
  apiVersion: v1
  defaultAddCapabilities: []
  fsGroup:
    type: RunAsAny
  groups:
  - system:cluster-admins
  - system:authenticated
  kind: SecurityContextConstraints
  metadata:
    annotations:
      kubernetes.io/description: anyuid provides all features of the restricted SCC
        but allows users to run with any UID and any GID.
    creationTimestamp: 2017-08-14T12:28:36Z
    name: anyuid
    namespace: ""
    resourceVersion: "553469"
    selfLink: /api/v1/securitycontextconstraints/anyuid
    uid: 184bf726-80ec-11e7-ab26-0cda411d0456
  priority: 10
  readOnlyRootFilesystem: false
  requiredDropCapabilities:
  - MKNOD
  - SYS_CHROOT
  runAsUser:
    type: RunAsAny
  seLinuxContext:
    type: MustRunAs
  supplementalGroups:
    type: RunAsAny
  users:
  - system:serviceaccount:openshift-infra:default
  - system:serviceaccount:openshift-infra:cassandra
  - system:serviceaccount:openshift-infra:hawkular
  - system:serviceaccount:openshift-infra:heapster
  - system:serviceaccount:openshift-infra:deployer
  - system:serviceaccount:mariadb:default
  - system:serviceaccount:dev:admin
  - system:serviceaccount:mariadb:deployer
  - system:serviceaccount:mariadb:builder
  - system:serviceaccount:mariadb:mysql3
  - system:serviceaccount:mysql3:deployer
  - system:serviceaccount:mysql3:default
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret
- allowHostDirVolumePlugin: true
  allowHostIPC: true
  allowHostNetwork: true
  allowHostPID: true
  allowHostPorts: true
  allowPrivilegedContainer: false
  allowedCapabilities: []
  apiVersion: v1
  defaultAddCapabilities: []
  fsGroup:
    type: MustRunAs
  kind: SecurityContextConstraints
  metadata:
    annotations:
      kubernetes.io/description: 'hostaccess allows access to all host namespaces
        but still requires pods to be run with a UID and SELinux context that are
        allocated to the namespace. WARNING: this SCC allows host access to namespaces,
        file systems, and PIDS.  It should only be used by trusted pods.  Grant with
        caution.'
    creationTimestamp: 2017-08-14T12:28:36Z
    name: hostaccess
    namespace: ""
    resourceVersion: "198"
    selfLink: /api/v1/securitycontextconstraints/hostaccess
    uid: 184b72f6-80ec-11e7-ab26-0cda411d0456
  priority: null
  readOnlyRootFilesystem: false
  requiredDropCapabilities: []
  runAsUser:
    type: MustRunAsRange
  seLinuxContext:
    type: MustRunAs
  supplementalGroups:
    type: RunAsAny
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - hostPath
  - persistentVolumeClaim
  - projected
  - secret
- allowHostDirVolumePlugin: true
  allowHostIPC: false
  allowHostNetwork: false
  allowHostPID: false
  allowHostPorts: false
  allowPrivilegedContainer: false
  allowedCapabilities: []
  apiVersion: v1
  defaultAddCapabilities: []
  fsGroup:
    type: RunAsAny
  kind: SecurityContextConstraints
  metadata:
    annotations:
      kubernetes.io/description: 'hostmount-anyuid provides all the features of the
        restricted SCC but allows host mounts and any UID by a pod.  This is primarily
        used by the persistent volume recycler. WARNING: this SCC allows host file
        system access as any UID, including UID 0.  Grant with caution.'
    creationTimestamp: 2017-08-14T12:28:36Z
    name: hostmount-anyuid
    namespace: ""
    resourceVersion: "296410"
    selfLink: /api/v1/securitycontextconstraints/hostmount-anyuid
    uid: 184b321b-80ec-11e7-ab26-0cda411d0456
  priority: null
  readOnlyRootFilesystem: false
  requiredDropCapabilities: []
  runAsUser:
    type: RunAsAny
  seLinuxContext:
    type: MustRunAs
  supplementalGroups:
    type: RunAsAny
  users:
  - system:serviceaccount:openshift-infra:pv-recycler-controller
  - system:serviceaccount:openshift-infra:default
  - system:serviceaccount:openshift-infra:cassandra
  - system:serviceaccount:openshift-infra:hawkular
  - system:serviceaccount:openshift-infra:heapster
  - system:serviceaccount:openshift-infra:deployer
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - hostPath
  - nfs
  - persistentVolumeClaim
  - projected
  - secret
- allowHostDirVolumePlugin: false
  allowHostIPC: false
  allowHostNetwork: true
  allowHostPID: false
  allowHostPorts: true
  allowPrivilegedContainer: false
  allowedCapabilities: []
  apiVersion: v1
  defaultAddCapabilities: []
  fsGroup:
    type: MustRunAs
  kind: SecurityContextConstraints
  metadata:
    annotations:
      kubernetes.io/description: hostnetwork allows using host networking and host
        ports but still requires pods to be run with a UID and SELinux context that
        are allocated to the namespace.
    creationTimestamp: 2017-08-14T12:28:36Z
    name: hostnetwork
    namespace: ""
    resourceVersion: "17338"
    selfLink: /api/v1/securitycontextconstraints/hostnetwork
    uid: 184c34c7-80ec-11e7-ab26-0cda411d0456
  priority: null
  readOnlyRootFilesystem: false
  requiredDropCapabilities:
  - KILL
  - MKNOD
  - SYS_CHROOT
  - SETUID
  - SETGID
  runAsUser:
    type: MustRunAsRange
  seLinuxContext:
    type: MustRunAs
  supplementalGroups:
    type: MustRunAs
  users:
  - system:serviceaccount:default:router
  - system:serviceaccount:default:registry
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret
- allowHostDirVolumePlugin: false
  allowHostIPC: false
  allowHostNetwork: false
  allowHostPID: false
  allowHostPorts: false
  allowPrivilegedContainer: false
  allowedCapabilities: []
  apiVersion: v1
  defaultAddCapabilities: []
  fsGroup:
    type: RunAsAny
  kind: SecurityContextConstraints
  metadata:
    annotations:
      kubernetes.io/description: nonroot provides all features of the restricted SCC
        but allows users to run with any non-root UID.  The user must specify the
        UID or it must be specified on the by the manifest of the container runtime.
    creationTimestamp: 2017-08-14T12:28:36Z
    name: nonroot
    namespace: ""
    resourceVersion: "196"
    selfLink: /api/v1/securitycontextconstraints/nonroot
    uid: 184af209-80ec-11e7-ab26-0cda411d0456
  priority: null
  readOnlyRootFilesystem: false
  requiredDropCapabilities: []
  runAsUser:
    type: MustRunAsNonRoot
  seLinuxContext:
    type: MustRunAs
  supplementalGroups:
    type: RunAsAny
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret
- allowHostDirVolumePlugin: true
  allowHostIPC: true
  allowHostNetwork: true
  allowHostPID: true
  allowHostPorts: true
  allowPrivilegedContainer: true
  allowedCapabilities:
  - '*'
  apiVersion: v1
  defaultAddCapabilities: []
  fsGroup:
    type: RunAsAny
  groups:
  - system:cluster-admins
  - system:nodes
  - system:serviceaccount:mariadb
  kind: SecurityContextConstraints
  metadata:
    annotations:
      kubernetes.io/description: 'privileged allows access to all privileged and host
        features and the ability to run as any user, any group, any fsGroup, and with
        any SELinux context.  WARNING: this is the most relaxed SCC and should be
        used only for cluster administration. Grant with caution.'
    creationTimestamp: 2017-08-14T12:28:36Z
    name: privileged
    namespace: ""
    resourceVersion: "553176"
    selfLink: /api/v1/securitycontextconstraints/privileged
    uid: 184aabf2-80ec-11e7-ab26-0cda411d0456
  priority: null
  readOnlyRootFilesystem: false
  requiredDropCapabilities: []
  runAsUser:
    type: RunAsAny
  seLinuxContext:
    type: RunAsAny
  seccompProfiles:
  - '*'
  supplementalGroups:
    type: RunAsAny
  users:
  - system:serviceaccount:openshift-infra:build-controller
  - system:serviceaccount:management-infra:management-admin
  - system:serviceaccount:management-infra:inspector-admin
  - system:serviceaccount:logging:aggregated-logging-fluentd
  - system:serviceaccount:openshift-infra:default
  - system:serviceaccount:openshift-infra:cassandra
  - system:serviceaccount:openshift-infra:hawkular
  - system:serviceaccount:openshift-infra:heapster
  - system:serviceaccount:openshift-infra:deployer
  - admin
  - cicd
  - system:serviceaccount:dev:default
  - system:serviceaccount:dev:builder
  - system:serviceaccount:dev:deployer
  - test
  - system:serviceaccount:mariadb:default
  - system:serviceaccount:stage:default
  - system:serviceaccount:stage:builder
  - system:serviceaccount:stage:deployer
  - system:serviceaccount:cicd:deployer
  - system:serviceaccount:cicd:default
  - system:serviceaccount:cicd:jenkins
  - system:serviceaccount:cicd:builder
  - tets
  volumes:
  - '*'
- allowHostDirVolumePlugin: false
  allowHostIPC: false
  allowHostNetwork: false
  allowHostPID: false
  allowHostPorts: false
  allowPrivilegedContainer: false
  allowedCapabilities: []
  apiVersion: v1
  defaultAddCapabilities: []
  fsGroup:
    type: RunAsAny
  groups:
  - system:authenticated
  kind: SecurityContextConstraints
  metadata:
    annotations:
      kubernetes.io/description: restricted denies access to all host features and
        requires pods to be run with a UID, and SELinux context that are allocated
        to the namespace.  This is the most restrictive SCC and it is used by default
        for authenticated users.
    creationTimestamp: 2017-08-14T12:28:36Z
    name: restricted
    namespace: ""
    resourceVersion: "534201"
    selfLink: /api/v1/securitycontextconstraints/restricted
    uid: 184bb7fc-80ec-11e7-ab26-0cda411d0456
  priority: null
  readOnlyRootFilesystem: false
  requiredDropCapabilities:
  - KILL
  - MKNOD
  - SYS_CHROOT
  - SETUID
  - SETGID
  runAsUser:
    type: RunAsAny
  seLinuxContext:
    type: RunAsAny
  supplementalGroups:
    type: RunAsAny
  volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - persistentVolumeClaim
  - projected
  - secret
kind: List
metadata: {}
resourceVersion: ""
selfLink: ""

[php-coder]

In this case I'd try to grant access to the privileged SCC to the system:serviceaccount:mariadb:deployer service account: oc adm policy add-scc-to-group privileged system:serviceaccount:mariadb:deployer.

Let me know if it works.

[php-coder]

BTW are you sure that you really need to run the database under root user and with privileged mode? This isn't secure and it should be avoided when possible. If you only need it to have an access to the files on the disk, then there should be alternative ways to achieve that.

[lilianmoraru]

@php-coder I just try to use the default redhat-openjdk18-openshi deployment and it fails instantly(permission denied):

Starting the Java application using /opt/run-java/run-java.sh ...
/opt/run-java/run-java.sh: line 44: cd: /home/jboss: Permission denied

[AgainstWind]

@php-coder @lilianmoraru sorry, the problem has resolved, it is not the scc's problem, i have error on nfs server directory permission assignment.
The error nfs permission configuration:
# chown -R nfsnobody:nfsnobody /home/nfs
and after modify to:
# chmod 766 -R /home/nfs
the problem is resolved.

[php-coder]

@AgainstWind Ok, thanks for letting me know. I'm closing this issue then.

@lilianmoraru Could you fill a separate issue with the steps to reproduce it?