Trident: v20.07.01 ontap-san unable to write

[Numblesix]

Describe the bug
I had to install a Cluster from scratch today while deploying it with trident v20.07.1 i always (4 times) ended up with volumes (created from ontap-san driver) which were not writeable from within the pod. As soon as i deleted the Cluster and reinstalled with v20.07.0 it worked again, as i use a playbook to install my cluster i can guarantee that there hasnt been any change inbetween those two versions.

Environment
Dev

• Trident version: 20.07.01
• Trident installation flags used: operator install with autosupport disabled
• Container runtime: CRIO
• Kubernetes version: v1.18.3+b0068a8
• Kubernetes orchestrator: OpenShift 4.5.11
• Kubernetes enabled feature gates: Default
• OS: Rhel CoreOS Nodes on 4.5.11
• NetApp backend types: ontap-san

To Reproduce
Create Cluster from Scratch with v20.07.1, create a pvc with backend type ontap-san and try to write to the disk

Expected behavior
Write should be possible

Additional context
Please note that this only occured on ontap-san my other pvc's with ontap-nas were working as expected. I also noted that writing on the Host directly as root to the disk was working, only when inside the pod (which doesnt run as root) it didnt work.
If you need more Infos let me know i will install another Cluster next week so i could gather Logs for you :)

[gnarl]

Hi @Numblesix,

We test the ontap_san storage driver in a fresh OCP 4.5 cluster in our CI. Obtaining logs using tridentctl logs -a for the issue you are experiencing would be very helpful. We'll also want to open a NetApp Support case. Let me know if you need any assistance.

[sitzm]

Hello,

I have the same problem, with v20.07.1 the pod cannot write with random uid:
(openshift 4.5.11 and 4.5.13)

only root is allowed to write to volume

$ls -al /iscsi-20-07-1/
total 20
drwxr-xr-x. 3 root root  4096 Oct  6 10:45 .
drwxr-xr-x. 1 root root   117 Oct  6 10:50 ..
drwx------. 2 root root 16384 Oct  6 10:45 lost+found

after a downgrade to v20.07 and create a new volume:

ls -al /iscsi-20-07/ 
total 20
drwxrwsr-x. 3 root 1000910000  4096 Oct  6 10:50 .
drwxr-xr-x. 1 root root         117 Oct  6 10:50 ..
drwxrws---. 2 root 1000910000 16384 Oct  6 10:50 lost+found

[Numblesix]

@sitzm that was exactly my issue could you share logs via tridentctl logs -a

[gnarl]

@sitzm and @Numblesix we haven't been able to reproduce this issue in our OCP CI tests. We suspect that you may be running into an issue with the new version of the CSI provisioner sidecar based on your configuration. Please provide examples of the Storage Class, Backend config (redact personal information), PVC, and Pod definitions.

[sitzm]

@gnarl here is the configuration:
backend:

{
    "version": 1,
    "storageDriverName": "ontap-san",
    "backendName": "OpenShift RF",
    "managementLIF": "10.x.x.135",
    "dataLIF": "10.x.x.128",
    "svm": "svm1234",
    "igroupName": "eGov_client4",
    "username": "username",
    "password": "password",
    "storagePrefix": "nonprod"
}

Storageclass

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: trident-iscsi
parameters:
  backendType: ontap-san
provisioner: netapp.io/trident
reclaimPolicy: Delete
volumeBindingMode: Immediate

pvc

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: csi.trident.netapp.io
  selfLink: /api/v1/namespaces/test-sim/persistentvolumeclaims/test-iscsi-4-5-13
  resourceVersion: '25896884'
  name: test-iscsi-4-5-13
  uid: d8ef5807-8779-46b3-8740-9864017acb87
  creationTimestamp: '2020-10-06T10:32:59Z'
  namespace: test-sim
  finalizers:
    - kubernetes.io/pvc-protection
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  volumeName: pvc-d8ef5807-8779-46b3-8740-9864017acb87
  storageClassName: trident-iscsi
  volumeMode: Filesystem
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 2Gi

pod

kind: Pod
apiVersion: v1
metadata:
  generateName: nginx-7bf97cbfb6-
  annotations:

    openshift.io/generated-by: OpenShiftWebConsole
    openshift.io/scc: restricted
  selfLink: /api/v1/namespaces/test-sim/pods/nginx-7bf97cbfb6-kxhw2
  resourceVersion: '26854815'
  name: nginx-7bf97cbfb6-kxhw2
  uid: 8403ff59-8f83-454b-a9ce-98a0c63dc1cd
  namespace: test-sim
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: nginx-7bf97cbfb6
      uid: c3a9aa63-011f-4cfb-8f97-e2cb23f7c019
      controller: true
      blockOwnerDeletion: true
  labels:
    app: nginx
    deploymentconfig: nginx
    pod-template-hash: 7bf97cbfb6
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ''
  restartPolicy: Always
  serviceAccountName: default
  imagePullSecrets:
    - name: default-dockercfg-f2fss
  priority: 0
  schedulerName: default-scheduler
  enableServiceLinks: true
  terminationGracePeriodSeconds: 30
  nodeName: node1
  securityContext:
    seLinuxOptions:
      level: 's0:c30,c20'
    fsGroup: 1000910000
  containers:
    - resources: {}
      terminationMessagePath: /dev/termination-log
      name: nginx
      securityContext:
        capabilities:
          drop:
            - KILL
            - MKNOD
            - SETGID
            - SETUID
        runAsUser: 1000910000
      ports:
        - containerPort: 8080
          protocol: TCP
        - containerPort: 8443
          protocol: TCP
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - name: test-iscsi-4-5-13
          mountPath: /test
        - name: default-token-f6lp5
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePolicy: File
      image: >-
        bitnami/nginx@sha256:ff9da47b2c33fb253550f1ad50350c1e65edc9c18770435ff752a3f017fb7ab6
  serviceAccount: default
  volumes:
    - name: test-iscsi-4-5-13
      persistentVolumeClaim:
        claimName: test-iscsi-4-5-13
    - name: default-token-f6lp5
      secret:
        secretName: default-token-f6lp5
        defaultMode: 420
  dnsPolicy: ClusterFirst

[tcarland]

I can also confirm this issue with 20.07.1 and ontap-nas (running k8s 1.17.9). Specifically, I noticed that post the PVC mount that the securityContext: fsgroup was not being applied as usual. kubelet logs would normally register a step for "Setting volume ownership" after completion of the WaitForAttach and MountDevice steps.

I also verified with a test pod that the pvc only had the default 0755, root:root permissions where in 20.07.0 I have the set ownership step logged and 2775 root:mygroup on the volume as defined by the securityContext. Downgrading to 20.07.0, removing the underlying volume (must do this) and recreating fixed the issue and perms were correct again. I reproduced the issue in a separate k8s environment as well with the same fix.

[tksm]

We got the same problem with Trident v20.07.1 and Kubernetes v1.18.6.

It seems that the issue was due to the absence of fsType=ext4 in PersistentVolume created by v20.07.1. When we set fsType=ext4 parameter explicitly in StorageClass, we do not see this issue.

I think updating external-provisioner to v2.0.1 in https://github.com/NetApp/trident/commit/b33021d64da7588d65b7f38cda7cd01873850e14 introduced this issue. I guess default-fstype argument should have been added with this update.

https://github.com/kubernetes-csi/external-provisioner/blob/release-2.0/CHANGELOG/CHANGELOG-2.0.md#changelog-since-v160
The fstype on provisioned PVs no longer defaults to "ext4". A defaultFStype arg is added to the provisioner. Admins can also specify this fstype via storage class parameter. If fstype is set in storage class parameter, it will be used. The sidecar arg is only checked if fstype is not set in the SC param. (#400, @humblec)

[Numblesix]

I can also share some stuff now :)

StorageClass

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-csi-block
mountOptions:
- discard
parameters:
  backendType: ontap-san
provisioner: csi.trident.netapp.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

Backend

{
    "version": 1,
    "storageDriverName": "ontap-san",
    "backendName": "ontap_san",
    "managementLIF": "1.2.3.4",
    "svm": "netapp-server",
    "username": "trident",
    "password": "adsfasdfasdf",
    "storagePrefix": "osr2_iscsi",
    "igroupName": "osr2_iqn",
    "useCHAP": true,
    "chapInitiatorSecret": "adsfadsf",
    "chapTargetInitiatorSecret": "adsfasdf",
    "chapTargetUsername": "adfadf",
    "chapUsername": "asdfadsf",
    "defaults": {
        "spaceReserve": "volume",
        "spaceAllocation": "false",
        "snapshotPolicy": "none",
    }
}

PVC

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-mysql-block
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: netapp-csi-block

Pod

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/network-status: |-
      [{
          "name": "openshift-sdn",
          "interface": "eth0",
          "ips": [
              "10.131.0.5"
          ],
          "default": true,
          "dns": {}
      }]
    k8s.v1.cni.cncf.io/networks-status: |-
      [{
          "name": "openshift-sdn",
          "interface": "eth0",
          "ips": [
              "10.131.0.5"
          ],
          "default": true,
          "dns": {}
      }]
    openshift.io/scc: restricted
  generateName: wordpress-mysql-77cb4466fc-
  labels:
    app: wordpress
    pod-template-hash: 77cb4466fc
    tier: mysql
spec:
  containers:
  - env:
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: mysql-pass
    image: mysql:5.6
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      periodSeconds: 10
      successThreshold: 1
      tcpSocket:
        port: 3306
      timeoutSeconds: 1
    name: mysql
    ports:
    - containerPort: 3306
      name: mysql
      protocol: TCP
    resources: {}
    securityContext:
      capabilities:
        drop:
        - KILL
        - MKNOD
        - SETGID
        - SETUID
      runAsUser: 1000660000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/lib/mysql
      name: mysql-persistent-storage
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-9v4rc
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: default-dockercfg-h8xcc
  nodeName: node.domain.de
  nodeSelector:
    node-role.kubernetes.io/app: ""
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1000660000
    seLinuxOptions:
      level: s0:c26,c5
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: mysql-persistent-storage
    persistentVolumeClaim:
      claimName: pvc-mysql-block
  - name: default-token-9v4rc
    secret:
      defaultMode: 420
      secretName: default-token-9v4rc
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-10-09T07:46:12Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-10-09T07:46:12Z"
    message: 'containers with unready status: [mysql]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-10-09T07:46:12Z"
    message: 'containers with unready status: [mysql]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-10-09T07:46:12Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: mysql:5.6
    imageID: ""
    lastState: {}
    name: mysql
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: secret "mysql-pass" not found
        reason: CreateContainerConfigError
  hostIP: 1.2.3.4
  phase: Pending
  podIP: 10.131.0.5
  podIPs:
  - ip: 10.131.0.5
  qosClass: BestEffort
  startTime: "2020-10-09T07:46:12Z"

[gnarl]

Thanks for all of the feedback! It does look like an issue with the CSI external-provisioner sidecar v2.0.1. We are investigating how to fix or workaround the issue.

[balaramesh]

It looks like the problem is here.

Since the CSI external-provisioner no longer enforces a default fsType (and is something that Trident should take care of from now on), there are two ways to fix it in the meantime:

1. Provide a fsType in your StorageClass.
2. Generate Trident's custom YAMLs and edit the trident-deployment.yaml file to include an additional arg for the csi-provisioner like this:

      - name: csi-provisioner
        image: quay.io/k8scsi/csi-provisioner:v2.0.1
        args:
        - "--v=2"
        - "--timeout=600s"
        - "--csi-address=$(ADDRESS)"
        - "--default-fstype=ext4"

Option (1) is the easiest way to fix it for now.

[ysakashita]

@gnarl @balaramesh I believe that the bug are occurred by missed fsType. As you may know, the parameter in StorageClass is immutable value. So we need to delete and re-create it. I think the bug is critical for customer. When we use Trident Operator, we can't chose 2nd workaround in https://github.com/NetApp/trident/issues/454#issuecomment-706301546 . Customer has ask it to us. I hope that NetApp provide Trident v20.07.2.

[gnarl]

This bug exists outside of Trident and additional information can be found in this NetApp KB article.

The fix for this issue is to use a StorageClass that specifies the fsType to be used with SAN volumes. Since StorageClasses cannot be patched, existing StorageClasses must be deleted and recreated as is, along with parameter.fsType defined.

This will not impact any existing volume already created and does not carry any dependency with pre-provisioned volumes associated with the StorageClass.

[ysakashita]

@gnarl
As @tksm has already commented (https://github.com/NetApp/trident/issues/454#issuecomment-705926951 ) , it is necessary to specify the defaultFStype from external provisioner v2.0.1 .

https://github.com/kubernetes-csi/external-provisioner/blob/release-2.0/CHANGELOG/CHANGELOG-2.0.md#changelog-since-v160

The fstype on provisioned PVs no longer defaults to "ext4". A defaultFStype arg is added to the provisioner. 
Admins can also specify this fstype via storage class parameter. If fstype is set in storage class parameter, it will be used. 
The sidecar arg is only checked if fstype is not set in the SC param. (#400, @humblec)

Unfortunately, NetApp/Trident v20.07.1 doesn't set the defautlFStype.
I think that NetApp/Trident should set it.

So, I have a question.
Is the permanent workaround that sets fsType parameter in a StorageClass ?
Or,
the temporary workaround until the bug is fixed in NetApp/Trident .

Please let me know about the policy of NetApp/Trident for the bug.
Our support policy for the next Trident version depends on whether fsType in StorageClass is required.

[balaramesh]

Hello @ysakashita

Setting the fsType on a StorageClass is a permanent fix. The reason Trident does not set defaultFStype is because it also needs to create NFS volumes. For NFS volumes, the fsType needs to be blank. The fsType that is provided in the StorageClass will override the default value (which is blank)

[ysakashita]

@balaramesh Got it thanks

[gnarl]

Closing this issue as information regarding how to set the fsType is now included in the Trident documentation.