Trident: ontap-san driver does not consider snapshot reserve when creating volume

[wonderland]

Describe the bug
The ontap-san driver ignores the snapshot reserve that is configured for the backend when creating the Ontap volume. The volume size simply equals the PVC size. As the snapshot reserve is unavailable to the active file system, a ontap-san PVC can never be actually filled to its full size when a snap reserve is configured. Applications that actually try to use the full PVC size will experience write failures and/or LUNs going offline, even though there still appears to be free space in the PVC/LUN.

Environment

• Trident version: 21.01.1
• Trident installation flags used: default
• Container runtime: Docker
• Kubernetes version: 1.18
• Kubernetes orchestrator: vanilla k8s
• Kubernetes enabled feature gates:
• OS: RHEL
• NetApp backend types: Ontap SAN/iSCSI

To Reproduce

A simple ontap-san backend, snap reserve set to 50% for illustration purposes, the issue occurs with any value >0.:

{
    "version": 1,  
    "storageDriverName": "ontap-san",  
    "backendName": "ontap-san-backend",  
    "useCHAP": true,  
    "chapInitiatorSecret": "cl9qxWfnUFoinvk",  
    "chapTargetInitiatorSecret": "rwbwtuigONgbw",  
    "chapTargetUsername": "iJF4sgjrnwOwQ",  
    "chapUsername": "uh2a1io325bFFILn",
    "managementLIF": "192.168.0.136",
    "svm": "iscsi_svm",
    "username": "vsadmin",
    "password": "Netapp1!",
    "defaults": {
        "snapshotReserve": "50"
    }
}

A matching storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ontap-san
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-san"
allowVolumeExpansion: true

Provision a PVC with 2GB size and attach it to a simple pod:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: ontap-san-pvc-alpine
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: ontap-san
---
kind: Pod
apiVersion: v1
metadata:
  name: alpine-ontap-san-pod
spec:
  volumes:
    - name: volume1
      persistentVolumeClaim:
       claimName: ontap-san-pvc-alpine
  containers:
    - name: alpine
      image: alpine:3.2
      command:
        - /bin/sh
        - "-c"
        - "sleep 60m"
      volumeMounts:
        - mountPath: "/data"
          name: volume1

Ontap shows the volume at 2GB size (=PVC size) but with the 50% snap reserve, e.g. only 1Gb is actually usable from the volume. LUN is thin provisioned, therefore can be 2GB in size:

cluster1::> vol show -vserver iscsi_svm -volume  trident_pvc_ccb44846_a6a0_40c5_89f2_56cb533fa419 -fields percent-snapshot-space,size
vserver   volume                                           size percent-snapshot-space
--------- ------------------------------------------------ ---- ----------------------
iscsi_svm trident_pvc_ccb44846_a6a0_40c5_89f2_56cb533fa419 2GB  50%

At this point we already know that even though the k8s node and the pod see a 2GB LUN, it is impossible to actually write data beyond the 1GB mark, as the underlying volume only has 1GB of usable capacity. We can verify by exec'ing into the pod and creating a file >1GB (fails), while a 1Gb file works:

[root@rhel3 k8s]# k exec -it alpine-ontap-san-pod -- /bin/sh

/ # dd if=/dev/zero of=/data/file bs=1500M count=1
Killed

/ # dd if=/dev/zero of=/data/file bs=1000M count=1
1+0 records in
1+0 records out
/data # ls -al
total 1024024
drwxr-xr-x    3 root     root          4096 Mar 23 20:01 .
drwxr-xr-x    1 root     root            41 Mar 23 19:54 ..
-rw-r--r--    1 root     root     1048576000 Mar 23 20:02 file
drwx------    2 root     root         16384 Mar 23 19:54 lost+found

Expected behavior
Snapshot reserve is configured in Trident. This absolutely needs to be taken into account when creating the volume.
We have a similar issue with the ontap-nas driver, but there the pod/app at least sees that it runs out of space. With the ontap-san driver the pod still sees free space but cannot write any further data. Ontap reacts with taking LUNs offline when the volume is full, this can lead to all kinds of data inconsistencies, making this a critical bug.

Additional context
See #496 for the ontap-issue with snap reserve

[wonderland]

The logic to address this issue already exists in ontap_common.go/calculateOptimalSizeForFlexvol:

https://github.com/NetApp/trident/blob/e4b47c2d1873c5a0c5adbb8c120e93c446edaa93/storage_drivers/ontap/ontap_common.go#L3008

It is currently used by ontap-san-economy and ontap-nas-economy drivers, but not by the non-economy drivers.

[gnarl]

Hi @wonderland,

I'm moving this from bug to enhancement as it is new functionality for Trident. The difficulties that you are describing are understood but much of what you are describing deals with how ONTAP works. Your request is for Trident to work around an ONTAP issue by increasing the size of the FlexVol beyond what was actually requested in the PVC.

The economy drivers are different from the other storage drivers in that the FlexVol size is adjusted when new PVs are added or removed. For the other storage drivers the customer still has the ability to expand the size of the volume to avoid it going off line. In addition to Prometheus metrics available through Trident customers can also use Cloud Insights to monitor their volume capacity and create rules to automatically resize volumes. This is the suggested work around until we can implement this request.

[wonderland]

Thanks for looking into this @gnarl

I disagree with the enhancement vs. bug though. I hope the following does no come across too harsh, but this breaks the fundamental value of trident, e.g. dynamically providing a certain storage capacity. If a user requests a PVC of a certain size he should be able to actually write that amount of data to the PVC. If he can only write a fraction of what was requested and then everything break apart I clearly consider that a bug. This is like the cheap fake USB thumb drives that are sold as 32GB (and report as such to the host) but actually only have 4Gb of flash cells. Trident provisions a 10 GB PVC and reports the available space as such, but only supports writing 5Gb to it. Actually, from a user perspective, the amount of data that can be written to any given PVC becomes a random value that is smaller than the requested PVC size.

This is especially bad with ontap-san as the user has no way to figure out what is going on, he sees more than enough free space and any space constraints on the underlying volume are invisible.

The PVC size maps to the LUN size. The volume holding the LUN is just a unit of management for the storage backend, invisible from the k8s side. So Trident has to ensure that the volume size is appropriate to not immediately break anything. This is not an Ontap issue, it is Trident that is not using Ontap correctly.

Monitoring volume (=PVC) capacity does not address this issue as the capacity constraint is not at the LUN but at the Ontap-volume level - which is not reported to Prometheus. The usage reported in Prometheus might be at just 10% but the Ontap volume is full and we run into this issue. Without getting even close to any warning threshold in Prometheus.

In the current form, the ontap-san driver has to be considered broken as soon as a snapReserve is configured (wich is a valid and supported configuration item in a Trident backend) or a snapshot is created (which is a valid and supported Trident activity).

[mihailanga]

Hi @gnarl,
As a NetApp customer we've been directly impacted by this design flaw a few days ago. I say flaw and not lack of functionality because, just as wonderland detailed above, we consider this a bug.
From a storage administrator's perspective I've always provisioned my volumes to be just a bit bigger (usually around 5-10%, depending on how big they are) than the LUNs contained in them. This has been a best practice in SAN storage management on NetApp systems for years throughout the community and it's targeted specifically at avoiding the situation we encountered with using trident.
For us it was a surprise when we noticed that LUNs and containing volumes are provisioned with the same values.

Can you please confirm that there is active effort being put into creating a fix for this as soon as possible?
In addition can you please provide us with an estimate on when the fix is expected to be released?

Thanks in advance!

[prescott-core]

Hi guys,

I can confirm this behavior through our own experiences. Are there any news when this could be fixed? Trident looks promising but this bug would get us into much trouble.

[gnarl]

We are looking into addressing this issue in the Trident 21.07 release. Stay tuned for updates!

[gnarl]

This issue is fixed with commit e2c03fe which will be included in the Trident 21.07 release.