Trident: Creating volumesnapshot causes error status.restoreSize in body should be greater than or equal to 0

Created on 20 Sep 2021 · 16Comments · Source: NetApp/trident

Describe the bug

I am setting up snapshots on a openshift cluster so I have created a snapshotclass

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: nap-snapshots
driver: csi.trident.netapp.io
deletionPolicy: Delete

And a volumesnapshot:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: nap-nfs-pre-dr03-volym-pvc-snapshot-20210920-2
spec:
  volumeSnapshotClassName: nap-snapshots
  source:
    persistentVolumeClaimName: nap-nfs-pre-dr03-volym-pvc

However the volumesnapshot isnt created:

$ oc get volumesnapshots
NAME                                             READYTOUSE   SOURCEPVC                    SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                                    CREATIONTIME   AGE
nap-nfs-pre-dr03-volym-pvc-snapshot-20210920-2   false        nap-nfs-pre-dr03-volym-pvc                                         nap-snapshots   snapcontent-58753024-b897-481d-a493-7672eab98d95                  30s

$ oc describe volumesnapshot nap-nfs-pre-dr03-volym-pvc-snapshot-20210920-2
Name:         nap-nfs-pre-dr03-volym-pvc-snapshot-20210920-2
Namespace:    sfit
Labels:       <none>
Annotations:  <none>
API Version:  snapshot.storage.k8s.io/v1
Kind:         VolumeSnapshot
Metadata:
  Creation Timestamp:  2021-09-20T08:29:19Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  Generation:  1
    Manager:         snapshot-controller
    Operation:       Update
    Time:            2021-09-20T08:29:20Z
  Resource Version:  119764628
  UID:               58753024-b897-481d-a493-7672eab98d95
Spec:
  Source:
    Persistent Volume Claim Name:  nap-nfs-pre-dr03-volym-pvc
  Volume Snapshot Class Name:      nap-snapshots
Status:
  Bound Volume Snapshot Content Name:  snapcontent-58753024-b897-481d-a493-7672eab98d95
  Error:
    Message:     Failed to check and update snapshot content: error updating status for volume snapshot content snapcontent-58753024-b897-481d-a493-7672eab98d95: snapshot controller failed to update snapcontent-58753024-b897-481d-a493-7672eab98d95 on API server: VolumeSnapshotContent.snapshot.storage.k8s.io "snapcontent-58753024-b897-481d-a493-7672eab98d95" is invalid: status.restoreSize: Invalid value: -1744896: status.restoreSize in body should be greater than or equal to 0
    Time:        2021-09-20T08:29:20Z
  Ready To Use:  false
Events:
  Type    Reason            Age   From                 Message
  ----    ------            ----  ----                 -------
  Normal  CreatingSnapshot  56s   snapshot-controller  Waiting for a snapshot sfit/nap-nfs-pre-dr03-volym-pvc-snapshot-20210920-2 to be created by the CSI driver.

tridentctl logs gives me:

time="2021-09-20T08:29:19Z" level=warning msg="Snapshot not found." requestID=53771262-28eb-43c3-8cc1-67c1566103de requestSource=CSI snapshotName=snapshot-58753024-b897-481d-a493-7672eab98d95 volumeName=trident_pvc_902f8810_1daf_40dd_89b5_4ae5ebb77a85

The PVC is created with a storageclass defined as:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2021-05-31T08:35:55Z"
  name: nap-nfs-pre-dr03-volym
  resourceVersion: "42649804"
  uid: 0b365406-7fd9-47d2-9a4f-978b9d5ecf67
parameters:
  backendType: ontap-nas
  clones: "true"
  media: hybrid
  provisioningType: thin
  snapshots: "true"
  storagePools: my-backend
provisioner: csi.trident.netapp.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

(my-backend storage driver is "ontap-nas")

Environment
Provide accurate information about the environment to help us reproduce the issue.

Trident version: 21.07.1
Kubernetes version: v1.21.1+9807387
Kubernetes orchestrator: Openshift 4.8.10
NetApp backend types: ONTAP

To Reproduce

Create volumesnapshotclass
Create volumesnapshot

Expected behavior
Snapshot should be created

bug tracked

Source

gorantornqvist

All 16 comments

@gorantornqvist can you confirm that this is with Trident v21.07.1? The error you have shared (status.restoreSize: Invalid value: -1744896: status.restoreSize in body should be greater than or equal to 0) indicates that the restoreSize for the snapshot is inaccurate. Can you also share the output of kubectl get pvc <source-pvc>?

balaramesh on 22 Sep 2021

Hi,

$ tridentctl -n trident version
+----------------+----------------+
| SERVER VERSION | CLIENT VERSION |
+----------------+----------------+
| 21.07.1        | 21.07.1        |
+----------------+----------------+

$ kubectl get pvc nap-nfs-pre-dr03-volym-pvc -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: csi.trident.netapp.io
  creationTimestamp: "2021-06-21T11:28:32Z"
  finalizers:
  - kubernetes.io/pvc-protection
  - snapshot.storage.kubernetes.io/pvc-as-source-protection
  name: nap-nfs-pre-dr03-volym-pvc
  namespace: my-ns
  resourceVersion: "119764580"
  uid: 902f8810-1daf-40dd-89b5-4ae5ebb77a85
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 300Mi
  storageClassName: nap-nfs-pre-dr03-volym
  volumeMode: Filesystem
  volumeName: pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 300Mi
  phase: Bound
---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: csi.trident.netapp.io
  creationTimestamp: "2021-06-21T11:28:35Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/csi-trident-netapp-io
  name: pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85
  resourceVersion: "14697591"
  uid: 7f786085-35b2-4e43-94ae-74d2914d598d
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 300Mi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: nap-nfs-pre-dr03-volym-pvc
    namespace: my-ns
    resourceVersion: "14697526"
    uid: 902f8810-1daf-40dd-89b5-4ae5ebb77a85
  csi:
    driver: csi.trident.netapp.io
    volumeAttributes:
      backendUUID: 68564aae-6f05-43ca-8b28-0a7bf9f07645
      internalName: trident_pvc_902f8810_1daf_40dd_89b5_4ae5ebb77a85
      name: pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85
      protocol: file
      storage.kubernetes.io/csiProvisionerIdentity: 1623841646672-8081-csi.trident.netapp.io
    volumeHandle: pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85
  persistentVolumeReclaimPolicy: Delete
  storageClassName: nap-nfs-pre-dr03-volym
  volumeMode: Filesystem
status:
  phase: Bound

gorantornqvist on 23 Sep 2021

Many thanks, Göran. May I ask for the same output for the volumesnapshot (or the yaml file that was used to create it)? And, if possible, the Trident logs from the time when the snapshot was created?

Thank you

Hendrik

From: Göran Törnqvist @.>
Date: Thursday, 23. September 2021 at 08:59
To: NetApp/trident *@.>
Cc: Land, Hendrik @.>, Manual @.*>
Subject: Re: [NetApp/trident] Creating volumesnapshot causes error "Snapshot not found" in trident log (#645)
NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hi,

$ tridentctl -n trident version

+----------------+----------------+

| SERVER VERSION | CLIENT VERSION |

+----------------+----------------+

| 21.07.1 | 21.07.1 |

+----------------+----------------+

$ kubectl get pvc nap-nfs-pre-dr03-volym-pvc -o yaml

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

annotations:

pv.kubernetes.io/bind-completed: "yes"

pv.kubernetes.io/bound-by-controller: "yes"

volume.beta.kubernetes.io/storage-provisioner: csi.trident.netapp.io

creationTimestamp: "2021-06-21T11:28:32Z"

finalizers:

kubernetes.io/pvc-protection
snapshot.storage.kubernetes.io/pvc-as-source-protection

name: nap-nfs-pre-dr03-volym-pvc

namespace: my-ns

resourceVersion: "119764580"

uid: 902f8810-1daf-40dd-89b5-4ae5ebb77a85

spec:

accessModes:

ReadWriteOnce

resources:

requests:

storage: 300Mi

storageClassName: nap-nfs-pre-dr03-volym

volumeMode: Filesystem

volumeName: pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85

status:

accessModes:

ReadWriteOnce

capacity:

storage: 300Mi

phase: Bound

apiVersion: v1

kind: PersistentVolume

metadata:

annotations:

pv.kubernetes.io/provisioned-by: csi.trident.netapp.io

creationTimestamp: "2021-06-21T11:28:35Z"

finalizers:

kubernetes.io/pv-protection
external-attacher/csi-trident-netapp-io

name: pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85

resourceVersion: "14697591"

uid: 7f786085-35b2-4e43-94ae-74d2914d598d

spec:

accessModes:

ReadWriteOnce

capacity:

storage: 300Mi

claimRef:

apiVersion: v1

kind: PersistentVolumeClaim

name: nap-nfs-pre-dr03-volym-pvc

namespace: my-ns

resourceVersion: "14697526"

uid: 902f8810-1daf-40dd-89b5-4ae5ebb77a85

csi:

driver: csi.trident.netapp.io

volumeAttributes:

backendUUID: 68564aae-6f05-43ca-8b28-0a7bf9f07645

internalName: trident_pvc_902f8810_1daf_40dd_89b5_4ae5ebb77a85

name: pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85

protocol: file

storage.kubernetes.io/csiProvisionerIdentity: 1623841646672-8081-csi.trident.netapp.io

volumeHandle: pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85

persistentVolumeReclaimPolicy: Delete

storageClassName: nap-nfs-pre-dr03-volym

volumeMode: Filesystem

status:

phase: Bound

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://github.com/NetApp/trident/issues/645#issuecomment-925550979, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAEW4CNJ4VDACKH77W2LH7TUDLF53ANCNFSM5ELOHIEA.
Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

wonderland on 23 Sep 2021

Hi,
I enabled debugmode and created a new volumesnapshot from the same PVC.

$ kubectl get volumesnapshot nap-nfs-pre-dr03-volym-pvc-snapshot-20210923-1 -n sfit -o yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  creationTimestamp: "2021-09-23T07:24:08Z"
  finalizers:
  - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
  - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  generation: 1
  name: nap-nfs-pre-dr03-volym-pvc-snapshot-20210923-1
  namespace: sfit
  resourceVersion: "124094111"
  uid: 839597ba-60eb-4786-a476-8f5aab3c52b5
spec:
  source:
    persistentVolumeClaimName: nap-nfs-pre-dr03-volym-pvc
  volumeSnapshotClassName: nap-snapshots
status:
  boundVolumeSnapshotContentName: snapcontent-839597ba-60eb-4786-a476-8f5aab3c52b5
  error:
    message: 'Failed to check and update snapshot content: error updating status for
      volume snapshot content snapcontent-839597ba-60eb-4786-a476-8f5aab3c52b5: snapshot
      controller failed to update snapcontent-839597ba-60eb-4786-a476-8f5aab3c52b5
      on API server: VolumeSnapshotContent.snapshot.storage.k8s.io "snapcontent-839597ba-60eb-4786-a476-8f5aab3c52b5"
      is invalid: status.restoreSize: Invalid value: -2252800: status.restoreSize
      in body should be greater than or equal to 0'
    time: "2021-09-23T07:24:09Z"
  readyToUse: false

I can see the same restoreSize of -2252800 in the "GRPC response" in the trident logs

time="2021-09-23T07:24:08Z" level=debug msg="GRPC call: /csi.v1.Identity/GetPluginInfo" requestID=de9290b6-39e4-4bcd-b96d-e8195331fc71 requestSource=CSI
time="2021-09-23T07:24:08Z" level=debug msg="GRPC request: " requestID=de9290b6-39e4-4bcd-b96d-e8195331fc71 requestSource=CSI
time="2021-09-23T07:24:08Z" level=debug msg=">>>> GetPluginInfo" Method=GetPluginInfo Type=CSI_Identity requestID=de9290b6-39e4-4bcd-b96d-e8195331fc71 requestSource=CSI
time="2021-09-23T07:24:08Z" level=debug msg="<<<< GetPluginInfo" Method=GetPluginInfo Type=CSI_Identity requestID=de9290b6-39e4-4bcd-b96d-e8195331fc71 requestSource=CSI
time="2021-09-23T07:24:08Z" level=debug msg="GRPC response: name:\"csi.trident.netapp.io\" vendor_version:\"21.07.1\" " requestID=de9290b6-39e4-4bcd-b96d-e8195331fc71 requestSource=CSI
time="2021-09-23T07:24:08Z" level=debug msg="GRPC call: /csi.v1.Controller/CreateSnapshot" requestID=7d58cb06-c5b0-4e1a-83a8-2ef4df08eb24 requestSource=CSI
time="2021-09-23T07:24:08Z" level=debug msg="GRPC request: source_volume_id:\"pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85\" name:\"snapshot-839597ba-60eb-4786-a476-8f5aab3c52b5\" " requestID=7d58cb06-c5b0-4e1a-83a8-2ef4df08eb24 requestSource=CSI
time="2021-09-23T07:24:08Z" level=debug msg=">>>> CreateSnapshot" Method=CreateSnapshot Type=CSI_Controller requestID=7d58cb06-c5b0-4e1a-83a8-2ef4df08eb24 requestSource=CSI
time="2021-09-23T07:24:08Z" level=debug msg="Found no existing snapshot snapshot-839597ba-60eb-4786-a476-8f5aab3c52b5 in other volumes." requestID=7d58cb06-c5b0-4e1a-83a8-2ef4df08eb24 requestSource=CSI
time="2021-09-23T07:24:08Z" level=debug msg=AddVolumeTransaction name=pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85-snapshot-839597ba-60eb-4786-a476-8f5aab3c52b5 op=addSnapshot requestID=7d58cb06-c5b0-4e1a-83a8-2ef4df08eb24 requestSource=CSI
time="2021-09-23T07:24:08Z" level=debug msg="Attempting snapshot create." backend=sfit-container-ontapnas-dr03 requestID=7d58cb06-c5b0-4e1a-83a8-2ef4df08eb24 requestSource=CSI snapshot=snapshot-839597ba-60eb-4786-a476-8f5aab3c52b5 volume=snapshot-839597ba-60eb-4786-a476-8f5aab3c52b5 volumeInternal=
time="2021-09-23T07:24:08Z" level=warning msg="Snapshot not found." requestID=7d58cb06-c5b0-4e1a-83a8-2ef4df08eb24 requestSource=CSI snapshotName=snapshot-839597ba-60eb-4786-a476-8f5aab3c52b5 volumeName=trident_pvc_902f8810_1daf_40dd_89b5_4ae5ebb77a85
time="2021-09-23T07:24:09Z" level=debug msg=removeTransactionFinalizers logSource=trident-crd-controller requestID=d83ef789-f346-45dd-8e7e-dacb86b60307 requestSource=CRD tx.ObjectMeta.DeletionTimestamp="2021-09-23 07:24:09 +0000 UTC" tx.ResourceVersion=124094102
time="2021-09-23T07:24:09Z" level=debug msg="Has finalizers, removing them." logSource=trident-crd-controller requestID=d83ef789-f346-45dd-8e7e-dacb86b60307 requestSource=CRD
time="2021-09-23T07:24:09Z" level=debug msg="<<<< CreateSnapshot" Method=CreateSnapshot Type=CSI_Controller requestID=7d58cb06-c5b0-4e1a-83a8-2ef4df08eb24 requestSource=CSI
time="2021-09-23T07:24:09Z" level=debug msg="GRPC response: snapshot:<size_bytes:-2252800 snapshot_id:\"pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85/snapshot-839597ba-60eb-4786-a476-8f5aab3c52b5\" source_volume_id:\"pvc-902f8810-1daf-40dd-89b5-4ae5ebb77a85\" creation_time:<seconds:1632381848 > ready_to_use:true > " requestID=7d58cb06-c5b0-4e1a-83a8-2ef4df08eb24 requestSource=CSI

gorantornqvist on 23 Sep 2021

It looks like the snapshot is in a different namespace than the PVC (sfit vs. my-ns)? That is not possible, K8s prevents data in a PVC from being accessed outside its own namespace. Can you please try to create the snapshot in the same namespace as the source PVC?

wonderland on 23 Sep 2021

Sorry, for that, an obfuscation attempt on my side gone wrong :)
The volumesnapshot and PVC are in the same namespace which is sfit.

gorantornqvist on 23 Sep 2021

Hello, dear all.
I have exactly the same problem.

Trident version: 21.07.1
Kubernetes version: v1.20.7
NetApp backend types: ONTAP

unbreakabl3 on 27 Sep 2021

Ok. I think I have found a pattern - in my case, the scenario was:

PV/PVC was backed up by Velero
PV/PVC was restored from a backup by Velero
During the restore new PV/PVC was created from VolumeSnapshotContent
This restored PVC now has this dataSource spec, which indicated it was restored from a snapshot
Now, when Im trying to create a new snapshot on that restored PVC, I'm getting this restoreSize error
Important - even if kubectl return an invalid: status.restoreSize: Invalid value: -69894144: status.restoreSize in body should be greater than or equal to 0 error, the snapshot is created on a NetApp side. I can confirm it by running tridentctl get snapshot

P.S - I did a test. I did delete one of the test PVC, which was restored exactly the same like the problematic one and did create this test PVC from scratch. Now, its doesn't have this dataSource spec. I did try to create a snapshot on that new test PVC and I did work like expected.

unbreakabl3 on 27 Sep 2021

Quick question. Are those of you experiencing this only using ONTAP-NAS backends and have 'useREST=true' in your backend config?

ameade on 27 Sep 2021

No, I have useREST: false in all my backends ...

EDIT: I have actually only tested snapshots with ONTAP-NAS backends sofar, we have a few ONTAP-SAN backends as well but I havent tested snapshots on these yet.

gorantornqvist on 28 Sep 2021

Quick question. Are those of you experiencing this only using ONTAP-NAS backends and have 'useREST=true' in your backend config?

NAS in our case

unbreakabl3 on 28 Sep 2021

👍1

I am seeing the same issue.
Trident version: 21.07.1
Kubernetes version: 1.20.10
Ontap Version: 9.7P14

Using ONTAP-NAS, with useREST not configured (guess it's 'useREST=false' then)

Failed to check and update snapshot content: error updating status for volume snapshot content snapcontent-08ee5223-0e2d-491e-88a9-b31588a82d64: snapshot controller failed to update snapcontent-08ee5223-0e2d-491e-88a9-b31588a82d64 on API server: VolumeSnapshotContent.snapshot.storage.k8s.io "snapcontent-08ee5223-0e2d-491e-88a9-b31588a82d64" is invalid: status.restoreSize: Invalid value: -646946816: status.restoreSize in body should be greater than or equal to 0

Waiting for a snapshot cattle-prometheus/k10-csi-snap-72dhsdpmcxsd4q9f to be created by the CSI driver.

lnlasnic on 30 Sep 2021

👍1

I am seeing the same issue. Trident version: 21.07.1 Kubernetes version: 1.20.10 Ontap Version: 9.7P14

Using ONTAP-NAS, with useREST not configured (guess it's 'useREST=false' then)

Failed to check and update snapshot content: error updating status for volume snapshot content snapcontent-08ee5223-0e2d-491e-88a9-b31588a82d64: snapshot controller failed to update snapcontent-08ee5223-0e2d-491e-88a9-b31588a82d64 on API server: VolumeSnapshotContent.snapshot.storage.k8s.io "snapcontent-08ee5223-0e2d-491e-88a9-b31588a82d64" is invalid: status.restoreSize: Invalid value: -646946816: status.restoreSize in body should be greater than or equal to 0

Waiting for a snapshot cattle-prometheus/k10-csi-snap-72dhsdpmcxsd4q9f to be created by the CSI driver.

Just tried in another k8s Cluster where the Trident version is 21.04.0 - in this cluster volume snapshot works perfect.
Same Kubernetes version, same Ontap backend, snapshot controller and config.

lnlasnic on 30 Sep 2021

I am seeing the same issue. Trident version: 21.07.1 Kubernetes version: 1.20.10 Ontap Version: 9.7P14
Using ONTAP-NAS, with useREST not configured (guess it's 'useREST=false' then)
Failed to check and update snapshot content: error updating status for volume snapshot content snapcontent-08ee5223-0e2d-491e-88a9-b31588a82d64: snapshot controller failed to update snapcontent-08ee5223-0e2d-491e-88a9-b31588a82d64 on API server: VolumeSnapshotContent.snapshot.storage.k8s.io "snapcontent-08ee5223-0e2d-491e-88a9-b31588a82d64" is invalid: status.restoreSize: Invalid value: -646946816: status.restoreSize in body should be greater than or equal to 0
Waiting for a snapshot cattle-prometheus/k10-csi-snap-72dhsdpmcxsd4q9f to be created by the CSI driver.

Just tried in another k8s Cluster where the Trident version is 21.04.0 - in this cluster volume snapshot works perfect. Same Kubernetes version, same Ontap backend, snapshot controller and config.

Downgrading Trident from 21.07.1 to 21.04.0 does work.
Could see that the csi-snapshotter was downgraded to 3.0.3, so I modified the trident-csi deployment and selected csi-snapshotter 4.0.0 and 4.1.1 - both are working with trident 21.04.0 - so not related to newer version of the csi-snapshotter.

Status:
Bound Volume Snapshot Content Name:  snapcontent-407419da-95b3-4868-b594-0e78484986ab
Creation Time:                       2021-09-30T21:25:01Z
Ready To Use:                        true
Restore Size:                        10Gi

lnlasnic on 30 Sep 2021

We're working on resolving this issue and hope to have an update in the near future.