Trident: [tridentctl] import failed to get size of volume (ontap)

[sakaia]

Describe the bug

tridentctl import volume is failed with error message import failed to get size of volume.
but tridentctl get volume works fine.

$ ./tridentctl -n trident import volume customBackendName pvc-02432125-9c89-4bd3-8c5e-f03e1044d1ce -f importpvc.yaml
Error: could not import volume: volume import failed to get size of volume: volume pvc-02432125-9c89-4bd3-8c5e-f03e1044d1ce was not found (400 Bad Request)
command terminated with exit code 1
$ ./tridentctl -n trident get volume pvc-02432125-9c89-4bd3-8c5e-f03e1044d1ce
+------------------------------------------+---------+---------------+----------+--------------------------------------+--------+---------+
|                   NAME                   |  SIZE   | STORAGE CLASS | PROTOCOL |             BACKEND UUID             | STATE  | MANAGED |
+------------------------------------------+---------+---------------+----------+--------------------------------------+--------+---------+
| pvc-02432125-9c89-4bd3-8c5e-f03e1044d1ce | 1.0 GiB | basic-csi     | file     | 3ee86ea5-ab92-4a9e-a316-e55c9e766de1 | online | true    |
+------------------------------------------+---------+---------------+----------+--------------------------------------+--------+---------+

Environment

Provide accurate information about the environment to help us reproduce the issue.

• Trident version: 20.10
• Trident installation flags used: default
• Container runtime: Docker 19.03.13
• Kubernetes version: 1.18.5
• Kubernetes orchestrator: kubeadm
• Kubernetes enabled feature gates:
• OS: Ubuntu 18.04
• NetApp backend types: ONTAP Simulator 9.7 /ontap-nas
• Other:

To Reproduce

./tridentctl -n trident import volume customBackendName pvc-02432125-9c89-4bd3-8c5e-f03e1044d1ce -f importpvc.yaml

Expected behavior

tridentctl import volume works successfully

Additional context
This issue is based on sample code on Page 74 of TR-4798.
I use importpvc.yaml as follows (based on 74 page sample code)

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: importpvc
  namespace: airflow
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: basic-csi

[wonderland]

Did you check the name of the volume on the storage system? Connect to the System Manager UI or the CLI (volume show -vserver ). Usually a prefix is in place ("trident_" by default), so the name of the PV does not exactly match the name of the volume on the storage system. The import need the exact name of the volume on the storage system. That would explain the "volume pvc-02432125-9c89-4bd3-8c5e-f03e1044d1ce was not found" error message.

[sakaia]

Thank you for your help, One step forward.
Would you give me any suggestion?

$ ./tridentctl -n trident import volume customBackendName trident_pvc_02432125_9c89_4bd3_8c5e_f03e1044d1ce -f importpvc.yaml
Error: Post "http://127.0.0.1:8000/trident/v1/volume/import": EOF
command terminated with exit code 1

$ ./tridentctl -n trident logs 

snip

time="2021-01-20T09:46:54Z" level=debug msg="REST API call received." duration="15.616μs" method=POST requestID=bd52d03-4120-44f1-92aa-78d2dccf0a81 requestSource=REST route=ImportVolume uri=/trident/v1/volume/import
time="2021-01-20T09:46:54Z" level=debug msg="Found requested frontend." name=k8s_csi_helper requestID=bd52d0f3-4120-44f1-92aa-78d2dccf0a81 requestSource=REST
time="2021-01-20T09:46:54Z" level=debug msg=ImportVolume request="&{customBackendName trident_pvc_02432125_9c89_4bd3_8c5e_f03e1044d1ce false eyJhcGlWZXJzaW9uIjoidjEiLCJraW5kIjoiUGVyc2lzdGVudFZvbHVtZUNsYWltIiwibWV0YWRhdGEiOnsibmFtZSI6ImltcG9ydHB2YzQiLCJuYW1lc3BhY2UiOiJhaXJmbG93In0sInNwZWMiOnsiYWNjZXNzTW9kZXMiOlsiUmVhZFdyaXRlTWFueSJdLCJzdG9yYWdlQ2xhc3NOYW1lIjoiYmFzaWMtY3NpIn19}" requestID=bd52d0f3-4120-44f1-92aa-78d2dccf0a81 requestSource=REST
time="2021-01-20T09:46:54Z" level=debug msg="ImportVolume: received PVC data." PVC=importpvc4 PVC_accessModes="[ReadWriteMany]" PVC_annotations="map[]" PVC_requestedSize="{{0 0} {<nil>}  }" PVC_storageClass=basic-csi PVC_volume= requestID=bd52d0f3-4120-44f1-92aa-78d2dccf0a81 requestSource=REST

snip

time="2021-01-20T09:49:41Z" level=debug msg="Could not find cached PV object by name." name=pvc-18451098-90ee-433b-9b15-d67da5fdab87 requestID=bd52d0f3-4120-44f1-92aa-78d2dccf0a81 requestSource=REST
time="2021-01-20T09:49:41Z" level=debug msg="PV not yet in cache, waiting." increment=5.482476256s name=pvc-18451098-90ee-433b-9b15-d67da5fdab87 requestID=bd52d0f3-4120-44f1-92aa-78d2dccf0a81 requestSource=REST
time="2021-01-20T09:49:47Z" level=debug msg="Could not find cached PV object by name." name=pvc-18451098-90ee-433b-9b15-d67da5fdab87 requestID=bd52d0f3-4120-44f1-92aa-78d2dccf0a81 requestSource=REST
time="2021-01-20T09:49:47Z" level=debug msg="PV not yet in cache, waiting." increment=4.54940459s name=pvc-18451098-90ee-433b-9b15-d67da5fdab87 requestID=bd52d0f3-4120-44f1-92aa-78d2dccf0a81 requestSource=REST
time="2021-01-20T09:49:51Z" level=debug msg="Could not find cached PV object by name." name=pvc-18451098-90ee-433b-9b15-d67da5fdab87 requestID=bd52d0f3-4120-44f1-92aa-78d2dccf0a81 requestSource=REST
time="2021-01-20T09:49:51Z" level=error msg="error waiting for PV pvc-18451098-90ee-433b-9b15-d67da5fdab87; PV pvc-18451098-90ee-433b-9b15-d67da5fdab87 was not in cache after 180.00 seconds" handler=ImportVolume requestID=bd52d0f3-4120-44f1-92aa-78d2dccf0a81 requestSource=REST
time="2021-01-20T09:49:51Z" level=debug msg="REST API call complete." duration=2m57.369339508s method=POST requestID=bd52d0f3-4120-44f1-92aa-78d2dccf0a81 requestSource=REST route=ImportVolume status_code=400 uri=/trident/v1/volume/import

[wonderland]

It seems I didn't read your original post carefully enough. Page 74 of that TR talks about cloning. Could you help me understand what you are trying to achieve? Are you trying to import a volume that previously existed in this or another k8s cluster? Or are you trying to clone a volume that still exists within the k8s cluster?

[balaramesh]

@sakaia it looks like you are trying to import a volume that is already present. From the output you had shared:

$ ./tridentctl -n trident get volume pvc-02432125-9c89-4bd3-8c5e-f03e1044d1ce
+------------------------------------------+---------+---------------+----------+--------------------------------------+--------+---------+
|                   NAME                   |  SIZE   | STORAGE CLASS | PROTOCOL |             BACKEND UUID             | STATE  | MANAGED |
+------------------------------------------+---------+---------------+----------+--------------------------------------+--------+---------+
| pvc-02432125-9c89-4bd3-8c5e-f03e1044d1ce | 1.0 GiB | basic-csi     | file     | 3ee86ea5-ab92-4a9e-a316-e55c9e766de1 | online | true    |
+------------------------------------------+---------+---------------+----------+--------------------------------------+--------+---------+

The volume named trident_pvc_02432125_9c89_4bd3_8c5e_f03e1044d1ce looks to be already present in your Kubernetes environment and managed by Trident [the output above shows that there is already a volume with that name].

It looks like what you are attempting to do is to clone an existing volume on the storage cluster, and then import it. For the import to be successful, you will need to provide the name of the cloned volume. The syntax of the import command would look like this:

tridentctl import volume <name-of-backend> <name-of-cloned-volume-on-storage-cluster> -f <pvc-definition.yaml> -n trident

[wonderland]

Well, if this is about cloning an existing PVC that is still attached to the same cluster (and assuming you run a recent k8s version), just make use of the CSI snapshot and cloning capabilities. See https://netapp-trident.readthedocs.io/en/stable-v20.10/kubernetes/operations/tasks/volumes/snapshots.html#create-pvcs-from-volumesnapshots

[sakaia]

@balaramesh @wonderland Thanks for your help

I successfully create it by cleating the volume via ONTAP directly.

From looking around, I comment out following lines in sample code and add tentative file name, it causes this problem.

``
clonePvcName = "{{ task_instance.xcom_pull(task_ids='clone-source', key='clone_pvc_name') }}"
```

[zakkg3]

Did you check the name of the volume on the storage system? Connect to the System Manager UI or the CLI (volume show -vserver ). Usually a prefix is in place ("trident_" by default), so the name of the PV does not exactly match the name of the volume on the storage system. The import need the exact name of the volume on the storage system. That would explain the "volume pvc-02432125-9c89-4bd3-8c5e-f03e1044d1ce was not found" error message.

You can also get the name of the volume as it is in the svm by checking the PV

kubectl get pv pvc-4bea053e-7e95-49c4-823e-c6d309996f83 -o yaml

...
  csi:
    driver: csi.trident.netapp.io
    volumeAttributes:
      backendUUID: 5dc37e1e-8336-4f86-b9b9-70a2329759a7
      internalName: trident_pvc_4bea053e_7e95_49c4_823e_c6d309996f83

the internalName is the one we need here.