Trident: Expose kubelet_volume* metrics

[ChrisCavegn]

Since Kubernetes 1.8, kubelet can expose metrics about PVC usage, please see https://github.com/google/cadvisor/issues/1702.

Exposed metrics are (copied from said issue):

• kubelet_volume_stats_available_bytes
• kubelet_volume_stats_capacity_bytes
• kubelet_volume_stats_inodes
• kubelet_volume_stats_inodes_free
• kubelet_volume_stats_inodes_used
• kubelet_volume_stats_used_bytes

This is a feature request for trident to expose these metrics. As long as these metrics are not exposed, monitoring the PVC's isn't really possible.

[adkerr]

See https://github.com/kubernetes/kubernetes/blob/936f007cb99d1fc3d8d60e36e7cd0545914090b4/pkg/volume/volume.go#L37 for metrics interface

[msteenhu]

I have been studying Netapp api's for about a half day, and as far as I can see, there are no quota/usage stats exposed for qtrees (ontap-nas-economy). So already one show stopper for ontap...

[msteenhu]

For ontap-nas it should work out

[msteenhu]

Don't mind my ignorance before. The rest api can do this just fine:

```
2>/dev/null curl -u”SECRET:SECRET” -k -X GET --header 'Accept: application/json' 'https://HOSTPORTapi/4.0/ontap/qtrees?name=QTREE_NAME’ | jq '.result.records[0] | "scale=2; (.disk_used)/(.disk_limit)*100"' | sed 's/"//g' | bc
````

[kremers]

Voting for this issue +1

[pirminjanka]

+1

[mscarpi]

for nas-economy it should be possible by using non limiting tree quotas on the top volume and use quota report export to pull out the used data for all the qtrees.

Just my 2 cents.

[sevenval-admins]

Unfortunately this feature in our setup is a blocker for monitoring and routing alerts to the right contacts. We can check the fill level of the volumes via the metrics of Harvest, but the connection to the K8s namespace is missing and therefore the possibility to deliver alerts in a useful way.

So my question is whether there is a way to promote or prioritize this feature.

I would be very grateful for an implementation.

[ghouscht]

We got two different kinds of clusters, one is running quite an old version of trident (19.01.0) without CSI. On this cluster the kubelet_volume_* metrics are present. On our more up-to-date clusters we run trident version 20.01.1 in CSI mode and the metrics are missing there. So this is probably related to CSI.

According to this PR https://github.com/kubernetes/kubernetes/pull/76188 it should be possible for a CSI plugin to report metrics, but this must be implemented in the CSI plugin (=Trident) which is not the case at the moment https://github.com/NetApp/trident/blob/35be15cc054a3bc1e04559c188d6e57a8833a798/frontend/csi/node_server.go#L188-L194

So this is a huge step back for us in comparision to the non CSI mode we used in the past!

[emmahone]

Are there any plans ot implement NodeGetVolumeStats in the Trident CSI plugin? If so is there a tracker for that work? I have an end user who is requesting this functionality.

[gnarl]

@emmahone We are working on adding additional telemetry for the Trident 20.07 release. NodeGetVolumeStats should be included with that telemetry.

[ghouscht]

looks like it is implemented in master now: https://github.com/NetApp/trident/commit/f48637e302a6b42ef508bf11c3b9a8304ff978c0 - thank you!

[gnarl]

This enhancement is included in the Trident 20.07 release.