Longhorn: Volume expansion (resize)

[mhemrg]

Is it possible to update a running volume's size?

[yasker]

Not for now. And it's not on the plan yet.

Update on Nov 2019: It's a high priority feature in the list now.

[mhemrg]

So can I create a new volume (larger than previous) based on a snapshot?

[yasker]

No, currently you cannot do that as well.

[errorsandwarnings]

How is it planned to increase a volume size then ? That seems a big limitation

[yasker]

Increase volume size should not be hard to implement with Longhorn. But it will need cooperation from filesystem right now.

It's not high in priority at the moment.

[dnauck]

Article about this topic with CSI: https://kubernetes.io/blog/2018/08/02/dynamically-expand-volume-with-csi-and-kubernetes/

[mariusstaicu]

this is a big blocker for using longhorn atm, as well as for using rancher2. is it gonna be postponed indefinitely like other rancher features ?

[yasker]

@mariusstaicu We're not pushing back the feature indefinitely. As you see there is a milestone assigned to the feature, that means we're tracking and planning to work on it.

More and more users are asking for this feature, so we've prioritized it from "no milestone" to v0.6.0 this year. But later we've decided to introduce an architecture refactor in v0.6.0, so we have to push some features to v0.7.0 instead.

Hope you can understand. If you have questions regarding other features/issues planning, please let me know as well in the issue.

[shuo-wu]

If we want to expand Longhorn volume by modifying the related PVC, the following requirements should be satisfied:

1. Kubernetes version should be v1.16 since the feature CSI Volume expansion is beta in this version: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#csi-volume-expansion
2. Two new interfaces ControllerExpandVolume and NodeExpandVolume should be implemented. And they are introduced in CSI v1.1.0: https://kubernetes-csi.github.io/docs/volume-expansion.html
3. New sidecar container named CSI external-resizer should be deployed: https://kubernetes-csi.github.io/docs/external-resizer.html

For now, A new option can be provided on the Longhorn UI page to directly resize Longhorn volumes.

[gittygoo]

+1 that this is needed, cant understand how this is not a high priority feature

[yasker]

Due to the short cycle of v0.7.0 release (about a month after v0.6.2), we have no choice but to move this feature along with some other features originally planned for v0.7.0 like #46 to v0.8.0 release instead.

At this moment, the volume expansion feature is at the top of our list, and we will get to it as soon as we can.

[gittygoo]

Will this improvement help with https://serverfault.com/questions/955293/how-to-increase-disk-size-in-a-stateful-set ? I tried the commands there with longhorn and got "is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize"

[shuo-wu]

Will this improvement help with https://serverfault.com/questions/955293/how-to-increase-disk-size-in-a-stateful-set ? I tried the commands there with longhorn and got "is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize"

As the error mentions, you need to satisfy some conditions if you want to expand the volume by modifying the PVC capacity. Here are the details:

1. The PVC is dynamically provisioned by the StorageClass.
2. The Kubernetes is version 1.16+ or the feature gate for volume expansion is enabled.
3. The StorageClass should support resize:
   3.1. allowVolumeExpansion: true is set in the StorageClass
   3.2. The external resizer is introduced
   3.3. The CSI plugin of the StorageClass implemented the corresponding interface.

[yasker]

We should be able to support ext2/3/4 with resize2fs and xfs with xfs_growfs natively.

[meldafrawi]

Validation: PASS

Longhorn UI version: wangsiye/longhorn-ui:2d9d3d9

Steps to test: (1/3)

• Create multiple 4 volumes, each of size 5 GB, Attach them to nodes
• Format each volume using one of the following formats (ext 2/3/4 & xfs)
• Mount volumes to directories on the nodes.
• Check volume size and used space using df -h command (PASS)
• Write 4 GB data file to each volume
• For each volume, from Operation menu, Click Expand Volume, set size to 10 GB, and click OK
• Check volume size and used space using df -h command (PASS)
• Add more 4 GB data file to each volume
• Check volume size and used space using df -h command (PASS)
• Check volume size expanded on Volume & Volume Details pages

Steps to test: (2/3)

• Kubernetes version: 1.15
• Create a volume used by a pod using:
  kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/master/examples/pvc.yaml
• From Operation menu, Click Expand Volume, A message indicate that The capacity of related PV and PVC will not be updated is found
• Set size to 10 GB, and click OK
• Edit PV/PVC capacity to match new volume size.
• from inside the pod, check volume size using df -h command
• Write 8 GB data file to volume, and compute its md5sum checksum
• Create volume backup
• Restore volume backup with new volume name.
• Attach new volume to a node, and check data file exists, and verify it's checksum.

Steps to test: (3/3)

• Kubernetes version: 1.16
• Create a volume used by a pod using:
  kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/master/examples/pvc.yaml
• Find the corresponding PVC for Longhorn volume then modify requested storage of the PVC spec
• Modify the size to 20Gi, In Longhorn UI, Volume page & Volumes Details page, size of volume should changed to the new size
• Execute df -h inside the pod, and verify volume size changed accordingly.

[shuo-wu]

@meldafrawi

Add 2 more steps for the test1 and test3:

1. Fill up the expanded volumes and get the checksum.
2. Unmount & detach volumes, then reattach & remount volumes in different places. (For test3, it's better to use StatefulSet then scale down and scale up it.) Check the checksum.

Test offline expansion:

1. Create and attach a volume
2. Format and mount the volume. Fill up the volume and get the checksum.
3. Unmount and detach the volume.
4. Expand the volume and wait for the expansion complete. (Use UI or modify the PVC spec)
5. Reattach and remount the volume. Check the checksum and if the filesystem is expanded.
6. Fill up the expanded parts and get the checksum.
7. Unmount and detach the volume.
8. Launch a workload for it on a different node. Check the checksum and if the volume can be used as usual.

Test expansion with revert and backup:

1. Create and attach a volume.
2. Format and mount the volume. Fill up the volume and get the checksum.
3. Create the 1st snapshot and backup.
4. Expand the volume. Fill up the expanded part and get the checksum.
5. Create the 2nd snapshot and backup. Check if the backup volume size is expanded. Check the backup content by creating volumes from the backups.
6. Clean up then refill the volume. Get the checksum.
7. Create the 3rd snapshot and backup.
8. Revert to the 2nd snapshot. Check the checksum.
9. Revert to the 1st snapshot. Follow the doc to manually resize the filesystem. Check the checksum and if we can still use the expanded part.
10. Revert to the 3rd snapshot. Check the checksum.
11. Check if we can delete the 1st snapshot.
12. Check if we can delete backups without losing data.

[meldafrawi]

Validation: Partially Failed

• Volume size before expansion = 4GB
• Volume size after expansion = 7GB

Test expansion with revert and backup:

1. Create and attach a volume.
2. Format and mount the volume. Fill up the volume and get the checksum.
3. Create the 1st snapshot and backup.
4. Expand the volume. Fill up the expanded part and get the checksum.
5. Create the 2nd snapshot and backup. Check if the backup volume size is expanded. Check the backup content by creating volumes from the backups.
6. Clean up then refill the volume. Get the checksum.
7. Create the 3rd snapshot and backup.
8. Revert to the 2nd snapshot. Check the checksum. (PASS)
9. Revert to the 1st snapshot. Check the checksum and if we can still use the expanded part. (FAILED)

• After reverting to 1st snapshot using df -h indicates that volume size = Volume size before expansion = 4 GB & Used Percentage = 100%
• Can't add more data to volume

[shuo-wu]

@meldafrawi Can you follow this doc to modify the step 9 and re-test it?

[meldafrawi]

Validation: PASSED