Zfs: Allow removal of non-redundant devices from/shrinking of a pool

[krichter722]

Describe the problem you're observing

Adding a device to a pool causes the device to be bound to the pool forever in most cases since removal of the device requires recreation of the pool without the device to be removed. While it's common in professional environments to plan the storage capacity of pools based on usecases generated by IT consultant professionals which will hardly ever require a device to be removed/the pool to be shrinked after the pool creation a removal is a realistic usecase in other server and desktop environments.

It'd be nice to have a command or to change zpool remove so that it's possible to remove device like it's possible to remove PVs from LVM (LVM doesn't take care about the filesystem) or like btrfs offers.

[sempervictus]

This is a very old request, and effectively cannot be solved without a magical function referred to as block pointer rewrite (for this and defrag) which is about as likely to be written as an operatic duet for McCain and Putin (performed in each of their hometowns). Once a data vdev is created in the pool, it, and its layout, are there forever. Really sucks when you forget the words "cache" or "log" in your "zpool add" invocation.

[gmelikov]

If I understand the problem - zpool labelclear won't work on disk, which was zpool removed, and you need to full dd if=/dev/zero it to work. Maybe just make work this case for zpool labelclear?