Zfs: zvol space referenced is more than consumer reports
I'm observing dubious values for the used property of my ZVOL.
It is WAY higher than the actual space used (reported by e.g. df), and cannot be explained by snapshot usage.
At times it is even higher than the configured volsize (by a large margin), but does not fail in any way (yet).
This issue seems related to #4436 and #3255, but I do not use RAIDZ neither share via iSCSI.
I have included information before and after taking a new snapshot and destroying the old one to prove that the snapshots are not at fault (but experience the same "inflation").
Old Snapshot
ZVOL Properties:
root@WOLF-SERVER:/# zfs get all rpool/docker
NAME PROPERTY VALUE SOURCE
rpool/docker type volume -
rpool/docker creation Fri Dec 14 1:01 2018 -
rpool/docker used 91.0G -
rpool/docker available 10.2G -
rpool/docker referenced 51.9G -
rpool/docker compressratio 1.04x -
rpool/docker reservation none default
rpool/docker volsize 60G local
rpool/docker volblocksize 8K default
rpool/docker checksum on default
rpool/docker compression lz4 inherited from rpool
rpool/docker readonly off default
rpool/docker createtxg 239 -
rpool/docker copies 1 default
rpool/docker refreservation none default
rpool/docker primarycache all default
rpool/docker secondarycache all default
rpool/docker usedbysnapshots 39.1G -
rpool/docker usedbydataset 51.9G -
rpool/docker usedbychildren 0B -
rpool/docker usedbyrefreservation 0B -
rpool/docker logbias latency default
rpool/docker dedup off default
rpool/docker mlslabel none default
rpool/docker sync standard default
rpool/docker refcompressratio 1.04x -
rpool/docker written 44.7G -
rpool/docker logicalused 94.6G -
rpool/docker logicalreferenced 54.2G -
rpool/docker volmode default default
rpool/docker snapshot_limit none default
rpool/docker snapshot_count none default
rpool/docker snapdev hidden default
rpool/docker context none default
rpool/docker fscontext none default
rpool/docker defcontext none default
rpool/docker rootcontext none default
rpool/docker redundant_metadata all default
ZVOL Snapshot:
root@WOLF-SERVER:/# zfs list -r -t snapshot rpool/docker
NAME USED AVAIL REFER MOUNTPOINT
rpool/[email protected] 39.1G - 46.3G -
Actual Use:
root@WOLF-SERVER:/# df -h /dev/zvol/rpool/docker
Filesystem Size Used Avail Use% Mounted on
/dev/zd0 59G 6.6G 50G 12% /var/lib/docker
New Snapshot
ZVOL Properties:
root@WOLF-SERVER:/# zfs get all rpool/docker
NAME PROPERTY VALUE SOURCE
rpool/docker type volume -
rpool/docker creation Fri Dec 14 1:01 2018 -
rpool/docker used 52.0G -
rpool/docker available 49.5G -
rpool/docker referenced 51.9G -
rpool/docker compressratio 1.04x -
rpool/docker reservation none default
rpool/docker volsize 60G local
rpool/docker volblocksize 8K default
rpool/docker checksum on default
rpool/docker compression lz4 inherited from rpool
rpool/docker readonly off default
rpool/docker createtxg 239 -
rpool/docker copies 1 default
rpool/docker refreservation none default
rpool/docker primarycache all default
rpool/docker secondarycache all default
rpool/docker usedbysnapshots 37.1M -
rpool/docker usedbydataset 51.9G -
rpool/docker usedbychildren 0B -
rpool/docker usedbyrefreservation 0B -
rpool/docker logbias latency default
rpool/docker dedup off default
rpool/docker mlslabel none default
rpool/docker sync standard default
rpool/docker refcompressratio 1.04x -
rpool/docker written 36.9M -
rpool/docker logicalused 54.2G -
rpool/docker logicalreferenced 54.2G -
rpool/docker volmode default default
rpool/docker snapshot_limit none default
rpool/docker snapshot_count none default
rpool/docker snapdev hidden default
rpool/docker context none default
rpool/docker fscontext none default
rpool/docker defcontext none default
rpool/docker rootcontext none default
rpool/docker redundant_metadata all default
ZVOL Snapshot:
root@WOLF-SERVER:/# zfs list -r -t snapshot rpool/docker
NAME USED AVAIL REFER MOUNTPOINT
rpool/[email protected] 38.9M - 51.9G -
Actual Use:
root@WOLF-SERVER:/# df -h /dev/zvol/rpool/docker
Filesystem Size Used Avail Use% Mounted on
/dev/zd0 59G 6.6G 50G 12% /var/lib/docker
Background
The zfs pool is created on a singe SSD with ashift=12.
The ext4 formated ZVOL is used by docker, which in turn uses the overlay2 storage driver for its containers and images. (Even if docker has a zfs storage driver, I chose this path to be able to take snapshots and rollback the running containers).
I guess the overlay2 storage driver use on a zfs FS could lead to some kind of metadata inflation due to referencing incompatibilities between the two?
System information
Type | Version/Name
--- | ---
Distribution Name | Debian
Distribution Version | 9.7
Linux Kernel | 4.9.0-8
Architecture | amd64
ZFS Version | v0.7.12-1~bpo9+1
SPL Version | v0.7.12-1~bpo9+1
POOL Properties:
root@WOLF-SERVER:/# zpool get all rpool
NAME PROPERTY VALUE SOURCE
rpool size 111G -
rpool capacity 50% -
rpool altroot - default
rpool health ONLINE -
rpool version - default
rpool bootfs rpool/system/default2 local
rpool delegation on default
rpool autoreplace off default
rpool cachefile - default
rpool failmode wait default
rpool listsnapshots off default
rpool autoexpand off default
rpool dedupditto 0 default
rpool dedupratio 1.00x -
rpool free 55.0G -
rpool allocated 56.0G -
rpool readonly off -
rpool ashift 12 local
rpool comment - default
rpool expandsize - -
rpool freeing 0 -
rpool fragmentation 34% -
rpool leaked 0 -
rpool multihost off default
rpool feature@async_destroy enabled local
rpool feature@empty_bpobj active local
rpool feature@lz4_compress active local
rpool feature@multi_vdev_crash_dump disabled local
rpool feature@spacemap_histogram active local
rpool feature@enabled_txg active local
rpool feature@hole_birth active local
rpool feature@extensible_dataset active local
rpool feature@embedded_data active local
rpool feature@bookmarks enabled local
rpool feature@filesystem_limits enabled local
rpool feature@large_blocks enabled local
rpool feature@large_dnode disabled local
rpool feature@sha512 disabled local
rpool feature@skein disabled local
rpool feature@edonr disabled local
rpool feature@userobj_accounting active local
Greek64
All 3 comments
root@WOLF-SERVER:/# zfs get all rpool/docker
NAME PROPERTY VALUE SOURCE
rpool/docker type volume -
rpool/docker used 91.0G -
rpool/docker referenced 51.9G -
rpool/docker compressratio 1.04x -
rpool/docker reservation none default
rpool/docker volsize 60G local
rpool/docker usedbysnapshots 39.1G -
rpool/docker usedbydataset 51.9G -
rpool/docker usedbychildren 0B -
rpool/docker usedbyrefreservation 0B -
You can see the key space usage stats above. The zvol is referencing 51.9GB directly, plus the snapshots are using 39.1GB, for a total of 91GB.
The "actual use" that you reported (below) is unfortunately not very relevant. It is a report from the consumer of the ZVOL (ext4). The disconnect between the 6.6GB of space that ext4 has allocated and the 51.9GB of space that is allocated in the zvol is explained by the fact that any blocks of the zvol that were ever written by ext4 must remain allocated in the zvol, even though ext4 no longer needs them.
You may be able to work around this by instructing ext4 to write zeros or issue TRIM commands to the blocks it no longer needs. (But I don't know how to do that or if all the intervening layers support the TRIM command.)
root@WOLF-SERVER:/# df -h /dev/zvol/rpool/docker
Filesystem Size Used Avail Use% Mounted on
/dev/zd0 59G 6.6G 50G 12% /var/lib/docker
Let us know if you have additional questions on this.
ahrens
on 12 Feb 2019
The disconnect between the 6.6GB of space that ext4 has allocated and the 51.9GB of space that is allocated in the zvol is explained by the fact that any blocks of the zvol that were ever written by ext4 must remain allocated in the zvol, even though ext4 no longer needs them.
Oh, that explains a lot. Thank you very much.
Sorry for posting an issue, which was essentially a question. (next time I will use the mailing list first)
Greek64
on 12 Feb 2019
You may be able to work around this by instructing ext4 to write zeros or issue TRIM commands to the blocks it no longer needs. (But I don't know how to do that or if all the intervening layers support the TRIM command.)
I used the fstrim command, and zfs successfully understood that the blocks are no longer needed.
Of course, they are now referenced by the snapshot itself.
rpool/docker usedbysnapshots 46.9G -
rpool/docker usedbydataset 5.99G -
Adding the discard option for the mounted ZVOL in \etc\fstab effectively enables the Linux Kernel to issue the trim commands online, without the need to execute fstrim on-demand.
Greek64
on 12 Feb 2019
Related issues
geek-at
路
3Comments
RNCTX
路
3Comments
dbavatar
路
4Comments
cinterloper
路
3Comments
nh2
路
3Comments