Zfs: feature@async_destroy not changing state properly for snapshots
System information
Type | Version/Name
--- | ---
Distribution Name | CentOS
Distribution Version | 7
Linux Kernel | 4.9.11-elrepo
Architecture | x86_64
ZFS Version | 0.7.0-rc3_119_g912e2ba
SPL Version | 0.7.0-rc3_3_g8d5feec
Describe the problem you're observing
# zpool get all pool-storage1 NAME PROPERTY VALUE SOURCE pool-storage1 size 145T - pool-storage1 capacity 71% - pool-storage1 altroot - default pool-storage1 health ONLINE - pool-storage1 guid 16465121322032609374 - pool-storage1 version - default pool-storage1 bootfs - default pool-storage1 delegation on default pool-storage1 autoreplace off default pool-storage1 cachefile - default pool-storage1 failmode wait default pool-storage1 listsnapshots off default pool-storage1 autoexpand off default pool-storage1 dedupditto 0 default pool-storage1 dedupratio 1.00x - pool-storage1 free 41.0T - pool-storage1 allocated 104T - pool-storage1 readonly off - pool-storage1 ashift 12 local pool-storage1 comment - default pool-storage1 expandsize - - pool-storage1 freeing 6.57T - pool-storage1 fragmentation 20% - pool-storage1 leaked 0 - pool-storage1 feature@async_destroy enabled local pool-storage1 feature@empty_bpobj active local pool-storage1 feature@lz4_compress active local pool-storage1 feature@spacemap_histogram active local pool-storage1 feature@enabled_txg active local pool-storage1 feature@hole_birth active local pool-storage1 feature@extensible_dataset active local pool-storage1 feature@embedded_data active local pool-storage1 feature@bookmarks active local pool-storage1 feature@filesystem_limits enabled local pool-storage1 feature@large_blocks active local pool-storage1 feature@large_dnode disabled local pool-storage1 feature@sha512 disabled local pool-storage1 feature@skein disabled local pool-storage1 feature@edonr disabled local pool-storage1 feature@userobj_accounting disabled local
See zpool-features(5) which contradicts the bolded lines.
Describe how to reproduce the problem
$ zfs destroy pool-storage1/dataset@snap6%snap8
Include any warning/errors/backtraces from the system logs
Nothing applicable here
Edit: That's "zfs destroy", not "zpool destroy". :/
DeHackEd
All 15 comments
@DeHackEd doesn't feature@async_destroy only work on file systems? You're destroying snapshots, i don't think we should expect this feature to be active in this case.
It's true the man page states
This feature is only active while freeing is non-zero.
but it also says
When async_destroy is enabled the file system's data will be reclaimed by a background process
loli10K
on 14 Mar 2017
I assume "the file system" here means ZFS. It was definitely being reclaimed by a background process rather than in the foreground with a hung terminal.
DeHackEd
on 14 Mar 2017
My point still remains: you're destroying snapshots, not file systems:
zfs destroy pool-storage1/dataset@snap6%snap8
Running zfs destroy pool-storage1/dataset would have activated the feature.
loli10K
on 14 Mar 2017
@DeHackEd just in case i tried to reproduce this same behaviour on a smartos box:
[root@52-54-00-d3-7a-f3 ~]# uname -a
SunOS 52-54-00-d3-7a-f3 5.11 joyent_20170202T040152Z i86pc i386 i86pc
[root@52-54-00-d3-7a-f3 ~]# zfs list -t snap -r testpool/fs
NAME USED AVAIL REFER MOUNTPOINT
testpool/fs@snap1 100M - 100M -
testpool/fs@snap2 100M - 100M -
testpool/fs@snap3 100M - 100M -
testpool/fs@snap4 100M - 100M -
testpool/fs@snap5 100M - 100M -
testpool/fs@snap6 100M - 100M -
testpool/fs@snap7 100M - 100M -
testpool/fs@snap8 100M - 100M -
testpool/fs@snap9 100M - 100M -
testpool/fs@snap10 0 - 100M -
[root@52-54-00-d3-7a-f3 ~]# zfs destroy testpool/fs@snap3%snap7 && while true; do zpool get feature@async_destroy,freeing testpool; sleep 1; done
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 768M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 512M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 512M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 256M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 0 default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 0 default
^C
[root@52-54-00-d3-7a-f3 ~]# zfs destroy -r testpool/fs && while true; do zpool get feature@async_destroy,freeing testpool; sleep 1; done
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 12.0M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 12.0M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 10.0M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 9.03M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 8.03M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 7.03M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 7.03M default
[...]
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 0 default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 0 default
^C
[root@52-54-00-d3-7a-f3 ~]#
I think it's working as intended, feature@async_destroy is "active" only if you're destroying file systems, not snapshots. If you still believe this is a bug then it's also affecting upstream.
loli10K
on 15 Mar 2017
this seems like a bug
@kpande let's suppose this is indeed a bug and feature@async_destroy state doesn't get updated properly: how do you explain the fact we can exhibit the same "async" behaviour when destroying snapshots with feature@async_destroy=disabled? Does this mean disabled features are also broken?
root@debian-8-zfs:~# zpool get feature@async_destroy
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy disabled local
root@debian-8-zfs:~# zfs list -t snap -r testpool/fs
NAME USED AVAIL REFER MOUNTPOINT
testpool/fs@snap1 100M - 100M -
testpool/fs@snap2 100M - 100M -
testpool/fs@snap3 100M - 100M -
testpool/fs@snap4 100M - 100M -
testpool/fs@snap5 100M - 100M -
testpool/fs@snap6 100M - 100M -
testpool/fs@snap7 100M - 100M -
testpool/fs@snap8 100M - 100M -
testpool/fs@snap9 100M - 100M -
testpool/fs@snap10 0 - 100M -
root@debian-8-zfs:~# zfs destroy testpool/fs@snap1%snap10 && while true; do zpool get feature@async_destroy,freeing testpool; sleep 1; done
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy disabled local
testpool freeing 701M -
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy disabled local
testpool freeing 500M -
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy disabled local
testpool freeing 300M -
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy disabled local
testpool freeing 100M -
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy disabled local
testpool freeing 0 -
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy disabled local
testpool freeing 0 -
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy disabled local
testpool freeing 0 -
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy disabled local
testpool freeing 0 -
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy disabled local
testpool freeing 0 -
^C
root@debian-8-zfs:~#
Does this mean disabled features are also broken?
Possibly, yes.
DeHackEd
on 15 Mar 2017
@DeHackEd well, the fact that you edited you reply from "Yes" to "Possibly, yes" makes me hope i'm getting close to convince you, so here we go:
root@debian-8-zfs:~# #
root@debian-8-zfs:~# POOLNAME='testpool'
root@debian-8-zfs:~# TMPDIR='/var/tmp'
root@debian-8-zfs:~# #
root@debian-8-zfs:~# fallocate -l 64m $TMPDIR/zpool.dat
root@debian-8-zfs:~# zpool create -f $POOLNAME $TMPDIR/zpool.dat -o feature@lz4_compress=disabled
root@debian-8-zfs:~# zpool get feature@lz4_compress
NAME PROPERTY VALUE SOURCE
testpool feature@lz4_compress disabled local
root@debian-8-zfs:~# zfs set compression=lz4 $POOLNAME
cannot set property for 'testpool': pool and or dataset must be upgraded to set this property or value
root@debian-8-zfs:~# #
root@debian-8-zfs:~# zpool destroy $POOLNAME
root@debian-8-zfs:~# zpool labelclear $TMPDIR/zpool.dat
root@debian-8-zfs:~# zpool create -f $POOLNAME $TMPDIR/zpool.dat -o feature@lz4_compress=enabled
root@debian-8-zfs:~# zfs set compression=lz4 $POOLNAME
root@debian-8-zfs:~# zpool get feature@lz4_compress
NAME PROPERTY VALUE SOURCE
testpool feature@lz4_compress active local
root@debian-8-zfs:~# zfs get compression $POOLNAME
NAME PROPERTY VALUE SOURCE
testpool compression lz4 local
root@debian-8-zfs:~#
Are "disabled features" still broken now?
loli10K
on 16 Mar 2017
Well, not "all" features. I'm not suggesting the whole feature activation system is broken. Just that there's something fishy inside the async_destroy feature specifically that's not working properly.
And just for the sake of putting a nail into the coffin, for reference the freeing property was first introduced in commit 9ae529ec5dbdc828ff8326beae58062971d74b2e (or illumos/illumos-gate@ad135b5d644628e791c3188a6ecbd9c257961ef8 if you prefer). So, yes, I am calling that the zfs destroy command should be activating the feature any time a snapshot is destroyed as long as freeing becomes non-zero.
DeHackEd
on 16 Mar 2017
And just for the sake of putting a nail into the coffin, for reference the freeing property was first introduced in commit 9ae529e (or illumos/illumos-gate@ad135b5 if you prefer).
So, yes, I am calling that the zfs destroy command should be activating the feature any time a snapshot is destroyed as long as freeing becomes non-zero.
I just resurrected the first Joyent's smartos image to get illumos/illumos-gate@ad135b5:
Connected to domain smartos
Escape character is ^]
_____
____ ____
_____ _____ . .
__ __ | .-. . . .-. :--. |-
_____ _____ ;| || |(.-' | | |
____ ____ `--' `-' `;-| `-' ' ' `-'
_____ / ; Joyent Live Image v0.147+
`-' build: 20120809T221258Z
52-54-00-d3-7a-f3 ttya login: root
Password:
Last login: Thu Mar 16 18:00:06 on term/a
- SmartOS Live Image v0.147+ build: 20120809T221258Z
[root@52-54-00-d3-7a-f3 ~]# zpool get feature@async_destroy testpool
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
[root@52-54-00-d3-7a-f3 ~]# zfs list -t snapshot -r testpool/fs
NAME USED AVAIL REFER MOUNTPOINT
testpool/fs@snap1 100M - 100M -
testpool/fs@snap2 100M - 100M -
testpool/fs@snap3 100M - 100M -
testpool/fs@snap4 100M - 100M -
testpool/fs@snap5 100M - 100M -
testpool/fs@snap6 100M - 100M -
testpool/fs@snap7 100M - 100M -
testpool/fs@snap8 100M - 100M -
testpool/fs@snap9 100M - 100M -
testpool/fs@snap10 0 - 100M -
[root@52-54-00-d3-7a-f3 ~]# zfs destroy testpool/fs@snap3%snap7 && while true;
do zpool get feature@async_destroy,freeing testpool; sleep 1; done
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 400M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 400M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 400M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 400M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 300M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 100M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 0 default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 0 default
^C
[root@52-54-00-d3-7a-f3 ~]# zfs destroy -r testpool/fs && while true; do zpool
get feature@async_destroy,freeing testpool; sleep 1; done
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 100M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 100M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 100M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 100M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 100M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 90.1M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 70.1M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 50.1M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy active local
testpool freeing 20.0M default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 0 default
NAME PROPERTY VALUE SOURCE
testpool feature@async_destroy enabled local
testpool freeing 0 default
^C
[root@52-54-00-d3-7a-f3 ~]#
So, if you are right i guess illumos/illumos-gate@ad135b5 has never worked as intended? That is, depending on your definition of "working" and intentions.
We should probably just ask the original developers ...
loli10K
on 16 Mar 2017
I can confirm that I have destroyed multiple filesystems (and innumerable snapshots) in my pool but feature@async_destroy is still listed as "enabled" but not "active." In contrast, other features like "large_blocks" are listed as active.
jwittlincohen
on 14 Apr 2017
I can confirm that I have destroyed multiple filesystems (and innumerable snapshots) in my pool but feature@async_destroy is still listed as "enabled" but not "active."
@jwittlincohen both "filesystems" and "snapshots"? The issue here is only about snapshots and i think we already established this behaviour has been consistent since its first implementation in https://github.com/illumos/illumos-gate/commit/ad135b5.
On the other hand destroying a filesystem should activate the feature: can you double-check this?
loli10K
on 14 Apr 2017
It does not appear to activate the feature. Here is the zpool get all and zpool history output for two of my pools:
root@storage-server:/home/jason# zpool get all data
NAME PROPERTY VALUE SOURCE
data size 87.2T -
data capacity 53% -
data altroot - default
data health ONLINE -
data guid 15746597605945671069 default
data version - default
data bootfs - default
data delegation on default
data autoreplace off default
data cachefile - default
data failmode wait default
data listsnapshots off default
data autoexpand off default
data dedupditto 0 default
data dedupratio 1.00x -
data free 40.9T -
data allocated 46.4T -
data readonly off -
data ashift 12 local
data comment - default
data expandsize - -
data freeing 0 default
data fragmentation 12% -
data leaked 0 default
data feature@async_destroy enabled local
data feature@empty_bpobj active local
data feature@lz4_compress active local
data feature@spacemap_histogram active local
data feature@enabled_txg active local
data feature@hole_birth active local
data feature@extensible_dataset active local
data feature@embedded_data active local
data feature@bookmarks enabled local
data feature@filesystem_limits enabled local
data feature@large_blocks active local
Relevant zpool history output:
2017-01-30.15:03:19 zfs destroy data/test
2017-01-30.15:35:31 zfs destroy data/test3
2017-01-30.15:50:59 zfs destroy data/Movies2
2017-01-30.16:22:34 zfs destroy data/test
2017-02-20.00:31:59 zfs destroy data/vol
2017-02-20.00:32:04 zfs destroy data/vol2
2017-03-17.11:38:56 zfs destroy data/Software
2017-04-06.23:58:38 zfs destroy -r data/documents_old
jason@backup-server:~$ sudo zpool get all data-backups
NAME PROPERTY VALUE SOURCE
data-backups size 16.2T -
data-backups capacity 70% -
data-backups altroot - default
data-backups health ONLINE -
data-backups guid 6838505270952418377 default
data-backups version - default
data-backups bootfs - default
data-backups delegation on default
data-backups autoreplace off default
data-backups cachefile - default
data-backups failmode wait default
data-backups listsnapshots off default
data-backups autoexpand off default
data-backups dedupditto 0 default
data-backups dedupratio 1.00x -
data-backups free 4.79T -
data-backups allocated 11.5T -
data-backups readonly off -
data-backups ashift 12 local
data-backups comment - default
data-backups expandsize - -
data-backups freeing 0 default
data-backups fragmentation 42% -
data-backups leaked 0 default
data-backups feature@async_destroy enabled local
data-backups feature@empty_bpobj active local
data-backups feature@lz4_compress active local
data-backups feature@spacemap_histogram active local
data-backups feature@enabled_txg active local
data-backups feature@hole_birth active local
data-backups feature@extensible_dataset active local
data-backups feature@embedded_data active local
data-backups feature@bookmarks enabled local
data-backups feature@filesystem_limits enabled local
data-backups feature@large_blocks active local
2017-03-17.11:36:41 zfs destroy data-backups/downloads
2017-03-25.09:40:03 zfs destroy data-backups/Videos
2017-04-10.18:53:38 zfs destroy data-backups/temp
Your situations as shown do not show the bug.
The argument is that feture@async_destroy == active if and only if freeing != 0. This means it's only active while the pool is reclaiming space, then it transitions back to enabled. The bug is that destroying snapshots specifically does not cause this to happen.
DeHackEd
on 14 Apr 2017
@DeHackEd
Ah, I see. Sorry for the confusion. I can confirm that it works as expected with filesystems. I captured this output while destroying a 37GB filesystem.
root@backup-server:/home/jason# zpool get all data-backups
NAME PROPERTY VALUE SOURCE
data-backups size 16.2T -
data-backups capacity 70% -
data-backups altroot - default
data-backups health ONLINE -
data-backups guid 6838505270952418377 default
data-backups version - default
data-backups bootfs - default
data-backups delegation on default
data-backups autoreplace off default
data-backups cachefile - default
data-backups failmode wait default
data-backups listsnapshots off default
data-backups autoexpand off default
data-backups dedupditto 0 default
data-backups dedupratio 1.00x -
data-backups free 4.74T -
data-backups allocated 11.5T -
data-backups readonly off -
data-backups ashift 12 local
data-backups comment - default
data-backups expandsize - -
data-backups freeing 19.7G default
data-backups fragmentation 42% -
data-backups leaked 0 default
data-backups feature@async_destroy active local
data-backups feature@empty_bpobj active local
data-backups feature@lz4_compress active local
data-backups feature@spacemap_histogram active local
data-backups feature@enabled_txg active local
data-backups feature@hole_birth active local
data-backups feature@extensible_dataset active local
data-backups feature@embedded_data active local
data-backups feature@bookmarks enabled local
data-backups feature@filesystem_limits enabled local
data-backups feature@large_blocks active local
This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.
![stale[bot] picture](https://avatars3.githubusercontent.com/in/1724?v=4&s=40)
stale[bot]
on 25 Aug 2020
Related issues
mabod
路
53Comments
wrouesnel
路
57Comments
pruiz
路
60Comments
chrisrd
路
65Comments
user318
路
51Comments
Most helpful comment
Your situations as shown do not show the bug.
The argument is that
feture@async_destroy == activeif and only iffreeing != 0. This means it's only active while the pool is reclaiming space, then it transitions back to enabled. The bug is that destroying snapshots specifically does not cause this to happen.