zfs status during scrub show unrealistic speeds
System information
Type | Version/Name
--- | ---
Distribution Name | Debian GNU/Linux,
Distribution Version | buster (stable), 10.3+
Linux Kernel | 4.19.98-1
Architecture | amd64
ZFS Version | v0.8.0-583_gdda702fd1,
Describe the problem you're observing
Pool layout:
# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
dtank2 32.5T 21.3T 11.2T - - 34% 65% 1.00x ONLINE -
# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
dextank2 32.5T 21.3T 11.2T - - 34% 65% 1.00x ONLINE -
raidz2 16.2T 10.7T 5.54T - - 34% 65.9% - ONLINE
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456701-part1 - - - - - - - - ONLINE
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456702-part1 - - - - - - - - ONLINE
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456703-part1 - - - - - - - - ONLINE
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456704-part1 - - - - - - - - ONLINE
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456705-part1 - - - - - - - - ONLINE
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456706-part1 - - - - - - - - ONLINE
raidz2 16.2T 10.6T 5.62T - - 35% 65.4% - ONLINE
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456707-part1 - - - - - - - - ONLINE
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456708-part1 - - - - - - - - ONLINE
ata-WDC_WD30EFRX-68EUZN0_WD-WCC123456709-part1 - - - - - - - - ONLINE
ata-WDC_WD30EFRX-68EUZN0_WD-WCC123456710-part1 - - - - - - - - ONLINE
ata-WDC_WD30EFRX-68EUZN0_WD-WCC123456711-part1 - - - - - - - - ONLINE
ata-WDC_WD30EFRX-68EUZN0_WD-WCC123456712-part1 - - - - - - - - ONLINE
logs - - - - - - - - -
mirror 80.5G 416K 80.5G - - 0% 0.00% - ONLINE
sda4_crypt - - - - - - - - ONLINE
sdb4_crypt - - - - - - - - ONLINE
# zfs status
...
config:
NAME STATE READ WRITE CKSUM
dtank2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456701-part1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456702-part1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456703-part1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456704-part1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456705-part1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456706-part1 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456707-part1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68AX9N0_WD-WCC123456708-part1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC123456709-part1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC123456710-part1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC123456711-part1 ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WCC123456712-part1 ONLINE 0 0 0
logs
mirror-2 ONLINE 0 0 0
sda4_crypt ONLINE 0 0 0
sdb4_crypt ONLINE 0 0 0
There is 22 datasets of varying size, few with lz4 compression. No dedup used.
When scrub is started and zpool status is inspected it shows unrealistic scrub / scan speeds:
root@d:~/zfs-git# zpool scrub dtank2
root@d:~/zfs-git# cd
root@d:~# zpool status | egrep 'scanned|scrub|repair'
scan: scrub in progress since Mon Feb 10 22:54:23 2020
25.7G scanned at 6.42G/s, 528K issued at 132K/s, 21.3T total
0B repaired, 0.00% done, no estimated completion time
root@d:~# zpool status | grep scanned
224G scanned at 31.9G/s, 528K issued at 75.4K/s, 21.3T total
root@d:~# zpool status | grep scanned
230G scanned at 25.6G/s, 528K issued at 58.7K/s, 21.3T total
root@d:~# zpool status | grep scanned
277G scanned at 27.7G/s, 528K issued at 52.8K/s, 21.3T total
root@d:~# zpool status | grep scanned
546G scanned at 42.0G/s, 528K issued at 40.6K/s, 21.3T total
root@d:~# zpool status | grep scanned
598G scanned at 46.0G/s, 528K issued at 40.6K/s, 21.3T total
root@d:~# zpool status | grep scanned
930G scanned at 51.7G/s, 540K issued at 30K/s, 21.3T total
root@d:~# zpool status | grep scanned
933G scanned at 49.1G/s, 540K issued at 28.4K/s, 21.3T total
root@d:~# zpool status | grep scanned
986G scanned at 51.9G/s, 540K issued at 28.4K/s, 21.3T total
root@d:~# zpool status | grep scanned
1.03T scanned at 52.5G/s, 540K issued at 27K/s, 21.3T total
root@d:~# zpool status | grep scanned
1.08T scanned at 52.5G/s, 552K issued at 26.3K/s, 21.3T total
root@d:~# zpool status | grep scanned
1.08T scanned at 50.1G/s, 552K issued at 25.1K/s, 21.3T total
root@d:~# zpool status | grep scanned
1.08T scanned at 48.0G/s, 552K issued at 24K/s, 21.3T total
root@d:~# zpool status | grep scanned
1.08T scanned at 48.0G/s, 552K issued at 24K/s, 21.3T total
root@d:~# zpool status | grep scanned
1.08T scanned at 46.0G/s, 552K issued at 23K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:54:51 PM CET
1.09T scanned at 39.8G/s, 552K issued at 19.7K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:54:54 PM CET
1.09T scanned at 36.0G/s, 552K issued at 17.8K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:54:58 PM CET
1.09T scanned at 32.0G/s, 552K issued at 15.8K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:55:01 PM CET
1.09T scanned at 29.5G/s, 552K issued at 14.5K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:55:02 PM CET
1.09T scanned at 28.7G/s, 552K issued at 14.2K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:55:03 PM CET
1.09T scanned at 28.0G/s, 552K issued at 13.8K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:55:04 PM CET
1.09T scanned at 27.3G/s, 552K issued at 13.5K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:55:05 PM CET
1.09T scanned at 26.7G/s, 552K issued at 13.1K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:55:16 PM CET
1.52T scanned at 29.4G/s, 552K issued at 10.4K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:55:18 PM CET
1.54T scanned at 28.6G/s, 564K issued at 10.3K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:55:19 PM CET
1.54T scanned at 28.1G/s, 564K issued at 10.1K/s, 21.3T total
root@d:~# date; zpool status | grep scanned
Mon 10 Feb 2020 10:55:20 PM CET
1.54T scanned at 27.6G/s, 564K issued at 9.89K/s, 21.3T total
Clearly this can't be done on my hardware and is bogus.
Also this machine has 31GiB of RAM.
Eventually (1-2 hours) reported scan speed goes into realistic 908MB/s. But zpool iostat -v show more like 470MB/s. So the 908MB/s is probably still total runtime average, which is skewed up due to bogus accounting of scanned data amount.
Describe how to reproduce the problem
I don't know? Have a big pool and start a scrub? I have see this problem on multiple zfs versions for some time on my system.
baryluk
All 2 comments
The "scanned" value are related to metadata scanning, and basically shows the speed at which the zpool is scanned to discover and collect metadata. This approach is used in newer ZFS releases (0.8.x) to speed up pool scrub via LBA sorting, issuing (hopefully) sequential reads for the later data scan/scrub.
You can see it as the speed of a "cursory view" of pool metadata which, by that very definition, is scanned at much faster rate than what the hardware really provides.
shodanshok
on 11 Feb 2020
👍5
Please close this out, this is expected behavior with ZFS 0.8x
kneutron
on 6 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
nwf
路
4Comments
cinterloper
路
3Comments
nh2
路
3Comments
mailinglists35
路
4Comments
kernelOfTruth
路
4Comments
Most helpful comment
The "scanned" value are related to metadata scanning, and basically shows the speed at which the zpool is scanned to discover and collect metadata. This approach is used in newer ZFS releases (0.8.x) to speed up pool scrub via LBA sorting, issuing (hopefully) sequential reads for the later data scan/scrub.
You can see it as the speed of a "cursory view" of pool metadata which, by that very definition, is scanned at much faster rate than what the hardware really provides.