Zfs: After the upgrade to 0.8.2 z_rd_int and z_wr_iss consume high amounts of CPU

[Cogitri]

System information

Type | Version/Name
--- | ---
Distribution Name | Alpine Linux
Distribution Version | Edge
Linux Kernel | 4.19.75
Architecture | x86_64
ZFS Version | 0.8.2
SPL Version | 0.8.2

Describe the problem you're observing

After the upgrade I've noticed checking top I see z_rd_int and z_wr_iss taking up a lot of CPU.

Describe how to reproduce the problem

Use the system with ZFS 0.8.2. Using https://github.com/alpinelinux/aports/pull/11704. It might be due to me having many subvolumes, I have some 1020 subvolumes due to Docker. I'm using ZFS native encryption.

[gedia]

I think I may be experiencing this too. Also using native encryption with 0.8.2 (on Gentoo), but I don't have that many subvolumes.

[chymian]

same here, native encryption on a new zfs-pool (~50 subvols) @ nvme, debian buster/proxmox-kernel 5.0.21-2-pve, zfs 0.8.2
on my "old" sata-ssd with luks-encryption underneath, I don't see the issues.

@kpande In the future, if you have encryption related issues feel free to @ me. I don't monitor all of the issues that come through the Github page myself, and sometimes issues slip through the cracks and I don't get notified.

[behlendorf]

Would it be possible to run perf top on system when it's using a lot of CPU to identify the functions consuming the most CPU.

[chymian]

@behlendorf, sure.
I copied a few ISOs from SATA-zfs-pool (w/o enc) to the new nvme-zfs-pool (with native-encryption).
while htop shows nearly 100% load on all 8 cpus (4/8)

perf top shows:

  40,13%  [kernel]                          [k] aes_generic_encrypt
  27,48%  [kernel]                          [k] gcm_pclmulqdq_mul
   3,93%  [kernel]                          [k] gcm_mode_encrypt_contiguous_blocks

cpu-type:

Model name:          Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz
Stepping:            12
CPU MHz:             3629.123
CPU max MHz:         3900,0000

let me now if you need anything else.

a screenshot is avail. here

[chymian]

in addition, during read operations, copying the same ISOs in reverse direction:

  34,77%  [kernel]                           [k] aes_generic_encrypt
  23,38%  [kernel]                           [k] gcm_pclmulqdq_mul
   2,89%  [kernel]                           [k] gcm_mul_pclmulqdq
   2,54%  [kernel]                           [k] aes_encrypt_block
   1,83%  [kernel]                           [k] gcm_decrypt_final
   1,83%  [kernel]                           [k] aes_xor_block
   1,73%  [kernel]                           [k] lz4_compress_zfs
   1,36%  [kernel]                           [k] update_blocked_averages
   1,28%  [kernel]                           [k] memcpy_erms
   0,83%  [kernel]                           [k] fletcher_4_avx2_native

[behlendorf]

@chymian thanks. It looks like you're bottle necked on the non-accelerated encryption algorithms. This issue is expected to be resolved in the 0.8.3 release with the backported patch in #9515.

[chymian]

@behlendorf, thanks for answer.
what exactly means 'non-accelerated encryption algorithms'? Hardware-supported encryption?
guess: since it's aes, which is also used on my SATA-SSD which has zfs on top of luks, where I don't see any higher CPU-load on r/w. has it to to with the much higher throughput from the NVME-SSD?

TIA

[lord-carlos]

@chymian LUKS has hardware accelerated encryption. Even less powerfull CPU can encrypt gigabits/s that way. Run cryptsetup benchmark to see speeds.
Because of some changes in the linux Kernel, ZoL does not have hardware acceleration at the moment, but 0.8.3 will have it again. Your current setup will use software to de/encrypt your data.

This is my first week using ZFS, I do hope I got it right :)

[chymian]

@lord-carlos thx for the explantion - understood.
waiting for the patch hitting proxmox/debian & 0.8.3

[stale[bot]]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.