Zfs: Support for io_uring API?

[Harvie]

I've noticed this new io_uring buzz going on, as it was released in Linux 5.1. They claim "up to two times performance gain on CEPH". It should improve multiqueue scalability for NVMe drives. Also the new API is supposed to be easier to use than currently used AIO API. So naturaly as enthusiastic ZFS user, i can't do better but ask what are the consequences for ZFS on Linux.

Is it likely that we will get some performance boost from this?
Is this hard to implement at ZoL side?

• https://kernelnewbies.org/Linux_5.1#High-performance_asynchronous_I.2FO_with_io_uring
• https://lwn.net/Articles/776703/

[behlendorf]

This is definitely an interface we'll want to investigate supporting.

[Harvie]

Update:
https://www.phoronix.com/scan.php?page=news_item&px=IO_uring-755x-Performance-Fix
https://www.phoronix.com/scan.php?page=news_item&px=Linux-io_uring-Fast-Efficient

[JonKohler]

Stumbled upon this today, has anyone given this a poke?

A few interesting thoughts - now that 5.4 long term is out and cooking, more people may feel comfortable getting on to a long term version of linux that has io_uring. Secondarily, Ubuntu 20.04 said that they're standardizing around 5.4.

The "buzz" @Harvie mentioned seems to still be buzzing

[Harvie]

I think i still do not fully understand how io_uring works... Initialy i was thinking that io_uring would lay between ZFS and harddrive, but now somebody told me it is gonna lay between ZFS and userspace application, so i am bit confused. Do you know more about how this actualy works?

[maxximino]

Is this actually relevant for ZFS? AFAIU io_uring is an interface between userspace applications and the kernel.
ZFS data paths between userspace and the kernel that I'm aware of are:

• Filesystem IO - that's an interface managed by the kernel, and I can't spot in kernel 5.7 fs/ dir any filesystem-specific io_uring code.
  Looking deeper, https://github.com/torvalds/linux/blob/1ee08de1e234d95b5b4f866878b72fceb5372904/fs/io_uring.c#L2607 should call
  https://github.com/openzfs/zfs/blob/657ce253575295ef680eb33cf4ef698548212a46/module/os/linux/zfs/zpl_file.c#L289-L302 (and it has a fallback in case we didn't implement it - so looks like doesn't expect integration work for every FS)

• Block devices/zvol IO - same as above, checked drivers/block/, no sign of anything special . Didn't follow the exact code path for block devices.
• ZFS send/receive: AFAIU that's one long-running IOCTL which dumps data directly into a file descriptor, so might not benefit from the reduced context switching of io_uring. (but this is just my speculation)

Ceph instead got huge performance gains because Ceph's ""backend"" is an userspace application which issues a lot of I/Os (see https://github.com/ceph/ceph/pull/27392). Hypotethically, if ZoL was not a native kernel module but still zfs-fuse, that would gain performance.