Zfs: Docker build performance extremely slow due to lack of renameat2/overlayfs on ZFS

[dbavatar]

System information

Type | Version/Name
--- | ---
Distribution Name | Any
Distribution Version | Any
Linux Kernel | Any
Architecture | Any
ZFS Version | Any
SPL Version | Any

Describe the problem you're observing

This is more of a feature request than a bug. I can't find anyone actively tracking this problem, however I do know users of docker that have stopped using ZFS because of this after having stated there's something wrong with ZFS, which is unfortunate.

When using docker on a zfs dataset, the only option is docker's "zfs driver" which uses zfs dataset operations to create layered filesystems. On other filesystems such as ext4 and btrfs, the "overlay2 driver" (utilizing the kernel's overlayfs module) can be used which is extremely fast. Docker build times can go from a fraction of a second on ext4/overlay2 to several minutes on zfs/zfs.

overlay2 driver selection is currently blacklisted on zfs by docker:
https://github.com/moby/moby/blob/master/daemon/graphdriver/overlay2/overlay.go#L165

Note that btrfs added the necessary support in kernel 4.7:
https://github.com/moby/moby/blob/master/daemon/graphdriver/overlay2/overlay.go#L169

Describe how to reproduce the problem

From a performance perspective:
time the build of a Dockerfile with several layers as so:

time docker build . -t date_image

```Dockerfile

date_image Dockerfile

FROM scratch
CMD [date]
CMD [date]
CMD [date]
CMD [date]
CMD [date]

repeat date layer to magnify performance slowdown

From a functional perspective:
```c
#include <errno.h>
#include <stdio.h>
#include <unistd.h>
#include <linux/fs.h>
#include <sys/syscall.h>

int main()
{
  int ret;

  ret = syscall(SYS_renameat2, 0, "/zfs/file1", 0, "/zfs/file2", RENAME_EXCHANGE);
  fprintf(stderr, "renameat2 ret=%d errno=%d\n", ret, errno);
  return ret;
}

Repeat test for RENAME_WHITEOUT flag as well.

I believe we should be able to use docker on ZFS w/overlay2 after both RENAME_EXCHANGE and RENAME_WHITEOUT are implemented in renameat2()

[cyphar]

I've been working on this and renameat2(2) flag support is not sufficient to get overlayfs-on-ZFS working. The main blocker is that currently ZFS uses d_revalidate which means overlayfs refuses to use it as an upperdir. In order to fix it (in addition to #8667) we will need to rework a fair amount of the ported dcache-interacting code so that d_revalidate is no longer needed.

[stale[bot]]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[dbavatar]

Too bad stalebot doesn't pick up related work - Seems like @snajpa is close to getting it done.

[snajpa]

Yeah, I've finally gotten around to write the tests for ZFS rollback, needed in #9600 ... give me at least a few days for a prototype :)