Lxd: Running LXD on ZFS 2.0 causes "Required tool 'zpool' is missing"

[zrav]

Required information

• Distribution: Ubuntu
• Distribution version: 20.04.1

config:
  core.https_address: '[::]:8443'
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: -
  architectures:
  - x86_64
  - i686
  certificate: -
  certificate_fingerprint: -
  driver: lxc
  driver_version: 4.0.4
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.4.0-42-generic
  lxc_features:
    cgroup2: "true"
    devpts_fd: "false"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "false"
  os_name: Ubuntu
  os_version: "20.04"
  project: default
  server: lxd
  server_clustered: false
  server_name: hostname
  server_pid: 3398
  server_version: "4.4"
  storage: zfs
  storage_version: 0.8.5-rc1 (<- this is after I downgraded ZFS)

Issue description

ZFS on Linux is gearing up for the release of version 2.0 and there's now an 2.0.0-rc1. When using this version compiled from source, LXD fails to start properly with "Required tool 'zpool' is missing". I tried following the advice from https://discuss.linuxcontainers.org/t/lxd-init-fails-to-find-zfs-tool/1597/5 , but to no effect. I didn't have time to investigate further as I had to get the server up and running again by downgrading ZFS.

Steps to reproduce

1. Set up Ubuntu server 20.04 and a ZFS pool using the provided packages
2. Set up LXD from Snap and configure to use the ZFS pool
3. Compile https://github.com/openzfs/zfs/commits/zfs-2.0-release following https://openzfs.github.io/openzfs-docs/Developer%20Resources/Custom%20Packages.html#dkms-1
   
   
   
   1. Install compiled ZFS packages, reboot system and see LXD not starting
   
   

Information to attach

Aug 26 16:10:18 hostname lxd.daemon[6070]: => Starting LXD
Aug 26 16:10:18 hostname lxd.daemon[6198]: t=2020-08-26T16:10:18+0200 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored"
Aug 26 16:10:18 hostname lxd.daemon[6198]: t=2020-08-26T16:10:18+0200 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
Aug 26 16:10:18 hostname lxd.daemon[6198]: t=2020-08-26T16:10:18+0200 lvl=eror msg="Failed to start the daemon: Failed initializing storage pool \"default\": Required tool 'zpool' is missing"
Aug 26 16:10:18 hostname lxd.daemon[6198]: Error: Failed initializing storage pool "default": Required tool 'zpool' is missing
Aug 26 16:10:19 hostname lxd.daemon[6070]: => LXD failed to start
Aug 26 16:10:19 hostname systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE

[monstermunchkin]

I think you're running into this problem because you're using the snap which ships its own zfs tools (versions 0.6, 0.7, and 0.8). Upon start, the daemon checks which zfs version you have and sets the appropriate entry in PATH. Now, since you're using version 2.0.0, and the snap doesn't, it won't be able to find any binaries.

Try building LXD from source and run it from a shell (not the snap). That should work.

[stgraber]

Correct, the LXD snap only supports stable ZFS releases, so 0.6, 0.7 and 0.8 are currently included.

[stgraber]

As building the ZFS tools is quite costly, we usually wait until the first distro includes a new ZFS version before we start bundling it in the snap. Though maybe this time around we can get rid of 0.6 as we add 2.0, we'll see.

[zrav]

I see. I understand that LXD shouldn't use the system provided tools due to snaps concept of being self contained, but isn't that also a potential source of problems, as the exact minor version of ZFS in use on the system is outside of snaps control?

Anyway, it seems the interface is stable enough so that compiling 2.0.0-rc1 as 0.8.5 (for instance), is enough to make LXD happy.

[stgraber]

As far as ZFS is concerned, they do not break userspace <-> kernel APIs in micro releases so this isn't a problem, what matters for bugfixes is the kernel side, the userspace is independent.

However there can (and definitely were) API changes between minor releases (0.6 to 0.7 for example) so that's why we ship the each of those in the snap and select based on kernel module version.

[dhpowrhost]

@zrav Did you find a way to install lxd with zfs 2.0. I would also like to do some testing on zfs 2.0 :)

[zrav]

@dhpowrhost like I said, compile ZFS 2.0.0 as 0.8.5 by editing the META file, then following the custom package following the instructions.

[Deltik]

I made a silly workaround script that lets you use the system ZFS 2.0 version number with the currently installed lxd snap:

#!/bin/bash
WORKDIR="$(mktemp -d)"
SNAP_DAEMON_SERVICE_NAME="snap.lxd.daemon.service"
PATCHED_SQUASHFS_FILENAME="hacked_lxd.squashfs"
cleanup() {
        rm -rf "$WORKDIR"
}
trap cleanup EXIT

read -r -d '' PATCH <<EOF
diff --git a/squashfs-root/commands/daemon.start b/squashfs-root/commands/daemon.start
index d3f75ba..0b3bd83 100755
--- a/squashfs-root/commands/daemon.start
+++ b/squashfs-root/commands/daemon.start
@@ -294,6 +294,9 @@ echo "==> Rotating logs"
 logrotate -f "\${SNAP}/etc/logrotate.conf" -s "/etc/logrotate.status"

 # Setup for ZFS
+echo "==> Setting up ZFS (0.8 override for https://github.com/lxc/lxd/issues/7815/)"
+export LD_LIBRARY_PATH="\${SNAP_CURRENT}/zfs-0.8/lib/:\${LD_LIBRARY_PATH}"
+export PATH="\${SNAP_CURRENT}/zfs-0.8/bin:\${PATH}"
 if [ -e /sys/module/zfs/version ]; then
     VERSION=\$(cat /sys/module/zfs/version)
 else
EOF

snap_lxd_mount="$(systemctl show "$SNAP_DAEMON_SERVICE_NAME" --property=Requires --value | xargs -n1 | grep -m 1 '\.mount$')"
snap_lxd_loop="$(systemctl show "$snap_lxd_mount" --property=What --value)"
snap_lxd_squashfs="$(losetup --noheadings --output BACK-FILE "$snap_lxd_loop")"
if [ $? -ne 0 ]; then
        snap_lxd_squashfs="$snap_lxd_loop"
fi

systemctl stop "$snap_lxd_mount"

cd "$WORKDIR"
unsquashfs "$snap_lxd_squashfs"
echo "$PATCH" | patch -p1
mksquashfs squashfs-root "$PATCHED_SQUASHFS_FILENAME" -b 1024k -comp xz -Xbcj x86
mv -v "$PATCHED_SQUASHFS_FILENAME" "$snap_lxd_squashfs"

systemctl restart "$SNAP_DAEMON_SERVICE_NAME"

This script will restart the lxd snap. It forces the lxd snap daemon to use the ZFS 0.8 binaries unless an older version is installed, in which case that version will be used instead.

To roll back, just run the script again and specify y when prompted.

[rajil]

My Archlinux host received ZFS 2 and LXD refuses to start. Is there any fix for this?

[stgraber]

Just pushed basic support to the edge snap, if that builds properly, I'll push it to candidate/stable next.

[deviantintegral]

The commonly used jonathonf/zfs PPA has also pushed updates to ZFS 2.0: https://launchpad.net/~jonathonf/+archive/ubuntu/zfs

@Deltik thanks for the script, it got my lxc containers back up and running. There wasn't any prompt to apply or remove the patch, and I'm not seeing anything that looks like it'd do that. However, I think that the easiest way to revert back will be to run sudo snap refresh lxd? If that's the case, I guess anyone using this will have to watch for lxd updates to the snap channel that _don't_ contain the ZFS 2 updates.

[stgraber]

Since edge appears to be behaving, I'd expect the next push to stable will include it too.

[rajil]

@stgraber Is LXD 4.9 the latest release with ZFS-2.0 support?

[stgraber]

4.9 definitely has the snap logic for it but I thought that the 4.8 currently in stable got it too