Lxd: LXD fails to mount btrfs storage pool

[cryptofuture]

LXD fails to mount storage pool (btrfs subvolume)
I can browse containers under /var/lib/lxd/disks/default/containers/ without any problem.
Happened on third power loss (some lovely person washed UPS) on the test backup lxd host. Also withbtrfs subvolume list -p /
I can ensure that all containers subvolumes still here, and can mount subvolumes manually.
Obviously, since lxd can't mount pool any operation like start, copy launch etc... fails.

lxc storage list
+---------+-------------+--------+----------------------------+---------+
|  NAME   | DESCRIPTION | DRIVER |           SOURCE           | USED BY |
+---------+-------------+--------+----------------------------+---------+
| default |             | btrfs  | /var/lib/lxd/disks/default | 32      |

lvl=info msg="Done pruning expired images" t=2017-06-13T18:25:38+0000
lvl=eror msg="failed to mount BTRFS storage pool \"/var/lib/lxd/disks/default\" onto \"/var/lib/lxd/storage-pools/default\" with mountoptions \"user_subvol_rm_allowed\": block device required" t=2017-06-13T18:25:40+0000
lvl=eror msg="failed to mount BTRFS storage pool \"/var/lib/lxd/disks/default\" onto \"/var/lib/lxd/storage-pools/default\" with mountoptions \"user_subvol_rm_allowed\": block device required" t=2017-06-13T18:25:40+0000
lvl=eror msg="failed to mount BTRFS storage pool \"/var/lib/lxd/disks/default\" onto \"/var/lib/lxd/storage-pools/default\" with mountoptions \"user_subvol_rm_allowed\": block device required" t=2017-06-13T18:45:35+0000
lvl=eror msg="failed to mount BTRFS storage pool \"/var/lib/lxd/disks/default\" onto \"/var/lib/lxd/storage-pools/default\" with mountoptions \"user_subvol_rm_allowed\": block device required" t=2017-06-13T18:53:13+0000
lvl=eror msg="failed to mount BTRFS storage pool \"/var/lib/lxd/disks/default\" onto \"/var/lib/lxd/storage-pools/default\" with mountoptions \"user_subvol_rm_allowed\": block device required" t=2017-06-13T19:01:21+0000
ephemeral=false lvl=info msg="Creating container" name=test t=2017-06-13T19:11:37+0000
ephemeral=false lvl=info msg="Created container" name=test t=2017-06-13T19:11:38+0000
lvl=eror msg="failed to mount BTRFS storage pool \"/var/lib/lxd/disks/default\" onto \"/var/lib/lxd/storage-pools/default\" with mountoptions \"user_subvol_rm_allowed\": block device required" t=2017-06-13T19:11:38+0000
created=2017-06-13T19:11:36+0000 ephemeral=false lvl=info msg="Deleting container" name=test t=2017-06-13T19:11:38+0000 used=1970-01-01T00:00:00+0000
created=2017-06-13T19:11:36+0000 ephemeral=false lvl=info msg="Deleted container" name=test t=2017-06-13T19:11:38+0000 used=1970-01-01T00:00:00+0000
lvl=eror msg="failed to mount BTRFS storage pool \"/var/lib/lxd/disks/default\" onto \"/var/lib/lxd/storage-pools/default\" with mountoptions \"user_subvol_rm_allowed\": block device required" t=2017-06-13T19:15:58+0000

Ubuntu 16.04.x

lxc info:
config:
  core.trust_password: true
  images.auto_update_interval: "0"
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
api_status: stable
api_version: "1.0"
auth: trusted
public: false
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIFUzCCAzugAwIBAgIRAI6Fx/jb3qu/8MF1FmDQ7NswDQYJKoZIhvcNAQELBQAw
    NDEcMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEUMBIGA1UEAwwLcm9vdEB0
    cGxpbmswHhcNMTcwMjIyMjAzNTEwWhcNMjcwMjIwMjAzNTEwWjA0MRwwGgYDVQQK
    ExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QHRwbGluazCCAiIw
    DQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAKVUDiJwGwFM+iYoZYciqxvW6qtb
    KnLZwn93NFWMjAOSePrQFSn/X8lhIe1ezSIUxM6AObBJ/1/EiDCQ71BwWiVdPFxs
    mwH93nD1+UZ9HTSMhfHhhhrUKhn98DtmgT2CoZhKuVp1Awi+y2ZfnaOPGQslHgUc
    CAmakSoZhsDa6Yk5pSfJBRBPCrRhezyfbIVqKrCL+xB0Ii4lrJwwodTYcgHO3/t7
    zhCDhMXk5deEstVhfiWbGV/QGJFZRgUMJKpQVnrTt4vWrzmQTvTc37IRf8WRs4tX
    aIZziu1tHlHnaHFvPMIR95jA3T2v2pfddcBbaOU3IGkKvvIyEv22+A0bdKu70t4y
    PoZ97ygcE/rD9mLfgkzjTQochJzsI2MtXwJzVXAx3lsFee51+xDLwWi0j5pjbAnG
    gcWmrCNTBP6dJ5UnlqhACq8MjnBeEmNJJuuiVyF/hu1IfVSdCdbcKrSDWhUCgD59
    B9jy0GrWs7BuAkwqKD0f6XNC/R6UukV+r24wtDIAjat3aGzPMnOuT3IEUO5S2fqa
    aapT33AbgjnR+jB9qcfvYBm9MhDJwcy9Znj0u6PDsKxrheGUFjvhtER5UqDxsQA1
    BfsmI6gDcXrVr7/B4890x+QhNvVtkkx+iO2i0gkX4TBnKlzWjqNeURRRbGkWr8fh
    M7PSoUpJyvKxfuRRAgMBAAGjYDBeMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAK
    BggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMCkGA1UdEQQiMCCCBnRwbGlua4cEwKgB
    eYcQIAEEcHdpAAACHXL//ile5zANBgkqhkiG9w0BAQsFAAOCAgEANnmc3xZ4Rp8D
    g693jdJ10VIKWuAlD9R7KEey5nVsHMYzDxaf+hmvxfTDIl9cca4P+bo9f/SQPUXl
    gSKbrPCt2mMSNC29Z0Udex33U+AD2fbwlY/vOlFMf2mJGDXletx48OXmofWWSo61
    57zUuJk4bJU3jr004FwKLEogFyFL2IgRIvxSxYCOcgZoU4j8U2iPN7P9KWMC9Wc9
    dBMi6r47BW7cBmEbquetU2vnuLEuJfwlAXJt4saBsctC2amPubTmvPWw4EJV5FkY
    ZiShglEJABKlXxKVhtmVkPFsTEckvXDTldeSP7Z6HEm9yNPUuptn3YkbxGAPVwoZ
    Gad2S6GzC4GQZQhEhg3mJ+HxYa337XeR5ZPFIar2O0TE34KPEMOZq5nyJLDOVXzG
    KWG0CSQCMmFNaONfpW63pQ7QhNb9dTFXDPRNs1j3bjDu0WKLrQ1x2SO700qGoRUK
    FrMT/GTxEcjGRlII2NRt6v5gnCZnf2vTJqjWr97poNB9Ms5uf58mKokHQAgIIjRj
    DO0im+4GB0v6QDAUNeVMaxvdajOyEnphQFBB4m5lPj2nSVrAbKJwJJXQfW6SrVDN
    a088n5n5MBQPmIawbGSo7oDimkYMGUqVDIBO/A2rJp3VZ2+S0n14Q3l6CF01O51A
    ZnevdiWoVuyz5K8iQ4lrlDHEoNbX+u8=
    -----END CERTIFICATE-----
  certificate_fingerprint: b9bdb309fc302561ad0e9fcb4ef76f07625401eaa45f7cdcfed4348cb5b86cc4
  driver: lxc
  driver_version: 2.0.8
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.4.0-80-generic
  server: lxd
  server_pid: 941
  server_version: "2.14"
  storage: btrfs
  storage_version: "4.4"

[stgraber]

Can you paste "lxc storage show default"?

[cryptofuture]

description: ""
config:
  source: /var/lib/lxd/disks/default
  volume.size: "0"
name: default
driver: btrfs
used_by:
- /1.0/containers/con1
- /1.0/containers/con3
- /1.0/containers/con6
- /1.0/containers/con7
- /1.0/containers/somehost-somecontainername-2016-08-22
- /1.0/containers/somehost-somecontainername-2017-01-09
- /1.0/containers/somehost-somecontainername-2016-08-22
- /1.0/containers/somehost-somecontainername-2017-01-09
- /1.0/containers/somehost-somecontainername-2017-01-09
- /1.0/containers/somehost-somecontainername-2016-08-21
- /1.0/containers/somehost-somecontainername-2016-09-26
- /1.0/containers/somehost-somecontainername-2017-01-09
- /1.0/containers/somehost-somecontainername-2016-08-21
- /1.0/containers/somehost-somecontainername-2016-09-26
- /1.0/containers/somehost-somecontainername-2017-01-09
- /1.0/containers/somehost-somecontainername-2016-08-21
- /1.0/containers/somehost-somecontainername-2016-09-26
- /1.0/containers/somehost-somecontainername-2017-01-09
- /1.0/containers/somehost-somecontainername-2016-08-20
- /1.0/containers/somehost-somecontainername-2017-01-09
- /1.0/containers/somehost-somecontainername-2016-08-20
- /1.0/containers/somehost-somecontainername-2016-09-26
- /1.0/containers/somehost-somecontainername-2017-01-09
- /1.0/containers/somehost-somecontainername-2016-08-20
- /1.0/containers/somehost-somecontainername-2016-09-27
- /1.0/containers/somehost-somecontainername-2017-01-09
- /1.0/containers/somehost-somecontainername-2016-08-21
- /1.0/containers/somehost-somecontainername-2016-09-26
- /1.0/containers/somehost-somecontainername-2017-01-09
- /1.0/profiles/default
- /1.0/profiles/limit
- /1.0/profiles/local

[stgraber]

Reproduced the issue, working on a fix.

[stgraber]

Testing a fix now

[stgraber]

Looks like that's working

[cryptofuture]

Thanks, thats a god speed. I think I need more than 10 minutes to build and to test it.

[stgraber]

I've built a LXD binary that's identical to the LXD 2.14 in Ubuntu 16.04 but with this fix added to it.
To install it, do the following as root:

• systemctl stop lxd.socket lxd.service
• mv /usr/bin/lxd ~/lxd.bak
• curl https://dl.stgraber.org/lxd-3409 > /usr/bin/lxd
• sha256sum /usr/bin/lxd (and confirm it's: 11f35551ed2dab478d83684ff5e80da834254c6efedba35f98f96fdd424eb2f7)
• chmod 755 /usr/bin/lxd
• systemctl start lxd.socket lxd.service

[cryptofuture]

Tested. It works! Pretty sure that was really funny to reproduce.

[stgraber]

Perfect, waiting for Jenkins to confirm that my fix didn't regress anything, then I'll let @brauner merge it and finally will push updated packages to Ubuntu to fix anyone else who'd have run into this.

[stgraber]

keeping the issue open, will be closed when the fix hits master