Lxd: Sort disk entries by target path (mount parent first)

[naisanza]

Required information

• Distribution: Ubuntu
• Distribution version: 16.04
• The output of "lxc info" or if that fails:
  
  
  
  • Kernel version: 4.4.0-31-generic
  
  
  • LXC version: 2.0.3
  
  
  • LXD version: 2.0.3
  
  
  • Storage backend in use: ZFS
    
    

    Issue description

  
  

The container will not start with added disk devices.

Sample error:
lxc 20160729063411.372 ERROR lxc_utils - utils.c:mkdir_p:253 - Permission denied - failed to create directory '/usr/lib/x86_64-linux-gnu/lxc/turtle/_data/downloads'

I'm adding ZFS datasets as the source for lxc device add. However, the container will not start.

But, if I start the container without any added devices, and add the devices _after_ the container is running, everything works.

Steps to reproduce that works

1. $ lxc launch f452cda3bccb outer-smb -p outer_rim (container will be running)

2. $ lxc config device add outer-smb data disk source=/turtle/_data path=/turtle/_data etc. etc.

   Steps to reproduce that doesn't work (method one)

3. $ lxc launch f452cda3bccb outer-smb -p outer_rim (container will be running)

4. $ lxc stop outer-smb
5. $ lxc config device add outer-smb data disk source=/turtle/_data path=/turtle/_data etc. etc.
6. $ lxc start outer-smb

7. container fails to start (see attached outer-smb.log)

   Steps to reproduce that doesn't work (method two)

8. $ lxc launch f452cda3bccb outer-smb -p outer_rim (container will be running)

9. $ lxc config device add outer-smb data disk source=/turtle/_data path=/turtle/_data etc. etc.
10. $ lxc restart outer-smb
11. container fails to start (see attached outer-smb.log)

ZFS datasets have the following extended attribute settings: xattr=sa, acltype=posixacl
I've also used setfacl to set all the datasets with identical permissions

Information to attach

• [x] any relevant kernel output (dmesg) none relevant
• [x] container log (lxc info NAME --show-log) lxc-info.log.txt
• [x] main daemon log (/var/log/lxd.log) lxd.log.txt
• [x] output of the client with --debug outer-smb.log.txt
• [ ] output of the daemon with --debug

[stgraber]

Can you include "lxc config show outer-smb --expanded" from after you added the device?

[naisanza]

@stgraber posting mobily, apologies for not just an uploaded file. Here's the output

name: outer-smb
profiles:
- outer_rim
config:
  volatile.base_image: f452cda3bccb2903e56d53e402b9d35334b4276783d098a879be5d74b04e62e2
  volatile.eth0.hwaddr: 00:16:3e:66:54:93
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
devices:
  data:
    path: /turtle/_data
    source: /turtle/_data
    type: disk
  downloads:
    path: /turtle/_data/downloads
    source: /turtle/_data/downloads
    type: disk
  downloads-books:
    path: /turtle/_data/downloads/books
    source: /turtle/_data/downloads/books
    type: disk
  downloads-space:
    path: /turtle/_data/downloads/space
    source: /turtle/_data/downloads/space
    type: disk
  downloads-unsorted:
    path: /turtle/_data/downloads/unsorted
    source: /turtle/_data/downloads/unsorted
    type: disk
  eth0:
    name: eth0
    nictype: bridged
    parent: outer_rim
    type: nic
  home:
    path: /turtle/_data/home
    source: /turtle/_data/home
    type: disk
  home-eric:
    path: /turtle/_data/home/eric
    source: /turtle/_data/home/eric
    type: disk
  recovery:
    path: /turtle/_data/recovery
    source: /turtle/_data/recovery
    type: disk
  root:
    path: /
    type: disk
  util:
    path: /turtle/_data/util
    source: /turtle/_data/util
    type: disk
ephemeral: false

[stgraber]

So my best guess is that LXD is attempting to mount downloads before data is mounted. We need to make our mount logic a bit more clever to sort the target mount points before asking for lxc to mount them.

[naisanza]

@stgraber I hoped I was just doing something wrong

[naisanza]

@stgraber also, another thing that happened was, when you add a device to a container, if the source is a ZFS parent dataset with nested datasets, only the parent dataset is accessible and not any of the nested datasets. Which led me to mounting each individual dataset.

I believe your assumption is correct. With each disk device mounted to a separate path or name, the container reboots without issue

[stgraber]

For the case of a mount having sub-mounts. Did you set the "recursive" property to true for your disk entry? That would cause LXD to do a rbind rather than a regular bind mount, which at least for other filesystems does mean getting the whole tree.

[naisanza]

@stgraber That. Worked. Awesome

The manpage for lxc is pretty scarce, so I guessed most of my way through using it. It would be great if https://github.com/lxc/lxd/blob/master/doc/configuration.md was used as man lxc or man lxd, since it's very thorough, and I always try to refer to the manual if a command doesn't work

[stgraber]

Cool. Good to hear that recursive works. Keeping this bug entry open as we should certainly be smarter about ordering when applying disk entries.