Required information
- Distribution: Ubuntu Server
- Distribution version: 16.04.3 (fully updated)
- The output of "lxc info" or if that fails:
config:
core.https_address: '[::]:8443'
core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
api_status: stable
api_version: "1.0"
auth: trusted
public: false
environment:
addresses:
- 172.0.0.0:8443
- 172.0.0.0:8443
architectures:
- x86_64
- i686
driver: lxc
driver_version: 2.0.8
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.10.0-28-generic
server: lxd
server_pid: 11456
server_version: "2.17"
storage: ceph
storage_version: ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4)
luminous (rc)
````
# Issue description
I am unable to provision a container -- I've verified that using the manual process for creating images / mounting / formatting works.
```lxc launch ubuntu:16.04
Creating the container
error: Failed container creation:
- https://cloud-images.ubuntu.com/releases: Failed to run: mkfs.ext4 -E nodiscard,lazy_itable_init=0,lazy_journal_init=0
/dev/rbd1: mke2fs 1.42.13 (17-May-2015)
Could not open
/dev/rbd1: File name too long
````
I've also tried it with a different image
```lxc launch ubuntu:17.04
Creating the container
error: Failed container creation:
- https://cloud-images.ubuntu.com/releases: Failed to run: mkfs.ext4 -E nodiscard,lazy_itable_init=0,lazy_journal_init=0
/dev/rbd1: mke2fs 1.42.13 (17-May-2015)
Could not open
/dev/rbd1: File name too long
Steps to reproduce
- use lxd init to create a ceph storage pool
- issue lxc launch ubuntu:16.04 using the newly created storage pool as its default storage
notes
I have been using LXD for quite a while now! Thanks for all the work!
grsmith-projects
All 53 comments
Right, so this seems to be a limitation of ceph not of LXD. To quote:
Starting with the Jewel release, the ceph-osd daemon will refuse to start if the configured max object name cannot be safely stored on ext4. If the cluster is only being used with short object names (e.g., RBD only), you can continue using ext4 by setting the following configuration option:
osd max object name len = 256
osd max object namespace len = 64
Maybe we should consider making xfs instead of ext4 the default.
brauner
on 28 Aug 2017
Related issue is here http://tracker.ceph.com/issues/16187 .
brauner
on 28 Aug 2017
You can also retry by setting
lxc storage set <pool-name> volume.block.filesystem xfs
Yeah, I was able to use ext4 manually (I already had those settings in the config) -- however when I tried to provision through LXD it did not work properly.
as well --
lxc storage set lxd volume.block.filesystem xfs
error: the key size cannot be used with CEPH storage pools
@Mantwon, thanks that looks like a bug. Let me take a look real quick!
brauner
on 28 Aug 2017
@Mantwon, is it possible for you to destroy the osd pool and create a new ceph storage pool via:
lxc storage create lxd ceph volume.block.filesystem=xfs
and report back if that works? If not it's fine as well.
brauner
on 28 Aug 2017
This is very weird, I would have though that would work!
```Creating the container
error: Failed container creation:
- https://cloud-images.ubuntu.com/releases: Failed to run: mkfs.xfs
/dev/rbd1: File name too long
Usage: mkfs.xfs
/* blocksize / [-b log=n|size=num]
/ metadata / [-m crc=0|1,finobt=0|1,uuid=xxx]
/ data subvol / [-d agcount=n,agsize=n,file,name=xxx,size=num,
(sunit=value,swidth=value|su=num,sw=num|noalign),
sectlog=n|sectsize=num
/ force overwrite / [-f]
/ inode size / [-i log=n|perblock=n|size=num,maxpct=n,attr=0|1|2,
projid32bit=0|1,sparse=0|1]
/ no discard / [-K]
/ log subvol / [-l agnum=n,internal,size=num,logdev=xxx,version=n
sunit=value|su=num,sectlog=n|sectsize=num,
lazy-count=0|1]
/ label / [-L label (maximum 12 characters)]
/ naming / [-n log=n|size=num,version=2|ci,ftype=0|1]
/ no-op info only / [-N]
/ prototype file / [-p fname]
/ quiet / [-q]
/ realtime subvol / [-r extsize=num,size=num,rtdev=xxx]
/ sectorsize / [-s log=n|size=num]
/ version */ [-V]
devicename
is required unless -d name=xxx is given.
is xxx (bytes), xxxs (sectors), xxxb (fs blocks), xxxk (xxx KiB),
xxxm (xxx MiB), xxxg (xxx GiB), xxxt (xxx TiB) or xxxp (xxx PiB).
is xxx (512 byte blocks).
```
grsmith-projects
on 28 Aug 2017
I just tried with current git master and I don't have any problems with creating ext4 or xfs based osd storage pools. The difference is that I'm on the Jewel (10) release of CEPH so the problems you're describing might really just kick in with later releases. Can you please run your daemon with
sudo lxd --debug --group lxd --logfile /some/path/to/a/logfile.log
try to create a new container and append logfile.log here. I've given the ceph driver quite some detailed logging so I'd like to see what exactly is going on.
As a side note:
lxc storage set lxd volume.block.filesystem xfs
error: the key size cannot be used with CEPH storage pools
is very confusing as it is unclear from the code why LXD would report size property not allowed when you sent it volume.block.filesystem. I was also unable to reproduce this step.
brauner
on 28 Aug 2017
I have a suspicion of what is going on. So what LXD is doing before it creates a container it will create an image. The image data set name will be:
<osd-pool-name>/images_<fingerprint>
I have no clear idea how ceph stores these names internally and what it additionally adds to the name itself in terms of metadata but he combination of strlen(images_<fingerprint>) + ceph_metadata might exceed the allowed limit on ext4 and xfs instances unless ceph is instructed to adhere to a certain limit as I outlined above.
brauner
on 28 Aug 2017
INFO[08-28|04:17:16] Creating container ephemeral=false name=selected-gnu
DBUG[08-28|04:17:17] Initializing a CEPH driver
DBUG[08-28|04:17:17] Checking if RBD storage volume for container "selected-gnu" on storage pool "lxd" is ready
DBUG[08-28|04:17:17] RBD storage volume for container "selected-gnu" on storage pool "lxd" does not exist
DBUG[08-28|04:17:17] Checking if RBD storage volume for container "selected-gnu" on storage pool "lxd" is ready
DBUG[08-28|04:17:17] RBD storage volume for container "selected-gnu" on storage pool "lxd" does not exist
DBUG[08-28|04:17:17] Checking if RBD storage volume for container "selected-gnu" on storage pool "lxd" is ready
DBUG[08-28|04:17:17] RBD storage volume for container "selected-gnu" on storage pool "lxd" does not exist
INFO[08-28|04:17:17] Created container ephemeral=false name=selected-gnu
DBUG[08-28|04:17:17] Creating RBD storage volume for container "selected-gnu" on storage pool "lxd"
DBUG[08-28|04:17:17] Creating RBD storage volume for image "d4b61f799ac371b36f790eaceff06d406c72729c0383c651df09fc546cd09d86" on storage pool "lxd"
DBUG[08-28|04:17:17] Created mountpoint "/var/lib/lxd/storage-pools/lxd/images/d4b61f799ac371b36f790eaceff06d406c72729c0383c651df09fc546cd09d86" for RBD storage volume for image "d4b61f799ac371b36f790eaceff06d406c72729c0383c651df09fc546cd09d86" on storage pool "lxd"
DBUG[08-28|04:17:17] RBD storage volume for image "d4b61f799ac371b36f790eaceff06d406c72729c0383c651df09fc546cd09d86" on storage pool "lxd" does not exist
DBUG[08-28|04:17:17] Retrieve size "10737418240B" of RBD storage volume for image "d4b61f799ac371b36f790eaceff06d406c72729c0383c651df09fc546cd09d86" on storage pool "lxd"
DBUG[08-28|04:17:17] Created RBD storage volume for image "d4b61f799ac371b36f790eaceff06d406c72729c0383c651df09fc546cd09d86" on storage pool "lxd"
DBUG[08-28|04:17:17] Mapped RBD storage volume for image "d4b61f799ac371b36f790eaceff06d406c72729c0383c651df09fc546cd09d86" on storage pool "lxd"
EROR[08-28|04:17:28] Failed to create filesystem "xfs" for RBD storage volume for image "d4b61f799ac371b36f790eaceff06d406c72729c0383c651df09fc546cd09d86" on storage pool "lxd":``
the rest of the error was the same one I had posted earlier.
grsmith-projects
on 28 Aug 2017
I will append: I created an image called something like "d4b61f799ac371b36f790eaceff06d406c72729c0383c651df09fc546cd09d86" manually, (I copied from one of the failed one), I mapped it, mkfs'd it, and mounted it no problem using manual process, so it accepted that fingerprint length
grsmith-projects
on 28 Aug 2017
@Mantwon, can you show:
rbd ls <pool-name>
please?
brauner
on 28 Aug 2017
rbd ls lxd
lxd_lxd
rbd image 'lxd_lxd':
size 0 bytes in 0 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.986616b8b4567
format: 2
features: layering
flags:
create_timestamp: Mon Aug 28 03:47:24 2017
grsmith-projects
on 28 Aug 2017
Ah right, I've implemented ceph sort of transactional, meaning everything should be reverted on failure. Ok, can you give me the exact steps when you manually created a dataset?
brauner
on 28 Aug 2017
root@node001:~# rbd create lxd/58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08 --size 10G --image-format=2 --image-feature=layering
root@node001:~# rbd map lxd/58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08
/dev/rbd1
root@node001:~# mkfs.xfs /dev/rbd1
meta-data=/dev/rbd1 isize=512 agcount=17, agsize=162816 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=1024 swidth=1024 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@node001:~# mount /dev/rbd1 /mnt
root@node001:~# cd /mnt
root@node001:/mnt# touch hey_this_works
root@node001:/mnt# ls
hey_this_works
root@node001:/mnt# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 61806648 0 61806648 0% /dev
tmpfs 12372356 83160 12289196 1% /run
/dev/sda1 28704764 5108376 22115224 19% /
tmpfs 61861768 0 61861768 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 61861768 0 61861768 0% /sys/fs/cgroup
/dev/sda6 91188312 57780 86475380 1% /home
tmpfs 12372356 0 12372356 0% /run/user/1000
/dev/sdb1 102400 16640 56320 23% /var/lib/ceph/osd/ceph-0
/dev/sdc1 102400 16640 56320 23% /var/lib/ceph/osd/ceph-1
/dev/sdd1 102400 16640 56320 23% /var/lib/ceph/osd/ceph-2
/dev/sde1 102400 16640 56320 23% /var/lib/ceph/osd/ceph-3
/dev/sdf1 102400 16640 56320 23% /var/lib/ceph/osd/ceph-4
/dev/sdg1 102400 16640 56320 23% /var/lib/ceph/osd/ceph-5
/dev/sdh1 102400 16640 56320 23% /var/lib/ceph/osd/ceph-6
/dev/sdi1 102400 16640 56320 23% /var/lib/ceph/osd/ceph-7
tmpfs 100 0 100 0% /var/lib/lxd/shmounts
tmpfs 100 0 100 0% /var/lib/lxd/devlxd
/dev/rbd1 10475520 44700 10430820 1% /mnt
root@node001:/mnt#
Hm, let me check if the code that parses out /dev/rbd/<idx> is doing something weird.
brauner
on 28 Aug 2017
Can you please try:
lxc storage volume create <pool-name> bla
and report back if that works?
brauner
on 28 Aug 2017
Did not work, same error
grsmith-projects
on 28 Aug 2017
Can you create + map + mkfs.\ ?lxd/images_58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08
brauner
on 28 Aug 2017
```
root@node001:/mnt# rbd create lxd/images_58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08 --size 10G --image-format=2 --image-feature=layering
root@node001:/mnt# rbd map lxd/images_58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08
/dev/rbd2
root@node001:/mnt# mkfs.xfs /dev/rbd2
meta-data=/dev/rbd2 isize=512 agcount=17, agsize=162816 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=1024 swidth=1024 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@node001:/mnt#
grsmith-projects
on 28 Aug 2017
Thanks!
brauner
on 28 Aug 2017
here is w/ a 256 char name as well:
root@node001:/mnt# rbd create lxd/YClHMl2WqrKo8EDI4YvZO3FpZj84jsDTDvNTDAODAUulZ783SgFIF2w5wNse890ILJcJVJHsQDVCrDmVR05CarzHhU6LILEKlLLkIpQFnLSvJoCRHjn7IwYHDiYJXaBgeZmdKJpZg9opKce7JttIvmU2SiOoQP0FF1d9US7oi6UIygLDb47AIDpbIcgNPExBAMQSJTuftGJNVyzlceVyoPwFGnqBzwOBBTSO5W7dwHAgK6BvT0mEFHStcTNp2C4U --size 10G --image-format=2 --image-feature=layering
root@node001:/mnt# rbd map lxd/YClHMl2WqrKo8EDI4YvZO3FpZj84jsDTDvNTDAODAUulZ783SgFIF2w5wNse890ILJcJVJHsQDVCrDmVR05CarzHhU6LILEKlLLkIpQFnLSvJoCRHjn7IwYHDiYJXaBgeZmdKJpZg9opKce7JttIvmU2SiOoQP0FF1d9US7oi6UIygLDb47AIDpbIcgNPExBAMQSJTuftGJNVyzlceVyoPwFGnqBzwOBBTSO5W7dwHAgK6BvT0mEFHStcTNp2C4U
/dev/rbd3
root@node001:/mnt# mkfs.xfs /dev/rbd3
meta-data=/dev/rbd3 isize=512 agcount=17, agsize=162816 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=1024 swidth=1024 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
The LXD daemon sits on the same node where you're calling mkfs.\<type\> from?
brauner
on 28 Aug 2017
yes, it does.
grsmith-projects
on 28 Aug 2017
I think I may have found the problem, it has to do with Bluestore + the 100MB partition, I was able to repro the error that was happening manually by:
root@node001:/var/lib/ceph/osd/ceph-0# touch aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
touch: cannot touch 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa': File name too long
root@node001:/var/lib/ceph/osd/ceph-0# touch aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
255 char vs 256 char... I wonder if there is a temp file of some sort that has a very long name writing some sort of metadata to
/var/lib/ceph/osd/ceph-0
I will continue to experiment.
grsmith-projects
on 29 Aug 2017
Yeah, I wasn't able to reproduce this with the luminous release you're using either. So I highly suspect this might be related to your storage that is somehow limited.
brauner
on 29 Aug 2017
I think I've found something...
it looks like LXD is expecting the image to be in a the ceph store, yet the default image store is set to use the local storage.
what is the LXD switch to point the image store at ceph instead of local storage?
I found this was being called
rbd --id admin --cluster ceph --image-feature layering clone lxd/image_58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08@readonly lxd/container_intense-lamprey
image_58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08 does not exist, it exists in /var/lib/lxd/images/ though.
grsmith-projects
on 30 Aug 2017
also @brauner it looks like that
'Config parsing error: the key size cannot be used with CEPH storage pools'
was caused from
'size: 15GB'
being set in the default pool config
grsmith-projects
on 30 Aug 2017
@Mantwon, I'm sorry, I don't really understand want you mean by local store and ceph store?
brauner
on 30 Aug 2017
it looks like LXD is expecting thar 'lxd/image_58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08@readonly' exists, but it doesn't.
That image only exists on in /var/lib/lxd/images
shiba@node001:~$ sudo ls /var/lib/lxd/images/
58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08
58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08.rootfs
shiba@node001:~$ sudo rbd ls lxd
lxd_lxd
so when it goes to clone the 16.04 image to create the root fs, its looking for an image in the ceph pool but it doesn't exist.
I think I need to tell lxd to save the 16.04 image as rbd://lxd/image_
grsmith-projects
on 30 Aug 2017
Ah, no what you referring to is just the tar file in the image store. This has nothing to do with the storage volumes itself. The logs clearly show that LXD tries to create the image in the rbd storage pool but fails due to the limit in the storage device that is used.
brauner
on 30 Aug 2017
We are still having a miscommunication - I was able to get this sort of working by:
rbd import --image-format 2 58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08.squashfs lxd/image_58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08
rbd snap create lxd/image_58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08@readonly
md5-83e471bfdb9db5a9e89518b07ce7320e
rbd snap protect lxd/image_58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08@readonly
md5-83e471bfdb9db5a9e89518b07ce7320e
lxc launch 58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08
I moved the image squashfs from /var/lib/lxd/images to the rbd storage -- and it seems to have sort of worked.. but failed due to missing meta data.
if you do a "rbd ls
grsmith-projects
on 30 Aug 2017
LXD internally will do:
rbd image create
mkfs.<type
rbd map
mount image
unpack + import tar contents into the image file
unmount image
unmap image
rbd snap create
rbd snap protect
rbd clone container from protected image
I ran it in debug mode, and caught it while the image still existed on rbd before it ran mkfs - I then used rbd map manually, and ran mkfs.xfs manually and it worked.
I am truly stumped to what is going on.
You can close this issue if you want, or leave it open and when I figure it out can post information.
grsmith-projects
on 30 Aug 2017
Update:
I got it to work -- by manually creating the image and importing it to RBD
INFO[08-29|18:33:12] Creating container ephemeral=false name=capital-stork
DBUG[08-29|18:33:12] Initializing a CEPH driver
DBUG[08-29|18:33:12] Checking if RBD storage volume for container "capital-stork" on storage pool "ceph" is ready
DBUG[08-29|18:33:12] RBD storage volume for container "capital-stork" on storage pool "ceph" does not exist
DBUG[08-29|18:33:12] Checking if RBD storage volume for container "capital-stork" on storage pool "ceph" is ready
DBUG[08-29|18:33:12] RBD storage volume for container "capital-stork" on storage pool "ceph" does not exist
DBUG[08-29|18:33:12] Checking if RBD storage volume for container "capital-stork" on storage pool "ceph" is ready
DBUG[08-29|18:33:12] RBD storage volume for container "capital-stork" on storage pool "ceph" does not exist
INFO[08-29|18:33:12] Created container ephemeral=false name=capital-stork
DBUG[08-29|18:33:12] Creating RBD storage volume for container "capital-stork" on storage pool "ceph"
DBUG[08-29|18:33:13] Cloned new RBD storage volume for container "capital-stork"
DBUG[08-29|18:33:13] Mapped RBD storage volume for container "capital-stork"
DBUG[08-29|18:33:13] Created mountpoint "/var/lib/lxd/storage-pools/ceph/containers/capital-stork" for container "capital-stork" for RBD storage volume
DBUG[08-29|18:33:13] Mounting RBD storage volume for container "capital-stork" on storage pool "ceph".
DBUG[08-29|18:33:14] Mounted RBD storage volume for container "capital-stork" on storage pool "ceph".
DBUG[08-29|18:33:14] Shifting root filesystem "/var/lib/lxd/containers/capital-stork/rootfs" for "capital-stork".
DBUG[08-29|18:33:22] Shifted rootfs for container "capital-stork"
DBUG[08-29|18:33:22] Changed mountpoint "/var/lib/lxd/storage-pools/ceph/containers/capital-stork" permissions to 0755 for container "capital-stork" for RBD storage volume
DBUG[08-29|18:33:22] Checking if RBD storage volume for container "capital-stork" on storage pool "ceph" is ready
DBUG[08-29|18:33:23] RBD storage volume for container "capital-stork" on storage pool "ceph" is ready
DBUG[08-29|18:33:23] Applied create template for container "capital-stork"
DBUG[08-29|18:33:23] Created RBD storage volume for container "capital-stork" on storage pool "ceph"
DBUG[08-29|18:33:23] Unmounting RBD storage volume for container "capital-stork" on storage pool "ceph".
DBUG[08-29|18:33:28] Unmounted RBD storage volume for container "capital-stork" on storage pool "ceph".
DBUG[08-29|18:33:28] Mounting RBD storage volume for container "capital-stork" on storage pool "ceph".
DBUG[08-29|18:33:29] Mounted RBD storage volume for container "capital-stork" on storage pool "ceph".
DBUG[08-29|18:33:29] Unmounting RBD storage volume for container "capital-stork" on storage pool "ceph".
DBUG[08-29|18:33:29] Unmounted RBD storage volume for container "capital-stork" on storage pool "ceph".
DBUG[08-29|18:33:29] Success for task operation: 80c2da15-9aa4-4775-99cd-a963296271d5
DBUG[08-29|18:33:29] handling ip=@ method=GET url=/1.0/containers/capital-stork
DBUG[08-29|18:33:29]
{
"type": "sync",
"status": "Success",
"status_code": 200,
"operation": "",
"error_code": 0,
"error": "",
"metadata": {
"architecture": "x86_64",
"config": {
"image.architecture": "amd64",
"image.description": "ubuntu 16.04 LTS amd64 (release) (20170815.1)",
"image.label": "release",
"image.os": "ubuntu",
"image.release": "xenial",
"image.serial": "20170815.1",
"image.version": "16.04",
"volatile.apply_template": "create",
"volatile.base_image": "58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08",
"volatile.idmap.base": "0",
"volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]",
"volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]"
},
"devices": {
"root": {
"path": "/",
"pool": "ceph",
"type": "disk"
}
},
"ephemeral": false,
"profiles": [
"default"
],
"stateful": false,
"description": "",
"created_at": "2017-08-30T01:33:12Z",
"expanded_config": {
"environment.http_proxy": "",
"image.architecture": "amd64",
"image.description": "ubuntu 16.04 LTS amd64 (release) (20170815.1)",
"image.label": "release",
"image.os": "ubuntu",
"image.release": "xenial",
"image.serial": "20170815.1",
"image.version": "16.04",
"user.network_mode": "",
"volatile.apply_template": "create",
"volatile.base_image": "58f90cbf68927c3fc43e6ee1386446a04f3d8068c1a75a291339cb2be01dec08",
"volatile.idmap.base": "0",
"volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]",
"volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]"
},
"expanded_devices": {
"root": {
"path": "/",
"pool": "ceph",
"type": "disk"
}
},
"name": "capital-stork",
"status": "Stopped",
"status_code": 102,
"last_used_at": "1970-01-01T00:00:00Z"
}
}
DBUG[08-29|18:33:29] Disconnected events listener: b0ea451b-963a-4776-9542-d643131a9b88
I went through all of the steps outlined in the code, and did them manually to just get it to the point the image exists, and the rest worked.
There appears to be a problem with my setup and the initial copy of the image over to the rbd volume (which you suspected), the rest of the process works great.
Can you see anything from that output that stands out?
grsmith-projects
on 30 Aug 2017
More:
DBUG[08-29|19:17:13] Mapped RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "ceph"
DBUG[08-29|19:17:25] Sleeping for 300
I added a sleep 300 at the part where it breaks, and then did that step manually:
root@node001:~# mkfs.xfs /dev/rbd/lxd/image_38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b
meta-data=/dev/rbd/lxd/image_38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b isize=512 agcount=17, agsize=162816 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=1024 swidth=1024 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@node001:~#
I hard coded '/dev/rbd/lxd/image_38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b' into the mkfs command, and it progressed passed that point, until it started using /dev/rbd2 as the RBDDevPath and failed again.. I believe it doesn't like using /dev/rbd(#).
Though /dev/rbd(#) works fine manually.
Would it make more sense to use /dev/rbd/
grsmith-projects
on 30 Aug 2017
We unfortunately can't use the /dev/rbd/... paths because those are dependent on a udev rule which isn't always present in all environments (like in our snap package for example).
stgraber
on 30 Aug 2017
Hm, can you try hardcoding /dev/rbd2 and see if it still fails.
brauner
on 30 Aug 2017
it does not work hardcoding /dev/rbd(x) -- but oddly, it fails silently and just says it can't run -- it doesn't give the same error.
grsmith-projects
on 30 Aug 2017
Maybe its a problem with Go?
TERMINAL 1:
INFO[08-30|00:46:42] Image downloaded alias=15.10 image=9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049 operation=265dedaf-f2b4-4d75-9321-6d138d983a62 server=https://cloud-images.ubuntu.com/releases trigger=/1.0/operations/265dedaf-f2b4-4d75-9321-6d138d983a62
INFO[08-30|00:46:42] Creating container ephemeral=false name=united-badger
DBUG[08-30|00:46:42] Initializing a CEPH driver
DBUG[08-30|00:46:42] Checking if RBD storage volume for container "united-badger" on storage pool "container_storage" is ready
DBUG[08-30|00:46:42] RBD storage volume for container "united-badger" on storage pool "container_storage" does not exist
DBUG[08-30|00:46:42] Checking if RBD storage volume for container "united-badger" on storage pool "container_storage" is ready
DBUG[08-30|00:46:42] RBD storage volume for container "united-badger" on storage pool "container_storage" does not exist
DBUG[08-30|00:46:42] Checking if RBD storage volume for container "united-badger" on storage pool "container_storage" is ready
DBUG[08-30|00:46:42] RBD storage volume for container "united-badger" on storage pool "container_storage" does not exist
INFO[08-30|00:46:42] Created container ephemeral=false name=united-badger
DBUG[08-30|00:46:42] Creating RBD storage volume for container "united-badger" on storage pool "container_storage"
DBUG[08-30|00:46:43] Creating RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:46:43] Created mountpoint "/var/lib/lxd/storage-pools/container_storage/images/9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" for RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:46:43] RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage" does not exist
DBUG[08-30|00:46:43] Retrieve size "10737418240B" of RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:46:43] Created RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:46:43] Mapped RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:46:43] PATH: /dev/rbd7
DBUG[08-30|00:46:43] You have 30 seconds to run ' mkfs.xfs /dev/rbd7 ' in a different terminal
DBUG[08-30|00:47:13] Created filesystem "xfs" for RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:47:13] Mounting RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage".
DBUG[08-30|00:47:13] Mounted RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage".
DBUG[08-30|00:47:38] Unpacked image for RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:47:38] Unmounting RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage".
DBUG[08-30|00:47:51] Unmounted RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage".
DBUG[08-30|00:47:51] Unmapped RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:47:52] Created snapshot for RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:47:52] Protected snapshot for RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:47:52] Createdd database entry for RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:47:52] Created RBD storage volume for image "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049" on storage pool "container_storage"
DBUG[08-30|00:47:53] Cloned new RBD storage volume for container "united-badger"
DBUG[08-30|00:47:53] Mapped RBD storage volume for container "united-badger"
DBUG[08-30|00:47:53] Created mountpoint "/var/lib/lxd/storage-pools/container_storage/containers/united-badger" for container "united-badger" for RBD storage volume
DBUG[08-30|00:47:53] Mounting RBD storage volume for container "united-badger" on storage pool "container_storage".
DBUG[08-30|00:47:54] Mounted RBD storage volume for container "united-badger" on storage pool "container_storage".
DBUG[08-30|00:47:54] Shifting root filesystem "/var/lib/lxd/containers/united-badger/rootfs" for "united-badger".
DBUG[08-30|00:48:08] Shifted rootfs for container "united-badger"
DBUG[08-30|00:48:08] Changed mountpoint "/var/lib/lxd/storage-pools/container_storage/containers/united-badger" permissions to 0755 for container "united-badger" for RBD storage volume
DBUG[08-30|00:48:08] Checking if RBD storage volume for container "united-badger" on storage pool "container_storage" is ready
DBUG[08-30|00:48:08] RBD storage volume for container "united-badger" on storage pool "container_storage" is ready
DBUG[08-30|00:48:08] Applied create template for container "united-badger"
DBUG[08-30|00:48:08] Created RBD storage volume for container "united-badger" on storage pool "container_storage"
DBUG[08-30|00:48:08] Unmounting RBD storage volume for container "united-badger" on storage pool "container_storage".
DBUG[08-30|00:48:12] Unmounted RBD storage volume for container "united-badger" on storage pool "container_storage".
DBUG[08-30|00:48:12] Mounting RBD storage volume for container "united-badger" on storage pool "container_storage".
DBUG[08-30|00:48:12] Mounted RBD storage volume for container "united-badger" on storage pool "container_storage".
DBUG[08-30|00:48:12] Unmounting RBD storage volume for container "united-badger" on storage pool "container_storage".
DBUG[08-30|00:48:13] Unmounted RBD storage volume for container "united-badger" on storage pool "container_storage".
DBUG[08-30|00:48:13] Success for task operation: 265dedaf-f2b4-4d75-9321-6d138d983a62
DBUG[08-30|00:48:13] handling ip=@ method=GET url=/1.0/containers/united-badger
DBUG[08-30|00:48:13]
{
"type": "sync",
"status": "Success",
"status_code": 200,
"operation": "",
"error_code": 0,
"error": "",
"metadata": {
"architecture": "x86_64",
"config": {
"image.architecture": "amd64",
"image.description": "ubuntu 15.10 amd64 (release) (20160715)",
"image.label": "release",
"image.os": "ubuntu",
"image.release": "wily",
"image.serial": "20160715",
"image.version": "15.10",
"volatile.apply_template": "create",
"volatile.base_image": "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049",
"volatile.idmap.base": "0",
"volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]",
"volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]"
},
"devices": {
"root": {
"path": "/",
"pool": "container_storage",
"type": "disk"
}
},
"ephemeral": false,
"profiles": [
"default"
],
"stateful": false,
"description": "",
"created_at": "2017-08-30T07:46:42Z",
"expanded_config": {
"environment.http_proxy": "",
"image.architecture": "amd64",
"image.description": "ubuntu 15.10 amd64 (release) (20160715)",
"image.label": "release",
"image.os": "ubuntu",
"image.release": "wily",
"image.serial": "20160715",
"image.version": "15.10",
"user.network_mode": "",
"volatile.apply_template": "create",
"volatile.base_image": "9b9de680184bc9951a4f6f771e40fc98284db21c94bd5384fe31f2434dfa9049",
"volatile.idmap.base": "0",
"volatile.idmap.next": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]",
"volatile.last_state.idmap": "[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":100000,\"Nsid\":0,\"Maprange\":65536}]"
},
"expanded_devices": {
"root": {
"path": "/",
"pool": "container_storage",
"type": "disk"
}
},
"name": "united-badger",
"status": "Stopped",
"status_code": 102,
"last_used_at": "1970-01-01T00:00:00Z"
}
}
DBUG[08-30|00:48:13] Disconnected events listener: 450e2341-d4fc-47fb-813a-8676127e9d48
Terminal 2:
root@node001:~# mkfs.xfs /dev/rbd7
meta-data=/dev/rbd7 isize=512 agcount=17, agsize=162816 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=1024 swidth=1024 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
root@node001:~#
Code I changed:
func makeFSType(path string, fsType string) (string, error) {
var err error
var msg string
switch fsType {
case "xfs":
logger.Debugf("PATH: %s", path)
logger.Debugf("You have 30 seconds to run ' mkfs.xfs %s ' in a different terminal", path)
time.Sleep(30 * time.Second)
default:
msg, err = shared.TryRunCommand(
"mkfs.ext4",
"-E", "nodiscard,lazy_itable_init=0,lazy_journal_init=0",
path)
}
if err != nil {
return msg, err
}
return "", nil
So one more thing I'd like you to try. Do the same stuff you did with stopping the daemon and then call mkfs.<type> on the corresponding /dev/rbd<idx> path and immediately after execute $? in the shell and report back the exit code. It maybe the case that the command actually creates the filesystem but still fails due to being unable to write the appropriate metadata. LXD calls TryRuncomman() which would retry for 30 seconds maybe this is why the error surfaces with LXD but not when you do it manually. Also, are the devices you're using for your Ceph cluster actual hard-disks or directories?
In general here's my current thinking on Ceph + ext4:
Tbh, the Ceph documentation is full of warning of using ext4 as the backing filesystem for Ceph OSD storage pools:
We recommend against using ext4 due to limitations in the size of xattrs it can store, and the problems this causes with the way Ceph handles long RADOS object names. Although these issues will generally not surface with Ceph clusters using only short object names (e.g., an RBD workload that does not include long RBD image names), other users like RGW make extensive use of long object names and can break.
Starting with the Jewel release, the ceph-osd daemon will refuse to start if the configured max object name cannot be safely stored on ext4.
Furthermore:
Note This may result in difficult-to-diagnose errors if you try to use RGW or other librados clients that do not properly handle or politely surface any resulting ENAMETOOLONG errors.
So at this point I'm about to can ext4 support in LXD and only support xfs and btrfs.
brauner
on 30 Aug 2017
Sorry, I also need to by way more clear about this for future reference: The problem doesn't come from the filesystem support for the RBD image itself. The problem comes from using ext4 as the underlying filesystem for the OSD storage pool itself. If you do that then we won't support it since Ceph is explicitly warning about his. Ideally we'd only support xfs. I talked to @javacruft about this as well.
About Bluestore the general idea seems to be that this is an interesting technology but as of yet not considered stable enough to be used. And the randomness of your failures seem to support that hypothesis. So I suggest you switch to a xfs based OSD storage pool layout and try to reproduce. If LXD is still failing weirdly please report back and we can keep on digging into this.
brauner
on 30 Aug 2017
The OSDs are using bluestore, and have an XFS based partition for the metadata -- the only time EXT4 was ever introduced was before I swapped the storage pool backend to xfs inside LXD, but that didn't seem to make any difference.
I will do more debug today and get back to you.
tl;dr -- there is no ext4 in my setup.
grsmith-projects
on 30 Aug 2017
Thanks!
brauner
on 30 Aug 2017
welp, I figured out what it is.
8589 execve("/sbin/mkfs.xfs", ["mkfs.xfs", "2017-08-30 14:55:43.409960 7f901a5c80c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb\n2017-08-30 14:55:43.410552 7f901a5c80c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb\n2017-08-30 14:55:43.417863 7f901a5c80c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb\n/dev/rbd8"],
8589 write(2, "2017-08-30 14:55:43.409960 7f901a5c80c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb\n2017-08-30 14:55:43.410552 7f901a5c80c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb\n2017-08-30 14:55:43.417863 7f901a5c80c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb\n/dev/rbd8: File name too long\n", 429) = 429
the warning messages that you can't disable in ceph for the bluestore stuff are being passed into the mkfs for some reason.
grsmith-projects
on 30 Aug 2017
Wait wait, do you get the message when you map the volume?
brauner
on 31 Aug 2017
Because that would explain why this gets passed to mkfs.
brauner
on 31 Aug 2017
Yes, when you map the volume manually -- that message comes up, and that does explain why.
I've upgraded to 12.2.0 where that setting is default now (?maybe?), and commented it out of the config - and it looks like everything is working.
12.2.0 was released yesterday :) fffffffffff
Maybe the fix for this could be to sanitize the output of the ceph client
grsmith-projects
on 31 Aug 2017
so anyone running ceph version lower than 12.2 and using bluestore will run into this same problem.
grsmith-projects
on 31 Aug 2017
I'm going to sanitize our path handling! Thanks so much man, you saved me a ton of debugging. We really really appreciate the effort! I'll send a fix for this tomorrow right away. (Baring any weird issues popping up.) :)
brauner
on 31 Aug 2017
@Mantwon, can you try to build with my #3744 it should fix your issue if we're right. I assume you're fine with building from source given your debugging skills. :)
brauner
on 31 Aug 2017
Confirmed works! nice.
DBUG[08-30|18:21:43] Created filesystem "xfs" for RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "lxd"
DBUG[08-30|18:21:43] Mounting RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "lxd".
DBUG[08-30|18:21:43] Mounted RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "lxd".
DBUG[08-30|18:21:46] Unpacked image for RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "lxd"
DBUG[08-30|18:21:46] Unmounting RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "lxd".
DBUG[08-30|18:22:03] Unmounted RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "lxd".
DBUG[08-30|18:22:03] Unmapped RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "lxd"
DBUG[08-30|18:22:04] Created snapshot for RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "lxd"
DBUG[08-30|18:22:04] Protected snapshot for RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "lxd"
DBUG[08-30|18:22:04] Createdd database entry for RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "lxd"
DBUG[08-30|18:22:04] Created RBD storage volume for image "38431cacc7971d3faa5c23c917897e840ac499fd3744625484720dfb38a4146b" on storage pool "lxd"
DBUG[08-30|18:22:04] Cloned new RBD storage volume for container "light-gobbler"
Related issues
tebanep
路
5Comments
AndreiPashkin
路
5Comments
rrva
路
5Comments
srkunze
路
3Comments
simos
路
3Comments
Most helpful comment
I'm going to sanitize our path handling! Thanks so much man, you saved me a ton of debugging. We really really appreciate the effort! I'll send a fix for this tomorrow right away. (Baring any weird issues popping up.) :)