Lxd: Error: Failed container creation: Error transferring container data: No root device could be found.
Required information
- Distribution: Ubuntu
- Distribution version: 16.04
- The output of "lxc info" or if that fails:
- Kernel version: Linux 4.4.0-127-generic x86_64
- LXC version: 3.0.1
- LXD version: 3.1
- Storage backend in use: ZFS
Issue description
Trying to move containers from a single LXD node to a LXD cluster node. I managed to move most containers properly except one.
I issued the command:
lxc move c1 vm1:
Error: Failed container creation: Error transferring container data: No root device could be found.
The target LXD server is showing this in the logs:
daemon log:
ephemeral=false lvl=info msg="Created container" name=c1 t=2018-06-08T06:00:52+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=c1 t=2018-06-08T06:00:52+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=c1 t=2018-06-08T06:00:52+0000
ephemeral=false lvl=info msg="Creating container" name=c1/daily t=2018-06-08T06:00:52+0000
created=2018-06-07T11:01:22+0000 ephemeral=false lvl=info msg="Deleting container" name=c1/daily t=2018-06-08T06:00:52+0000 used=1970-01-01T00:00:00+0000
created=2018-06-07T11:01:22+0000 ephemeral=false lvl=info msg="Deleted container" name=c1/daily t=2018-06-08T06:00:53+0000 used=1970-01-01T00:00:00+0000
ephemeral=false lvl=eror msg="Failed creating container" name=c1/daily t=2018-06-08T06:00:53+0000
err="No root device could be found." lvl=eror msg="Error during migration sink" t=2018-06-08T06:00:53+0000
created=2018-06-08T06:00:51+0000 ephemeral=false lvl=info msg="Deleting container" name=c1 t=2018-06-08T06:00:53+0000 used=1970-01-01T00:00:00+0000
created=2018-06-08T06:00:51+0000 ephemeral=false lvl=info msg="Deleted container" name=c1 t=2018-06-08T06:00:53+0000 used=1970-01-01T00:00:00+0000
monitor log:
No valid storage pool in the container's local root disk device and profiles
The container config everything appears to be normal, its using the same profile as other containers:
lxc config show c1
Name: c1
Remote: unix://
Architecture: x86_64
architecture: x86_64
config:
environment.http_proxy: ""
image.architecture: amd64
image.description: ubuntu 16.04 LTS amd64 (release) (20170919)
image.label: release
image.os: ubuntu
image.release: xenial
image.serial: "20170919"
image.version: "16.04"
raw.lxc: lxc.apparmor.allow_incomplete=1
security.privileged: "true"
user.network_mode: ""
volatile.base_image: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
volatile.eth0.hwaddr: xx:xx:xx:xx:xx:xx
volatile.idmap.base: "0"
volatile.idmap.next: '[]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: STOPPED
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: lxd
type: disk
ephemeral: false
profiles:
- default
stateful: false
The strange thing is that I'm able to move this container to another standalone LXD server but not to the LXD cluster.
So it seems to be related to the LXD cluster environment. The LXD cluster is on the same OS/kernel and LXD version as the standalone LXD servers.
Steps to reproduce
- lxc stop c1
- lxc move c1 vm1:
- Error: Failed container creation: Error transferring container data: No root device could be found.
Information to attach
- [ ] Any relevant kernel output (
dmesg) - [ ] Container log (
lxc info NAME --show-log) - [ ] Container configuration (
lxc config show NAME --expanded) - [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
- [ ] Output of the client with --debug
- [ ] Output of the daemon with --debug (alternatively output of
lxc monitorwhile reproducing the issue)
container_log.txt
dmesg.txt
lxc_client_debug.txt
lxc_config_expanded_view.txt
lxc_monitor_source.txt
lxc_monitor_target.txt
LXD_daemon_source_server.txt
LXD_daemon_target_server.txt
Bug
itcrowdsource
All 28 comments
What does lxc profile show vm1:default show you?
stgraber
on 8 Jun 2018
And lxc storage list vm1: too
stgraber
on 8 Jun 2018
sudo lxc profile show default
config:
raw.lxc: lxc.apparmor.allow_incomplete=1
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: local
type: disk
name: default
used_by:
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
sudo lxc storage list
+-------+-------------+--------+---------+---------+
| NAME | DESCRIPTION | DRIVER | STATE | USED BY |
+-------+-------------+--------+---------+---------+
| local | | zfs | CREATED | 12 |
+-------+-------------+--------+---------+---------+
Is that from vm1?
stgraber
on 8 Jun 2018
Yes, and vm1 is the target machine.
This is the output of the source machine vm01:
sudo lxc profile show default
config:
environment.http_proxy: ""
raw.lxc: lxc.apparmor.allow_incomplete=1
user.network_mode: ""
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: lxd
type: disk
name: default
used_by:
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
- /1.0/containers/xxxx
sudo lxc storage list
+------+-------------+--------+----------------------------------------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
+------+-------------+--------+----------------------------------------+---------+
| lxd | | zfs | /var/snap/lxd/common/lxd/disks/lxd.img | 23 |
+------+-------------+--------+----------------------------------------+---------+
Ok, can you try lxc config device remove c1 root?
That should remove the locally defined root device for the c1 container, making it inherit from the default profile instead, at which point the pool resolution should work fine during transfer.
With LXD 3.2, we'll have a -s flag to lxc move and lxc copy which lets you override the target pool, making remapping storage like in this case a bit nicer, but the device remove should work fine now.
stgraber
on 8 Jun 2018
That does not work, might this be the culprit? The other containers were migrated successfully. There seems to be something wrong with this specific container.
lxc config device remove c1 root
Error: The device doesn't exist
That's odd, can you post the lxc config show c1 output again for that container (don't pass --expanded).
stgraber
on 8 Jun 2018
The minimal view doesn't show the root device, the expanded view does.
lxc config show c1
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 16.04 LTS amd64 (release) (20170919)
image.label: release
image.os: ubuntu
image.release: xenial
image.serial: "20170919"
image.version: "16.04"
security.privileged: "true"
volatile.base_image: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
volatile.eth0.hwaddr: xx:xx:xx:xx:xx:xx
volatile.idmap.base: "0"
volatile.idmap.next: '[]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: STOPPED
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""
lxc config show c1 --expanded
architecture: x86_64
config:
environment.http_proxy: ""
image.architecture: amd64
image.description: ubuntu 16.04 LTS amd64 (release) (20170919)
image.label: release
image.os: ubuntu
image.release: xenial
image.serial: "20170919"
image.version: "16.04"
raw.lxc: lxc.apparmor.allow_incomplete=1
security.privileged: "true"
user.network_mode: ""
volatile.base_image: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
volatile.eth0.hwaddr: xx:xx:xx:xx:xx:xx
volatile.idmap.base: "0"
volatile.idmap.next: '[]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: STOPPED
devices:
eth0:
name: eth0
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: lxd
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""
That's weird... Can you show the output of the lxc move when you pass --debug to it?
stgraber
on 8 Jun 2018
lxc move c1 vm1: --debug
DBUG[06-08|08:18:18] Connecting to a local LXD over a Unix socket
DBUG[06-08|08:18:18] Sending request to LXD etag= method=GET url=http://unix.socket/1.0
DBUG[06-08|08:18:18] Got response struct from LXD
DBUG[06-08|08:18:18]
{
"config": {
"core.https_address": "vm01:8443",
"core.trust_password": true
},
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"macaroon_authentication",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "trusted",
"public": false,
"auth_methods": [
"tls"
],
"environment": {
"addresses": [
"vm01:8443"
],
"architectures": [
"x86_64",
"i686"
],
"certificate": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"certificate_fingerprint": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"driver": "lxc",
"driver_version": "3.0.1",
"kernel": "Linux",
"kernel_architecture": "x86_64",
"kernel_version": "4.4.0-127-generic",
"server": "lxd",
"server_pid": 3810,
"server_version": "3.1",
"storage": "zfs",
"storage_version": "0.6.5.6-0ubuntu20",
"server_clustered": false,
"server_name": "vm01"
}
}
DBUG[06-08|08:18:18] Connecting to a remote LXD over HTTPs
DBUG[06-08|08:18:18] Sending request to LXD etag= method=GET url=https://vm1:8443/1.0
DBUG[06-08|08:18:18] Got response struct from LXD
DBUG[06-08|08:18:18]
{
"config": {
"core.https_address": "vm1:8443",
"core.trust_password": true
},
"api_extensions": [
"storage_zfs_remove_snapshots",
"container_host_shutdown_timeout",
"container_stop_priority",
"container_syscall_filtering",
"auth_pki",
"container_last_used_at",
"etag",
"patch",
"usb_devices",
"https_allowed_credentials",
"image_compression_algorithm",
"directory_manipulation",
"container_cpu_time",
"storage_zfs_use_refquota",
"storage_lvm_mount_options",
"network",
"profile_usedby",
"container_push",
"container_exec_recording",
"certificate_update",
"container_exec_signal_handling",
"gpu_devices",
"container_image_properties",
"migration_progress",
"id_map",
"network_firewall_filtering",
"network_routes",
"storage",
"file_delete",
"file_append",
"network_dhcp_expiry",
"storage_lvm_vg_rename",
"storage_lvm_thinpool_rename",
"network_vlan",
"image_create_aliases",
"container_stateless_copy",
"container_only_migration",
"storage_zfs_clone_copy",
"unix_device_rename",
"storage_lvm_use_thinpool",
"storage_rsync_bwlimit",
"network_vxlan_interface",
"storage_btrfs_mount_options",
"entity_description",
"image_force_refresh",
"storage_lvm_lv_resizing",
"id_map_base",
"file_symlinks",
"container_push_target",
"network_vlan_physical",
"storage_images_delete",
"container_edit_metadata",
"container_snapshot_stateful_migration",
"storage_driver_ceph",
"storage_ceph_user_name",
"resource_limits",
"storage_volatile_initial_source",
"storage_ceph_force_osd_reuse",
"storage_block_filesystem_btrfs",
"resources",
"kernel_limits",
"storage_api_volume_rename",
"macaroon_authentication",
"network_sriov",
"console",
"restrict_devlxd",
"migration_pre_copy",
"infiniband",
"maas_network",
"devlxd_events",
"proxy",
"network_dhcp_gateway",
"file_get_symlink",
"network_leases",
"unix_device_hotplug",
"storage_api_local_volume_handling",
"operation_description",
"clustering",
"event_lifecycle",
"storage_api_remote_volume_handling",
"nvidia_runtime",
"container_mount_propagation",
"container_backup"
],
"api_status": "stable",
"api_version": "1.0",
"auth": "trusted",
"public": false,
"auth_methods": [
"tls"
],
"environment": {
"addresses": [
"vm1:8443"
],
"architectures": [
"x86_64",
"i686"
],
"certificate": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"certificate_fingerprint": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"driver": "lxc",
"driver_version": "3.0.1",
"kernel": "Linux",
"kernel_architecture": "x86_64",
"kernel_version": "4.4.0-127-generic",
"server": "lxd",
"server_pid": 7349,
"server_version": "3.1",
"storage": "zfs",
"storage_version": "0.6.5.6-0ubuntu20",
"server_clustered": true,
"server_name": "VM1"
}
}
DBUG[06-08|08:18:18] Sending request to LXD etag= method=GET url=http://unix.socket/1.0/containers/c1
DBUG[06-08|08:18:18] Got response struct from LXD
DBUG[06-08|08:18:18]
{
"architecture": "x86_64",
"config": {
"image.architecture": "amd64",
"image.description": "ubuntu 16.04 LTS amd64 (release) (20170919)",
"image.label": "release",
"image.os": "ubuntu",
"image.release": "xenial",
"image.serial": "20170919",
"image.version": "16.04",
"security.privileged": "true",
"volatile.base_image": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"volatile.eth0.hwaddr": "xx:xx:xx:xx:xx",
"volatile.idmap.base": "0",
"volatile.idmap.next": "[]",
"volatile.last_state.idmap": "[]",
"volatile.last_state.power": "STOPPED"
},
"devices": {},
"ephemeral": false,
"profiles": [
"default"
],
"stateful": false,
"description": "",
"created_at": "2017-11-26T09:39:22Z",
"expanded_config": {
"environment.http_proxy": "",
"image.architecture": "amd64",
"image.description": "ubuntu 16.04 LTS amd64 (release) (20170919)",
"image.label": "release",
"image.os": "ubuntu",
"image.release": "xenial",
"image.serial": "20170919",
"image.version": "16.04",
"raw.lxc": "lxc.apparmor.allow_incomplete=1",
"security.privileged": "true",
"user.network_mode": "",
"volatile.base_image": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"volatile.eth0.hwaddr": "xx:xx:xx:xx:xx",
"volatile.idmap.base": "0",
"volatile.idmap.next": "[]",
"volatile.last_state.idmap": "[]",
"volatile.last_state.power": "STOPPED"
},
"expanded_devices": {
"eth0": {
"name": "eth0",
"nictype": "bridged",
"parent": "br0",
"type": "nic"
},
"root": {
"path": "/",
"pool": "lxd",
"type": "disk"
}
},
"name": "c1",
"status": "Stopped",
"status_code": 102,
"last_used_at": "2018-06-08T05:46:07Z",
"location": ""
}
DBUG[06-08|08:18:18] Connected to the websocket
DBUG[06-08|08:18:18] Sending request to LXD etag= method=POST url=http://unix.socket/1.0/containers/c1
DBUG[06-08|08:18:18]
{
"name": "",
"migration": true,
"live": false,
"container_only": false,
"target": null
}
DBUG[06-08|08:18:18] Got operation from LXD
DBUG[06-08|08:18:18]
{
"id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"class": "websocket",
"description": "Migrating container",
"created_at": "2018-06-08T08:18:18.549572683Z",
"updated_at": "2018-06-08T08:18:18.549572683Z",
"status": "Running",
"status_code": 103,
"resources": {
"containers": [
"/1.0/containers/c1"
]
},
"metadata": {
"control": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"fs": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"may_cancel": false,
"err": ""
}
DBUG[06-08|08:18:18] Connected to the websocket
DBUG[06-08|08:18:18] Sending request to LXD etag= method=POST url=https://vm1:8443/1.0/containers
DBUG[06-08|08:18:18]
{
"architecture": "x86_64",
"config": {
"image.architecture": "amd64",
"image.description": "ubuntu 16.04 LTS amd64 (release) (20170919)",
"image.label": "release",
"image.os": "ubuntu",
"image.release": "xenial",
"image.serial": "20170919",
"image.version": "16.04",
"security.privileged": "true",
"volatile.base_image": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"volatile.eth0.hwaddr": "xx:xx:xx:xx:xx",
"volatile.idmap.base": "0",
"volatile.idmap.next": "[]",
"volatile.last_state.idmap": "[]",
"volatile.last_state.power": "STOPPED"
},
"devices": {},
"ephemeral": false,
"profiles": [
"default"
],
"stateful": false,
"description": "",
"name": "c1",
"source": {
"type": "migration",
"certificate": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"base-image": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"mode": "pull",
"operation": "https://vm01:8443/1.0/operations/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"secrets": {
"control": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"fs": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
},
"instance_type": ""
}
DBUG[06-08|08:18:19] Got operation from LXD
DBUG[06-08|08:18:19]
{
"id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"class": "task",
"description": "Creating container",
"created_at": "2018-06-08T08:18:19.949554799Z",
"updated_at": "2018-06-08T08:18:19.949554799Z",
"status": "Running",
"status_code": 103,
"resources": {
"containers": [
"/1.0/containers/c1"
]
},
"metadata": null,
"may_cancel": false,
"err": ""
}
DBUG[06-08|08:18:19] Sending request to LXD etag= method=GET url=https://vm1:8443/1.0/operations/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
DBUG[06-08|08:18:20] Got response struct from LXD
DBUG[06-08|08:18:20]
{
"id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"class": "task",
"description": "Creating container",
"created_at": "2018-06-08T08:18:19.949554799Z",
"updated_at": "2018-06-08T08:18:19.949554799Z",
"status": "Running",
"status_code": 103,
"resources": {
"containers": [
"/1.0/containers/c1"
]
},
"metadata": null,
"may_cancel": false,
"err": ""
}
Error: Failed container creation: Error transferring container data: No root device could be found.
You said the target is a cluster right, can you show lxc list and lxc cluster list?
I wonder if it's somehow one of the nodes that's unhappy with its config.
stgraber
on 8 Jun 2018
lxc cluster list
+------+---------------------------+----------+--------+-------------------+
| NAME | URL | DATABASE | STATE | MESSAGE |
+------+---------------------------+----------+--------+-------------------+
| VM1 | https://x.x.x.x:8443 | YES | ONLINE | fully operational |
+------+---------------------------+----------+--------+-------------------+
| VM2 | https://x.x.x.x:8443 | YES | ONLINE | fully operational |
+------+---------------------------+----------+--------+-------------------+
| VM3 | https://x.x.x.x:8443 | YES | ONLINE | fully operational |
+------+---------------------------+----------+--------+-------------------+
lxc list
root@VM2:~# lxc list
+---------------+---------+----------------------+------+------------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM1 |
| | | x.x.x.x (eth0) | | | | |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM2 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | STOPPED | | | PERSISTENT | 0 | VM2 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM1 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM2 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 2 | VM3 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM1 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM2 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM3 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM1 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM2 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | STOPPED | | | PERSISTENT | 0 | VM2 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM2 |
| | | x.x.x.x (eth0) | | | | |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 0 | VM1 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM1 |
+---------------+---------+----------------------+------+------------+-----------+----------+
| Cx | RUNNING | x.x.x.x (eth0) | | PERSISTENT | 1 | VM2 |
+---------------+---------+----------------------+------+------------+-----------+----------+
Ok, can you try lxc move but setting a specific target, starting with VM1, then VM2 and lastly VM3?
- lxc move c1 VM1: --target VM1
- lxc move c1 VM1: --target VM2
- lxc move c1 VM1: --target VM3
That'd confirm if it's a cluster-wide issue or something that's specific to a single node somehow.
stgraber
on 8 Jun 2018
On all three commands:
Error: You must use the same source and destination remote when using --target
Could it be a problem because I'm moving from a standalone LXD instance to a cluster?
itcrowdsource
on 8 Jun 2018
Hmm, that check is wrong... the only requirement for --target should be that the target be a cluster...
stgraber
on 8 Jun 2018
Sorry, could you clarify that?
I ran the following command:
lxc move c1 vm1: --target vm1
vm1 is one of the cluster nodes
itcrowdsource
on 8 Jun 2018
The fact that it failed is a bug, I'm working on a fix for that now.
stgraber
on 8 Jun 2018
In the meantime, to check if it's some weird profile state issue we're running into here, can you, for every one of the 3 nodes, connect to them and do:
systemctl reload snap.lxd.daemon- wait about 30s
lxc cluster listto confirm that it's properly back online- Move on to the next node
Effectively restarting LXD on all 3 nodes to force them to reload their database. systemctl reload will only restart LXD, containers will be kept running during that time.
Once that's done, try the lxc move c1 vm1: again
stgraber
on 8 Jun 2018
The reload didn't help. I'll wait till the fix arrives then:-) Please let me know if you wish to conduct some further tests,
itcrowdsource
on 8 Jun 2018
Is the machine you're running the lxc command from also using the snap packages?
stgraber
on 8 Jun 2018
If so, can you do (as root):
- curl https://dl.stgraber.org/lxc-4633 -o /var/snap/lxd/common/lxc.debug
- chmod +x /var/snap/lxd/common/lxc.debug
And then try lxc move c1 vm1: --target vm1 see what that does? If that fails, try with --target vm2 and --target vm3
stgraber
on 8 Jun 2018
👍1
all nodes (source and target) are running the same LXD snap version (3.1).
I've installed your script on the source server from which I ran the lxc move command.
The provided command doesn't work:
root@vm01:~/lxddebug# lxc move c1 vm1: --target vm1
Error: You must specify a destination container name when using --target
Therefore I added the destination container name:
root@vm01:~/lxddebug# lxc move c1 vm1:c1 --target vm1
Error: Failed to get target node: No such object
root@vm01:~/lxddebug# lxc move c1 vm1:c1 --target vm2
Error: Failed to get target node: No such object
root@vm01:~/lxddebug# lxc move c1 vm1:c1 --target vm3
Error: Failed to get target node: No such object
root@vm01:~/lxddebug# lxc move c1 vm2:c1 --target vm2
Error: Failed to get target node: No such object
root@vm01:~/lxddebug# lxc move c1 vm2:c1 --target vm3
Error: Failed to get target node: No such object
root@vm01:~/lxddebug# lxc move c1 vm3:c1 --target vm3
Error: Failed to get target node: No such object
I ran the same command as yesterday, now it seems to transfer the container to the cluster. Except deleting the source container, I guess that's a ZFS issue besides this issue right?
root@vm01:~/lxddebug# lxc move c1 vm1:
Error: Failed to destroy ZFS filesystem: cannot destroy snapshot lxd/containers/c1@snapshot-daily: dataset is busy
I also found some other strange behavior, not sure if it's related to this issue. But I'd like to mention it.
On two of the three newly created cluster nodes I now require to use sudo to run the lxc command (even if I did sudo su)
root@VM1:/# lxc list
bash: lxc: command not found
root@VM1:/# sudo lxc list
+---------------+---------+----------------------+------+------------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+---------------+---------+----------------------+------+------------+-----------+----------+
root@VM2:~# lxc list
+---------------+---------+----------------------+------+------------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+---------------+---------+----------------------+------+------------+-----------+----------+
root@VM3:~# lxc list
bash: lxc: command not found
root@VM3:~# sudo lxc list
+---------------+---------+----------------------+------+------------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+---------------+---------+----------------------+------+------------+-----------+----------+
The need to use sudo would indicate a problem with the PATH variable not including the /snap/bin path inside it. That can be caused by a custom .bashrc or maybe an issue somewhere in snapd.
I'll have to look into that No such object error, it looks a bit odd.
As for the move now doing something, that's good, though I'm not sure why it'd now start working.
The error you're getting during delete is because of a zfs bug which has since been fixed upstream but older kernels are still lacking... Can you confirm that your container is now on the cluster at last?
stgraber
on 10 Jun 2018
the sudo issue was indeed related to the path setting that's fixed. And I can confirm that the container is on the cluster right now. Unfortunately we couldn't determine what the cause was. But I'll keep an eye on it.
itcrowdsource
on 10 Jun 2018
No such object was because of using --target vm1 instead of --target VM1. That error is terrible and we need to fix it. I'll send a fix for it.
stgraber
on 18 Jun 2018
👍1
root@vm09:~# lxc launch ubuntu:16.04 c3 --target VM09
Creating c3
Error: Failed container creation: No cluster member called 'VM09'
Ah right I see, it's an upper/lowercase issue...
itcrowdsource
on 19 Jun 2018
Was this page helpful?
0 / 5 - 0 ratings
Related issues
AndreiPashkin
路
5Comments
shaun-ba
路
3Comments
kp3nguinz
路
5Comments
chuegel
路
3Comments
fwaggle
路
4Comments