Lxd: error: Failed to destroy ZFS filesystem: cannot destroy 'lxd/containers/lc3': dataset already exists

[tebanep]

Required information

• Distribution:
  Ubuntu
• Distribution version:
  16.04
• The output of "lxc info":

apicompat: 0
auth: trusted
environment:
addresses:

• 10.20.45.91:8443
• 10.176.160.1:8443
  architectures:
• x86_64
• i686
  certificate: |
  -----BEGIN CERTIFICATE-----
  -----END CERTIFICATE-----
  driver: lxc
  driverversion: 2.0.0
  kernel: Linux
  kernelarchitecture: x86_64
  kernelversion: 4.4.0-22-generic
  server: lxd
  serverpid: 21607
  serverversion: 2.0.0
  storage: zfs
  storageversion: "5"
  config:
  core.https_address: 0.0.0.0:8443
  core.trust_password: true
  storage.zfs_pool_name: lxd
  public: false

Issue description

Can't delete lxd container with ZFS backend after Failed Copy from remote.

I was trying to copy a container from a remote. Unfortunately the copy failed to download, however the container entry can be seen in lxc list in STOPPED state and it can't be started. Now, I'm unable to delete the broken container by using lxc delete lc3 --force because the following error raises:

error: Failed to destroy ZFS filesystem: cannot destroy 'lxd/containers/lc3': dataset already exists

Thanks for your help. Is there any work around to delete the container?

Steps to reproduce

1. Install CRIU in remote
2. Install CRIU in local
3. Run lxc copy vulcan:c3 lc3 and get:

error: Error transferring container data: checkpoint failed:
(00.005831) Error (sockets.c:129): Diag module missing (-2)
(00.007692) Error (sockets.c:129): Diag module missing (-2)
(00.009357) Error (sockets.c:129): Diag module missing (-2)
(00.086751) Error (cr-dump.c:1600): Dumping FAILED.

1. Run lxc info lc3 and get:

Name: lc3
Architecture: x86_64
Created: 2016/05/11 23:24 UTC
Status: Stopped
Type: persistent
Profiles: default

1. Run lxc delete lc3 --force and get the error:

error: Failed to destroy ZFS filesystem: cannot destroy 'lxd/containers/lc3': dataset already exists

Information to attach

• [ ] container log (lxc info NAME --show-log)

lxc info NAME --show-log

        lxc 20160511183249.870 INFO     lxc_confile - confile.c:config_idmap:1498 - read uid map: type u nsid 0 hostid 165536 range 65536
        lxc 20160511183249.870 INFO     lxc_confile - confile.c:config_idmap:1498 - read uid map: type g nsid 0 hostid 165536 range 65536
        lxc 20160511183249.870 DEBUG    lxc_conf - conf.c:instantiate_veth:2613 - instantiated veth 'veth0XDF8R/veth8TB6FV', index is '15'
        lxc 20160511183249.870 INFO     lxc_cgroup - cgroup.c:cgroup_init:68 - cgroup driver cgroupfs-ng initing for lc3
        lxc 20160511183249.935 DEBUG    lxc_conf - conf.c:lxc_assign_network:3047 - move 'eth0' to '23702'
        lxc 20160511183249.939 NOTICE   lxc_start - start.c:do_start:763 - switching to gid/uid 0/0 in new user namespace
        lxc 20160511183249.939 ERROR    lxc_conf - conf.c:mount_rootfs:807 - No such file or directory - failed to get real path for '/var/lib/lxd/containers/lc3/rootfs'
        lxc 20160511183249.939 ERROR    lxc_conf - conf.c:setup_rootfs:1221 - failed to mount rootfs
        lxc 20160511183249.939 ERROR    lxc_conf - conf.c:do_rootfs_setup:3611 - failed to setup rootfs for 'lc3'
        lxc 20160511183249.939 ERROR    lxc_conf - conf.c:lxc_setup:3693 - Error setting up rootfs mount after spawn
        lxc 20160511183249.939 ERROR    lxc_start - start.c:do_start:819 - failed to setup the container
        lxc 20160511183249.939 ERROR    lxc_sync - sync.c:__sync_wait:52 - An error occurred in another process (expected sequence number 2)
        lxc 20160511183249.939 WARN     lxc_conf - conf.c:lxc_delete_network:2908 - failed to remove interface 15 'eth0'
        lxc 20160511183249.939 ERROR    lxc_start - start.c:__lxc_start:1329 - failed to spawn 'lc3'
        lxc 20160511183249.979 INFO     lxc_conf - conf.c:run_script_argv:367 - Executing script '/usr/share/lxcfs/lxc.reboot.hook' for container 'lc3', config section 'lxc'
        lxc 20160511183250.481 INFO     lxc_conf - conf.c:run_script_argv:367 - Executing script '/usr/bin/lxd callhook /var/lib/lxd 4 stop' for container 'lc3', config section 'lxc'
        lxc 20160511183250.566 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:172 - command get_cgroup failed to receive response
        lxc 20160511183250.566 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:172 - command get_cgroup failed to receive response
        lxc 20160511183250.571 INFO     lxc_confile - confile.c:config_idmap:1498 - read uid map: type u nsid 0 hostid 165536 range 65536

lxc info NAME

Name: lc3
Architecture: x86_64
Created: 2016/05/11 23:24 UTC
Status: Stopped
Type: persistent
Profiles: default

• [ ] output of the client with --debug

lxc delete lc3 --debug

DBUG[05-11|18:56:30] Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"api_extensions":[],"api_status":"stable","api_version":"1.0","auth":"trusted","config":{"core.https_address":"0.0.0.0:8443","core.trust_password":true,"storage.zfs_pool_name":"lxd"},"environment":{"addresses":["10.20.45.91:8443","10.176.160.1:8443"],"architectures":["x86_64","i686"],"certificate":"-----BEGIN CERTIFICATE-----\nMIIF7jCCA9agAwIBAgIRAMexjTXozPaV6CMXldhHDY0wDQYJKoZIhvcNAQELBQAw\nUjEcMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEyMDAGA1UEAwwpcm9vdEBl\nZWNoZXZlcnJ5LUhQLUVOVlktVFMtMTUtTm90ZWJvb2stUEMwHhcNMTYwNTExMjA1\nOTMxWhcNMjYwNTA5MjA1OTMxWjBSMRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMu\nb3JnMTIwMAYDVQQDDClyb290QGVlY2hldmVycnktSFAtRU5WWS1UUy0xNS1Ob3Rl\nYm9vay1QQzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAKxkAyC/TSl4\njbJbe4Sv1/btyc6KXVgH08oB+tk+x4ItlmC7DjURdECWQe6hv+Sv9jrLjVPTCv+x\ncL2maYjGIxUV/zToY8qn3PTpWTyHnMjLd8/xRwuOcXFWT26GgxTL0OND1b0v4Du2\nSOUHCGcbqWLo6J2pj0BB6hDeTZ+kDZQgQvoE24EgjHM62IHCyqZmIb4euRgRmUon\nOCcHOfVV+vzgWtyCV59gOoyiTlzAOySJ/WNXJnYhlB4NJEYeCqtAwR56bGpqc5m0\ngNEogHIlKumdNWuIOxy4Q7h7vv5tCRvvRWEDFPgPc/98+yb24rjoBD61CBo3JHi3\nqx6HLhsxjzfiNzCEImfeIxRjq6wvH771XRVzM/W/00V+QjhE4JbxCX8BvjL3mNQP\nrY4Md/+qdH8kMTzRJZShXNJvFXEq7+QJPfYDKn/YZ5HXoXY/y8VY6oRjOt+TWoM8\nOFr9bWLZjGVUCGDnxkVmMsO0DM//D2gll3pED3UQyQHFVZcko2LunUDTgcUj4eK4\nsWAHbRjp9JdC8Z0QlhiLO3h2x7zaZGeowuIt5kyNMCjCdE7NnMoC1tz3ljeOoNJ4\n1jS6Ndu38UEztxCFv3gWfp0rfxq5Nyh4zjPpYMdZyo082TCOOTb5sC518qt9Wz0u\nXg7nLZt1oRgkv46CpATvab9EqvpEG2CPAgMBAAGjgb4wgbswDgYDVR0PAQH/BAQD\nAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwgYUGA1UdEQR+\nMHyCJGVlY2hldmVycnktSFAtRU5WWS1UUy0xNS1Ob3RlYm9vay1QQ4IOMTAuMjAu\nNDUuOTEvMjSCGmZlODA6OjhlMTpjNjYwOjYyMDo4YzUxLzY0ghxmZTgwOjpkNGQ2\nOjE1ZmY6ZmUxZjo4YTQxLzY0ggpmZTgwOjoxLzY0MA0GCSqGSIb3DQEBCwUAA4IC\nAQCJ1aBkU7HgsmzxOOtBI4BZvzg00HcnJ3j/RLbVsJqglmvMg4W2YEFFywzxsHGU\nRtwu28L7lxLgMjIm1tMIe8gM7GzriXBMvDEMJSWCJ5or54y9Wv904VOo4aIQTvL5\nK9GYnT+Emv9/QlQRvYFDNmVDjndJX0+vTQ2hfF3RJzha2ZZCoWVbKtslEI0IoJ6W\n0z8NSC+Ruu5PdVoFUlK36iRT6/t6BnDGX019wIDkGTORvylfpj/b/OpaPATts8d1\n2Hbav0AZvVOPwT5kgf8x5dtLT/j2l8zJcN6X/H1PXvqWswHeMf2f/14pSsVI//y4\nRoLMKa5jGtsXBF0hJ4cs47CSaQoo12w9u6nlt8+cBjGUU6YCjqeMeg4TudiHvFld\nlvzkL7u7V/STw42O4ZOYlu4KfjKb5P2/HUugFpEZLy/zSrIspMNd2r4VEkFRzs5m\nj5OXXActNHqj0iLG+ZZMN0+lpEgz3zvSP/ktytjkldbbW60sb6YuyXtycxYr3+Uo\njTPYq8XUrVSjNrDL7bQiyKW1MIAcVkpAfAWl2ejgDj2VGAA/wzmimWwJQGZCwgTS\nAy4qzlqWIz6H9C0sAANbsHtg3kiPNA5zzFRCJMFKuzavENXz1df5wEgzID0f0ZV9\nnbNyYCVl3B5NO9p57h4p6/rxAKdFqx8ZUy7r3/3VuF6N1Q==\n-----END CERTIFICATE-----\n","driver":"lxc","driver_version":"2.0.0","kernel":"Linux","kernel_architecture":"x86_64","kernel_version":"4.4.0-22-generic","server":"lxd","server_pid":21607,"server_version":"2.0.0","storage":"zfs","storage_version":"5"},"public":false}}

DBUG[05-11|18:56:30] Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"architecture":"x86_64","config":{"volatile.base_image":"f4c4c60a6b752a381288ae72a1689a9da00f8e03b732c8d1b8a8fcd1a8890800","volatile.eth0.hwaddr":"00:16:3e:75:d2:32","volatile.last_state.idmap":"[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":65536}]"},"created_at":"2016-05-11T18:24:40-05:00","devices":{"root":{"path":"/","type":"disk"}},"ephemeral":false,"expanded_config":{"volatile.base_image":"f4c4c60a6b752a381288ae72a1689a9da00f8e03b732c8d1b8a8fcd1a8890800","volatile.eth0.hwaddr":"00:16:3e:75:d2:32","volatile.last_state.idmap":"[{\"Isuid\":true,\"Isgid\":false,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":65536},{\"Isuid\":false,\"Isgid\":true,\"Hostid\":165536,\"Nsid\":0,\"Maprange\":65536}]"},"expanded_devices":{"eth0":{"name":"eth0","nictype":"bridged","parent":"lxdbr0","type":"nic"},"root":{"path":"/","type":"disk"}},"name":"lc3","profiles":["default"],"stateful":false,"status":"Stopped","status_code":102}}

DBUG[05-11|18:56:30] Deleting null
to http://unix.socket/1.0/containers/lc3
DBUG[05-11|18:56:30] Raw response: {"type":"async","status":"Operation created","status_code":100,"metadata":{"id":"2d2aefe7-3c0f-4adc-b773-4a0ca50533e8","class":"task","created_at":"2016-05-11T18:56:30.42556199-05:00","updated_at":"2016-05-11T18:56:30.42556199-05:00","status":"Running","status_code":103,"resources":{"containers":["/1.0/containers/lc3"]},"metadata":null,"may_cancel":false,"err":""},"operation":"/1.0/operations/2d2aefe7-3c0f-4adc-b773-4a0ca50533e8"}

DBUG[05-11|18:56:30] 1.0/operations/2d2aefe7-3c0f-4adc-b773-4a0ca50533e8/wait
DBUG[05-11|18:56:40] Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"id":"2d2aefe7-3c0f-4adc-b773-4a0ca50533e8","class":"task","created_at":"2016-05-11T18:56:30.42556199-05:00","updated_at":"2016-05-11T18:56:30.42556199-05:00","status":"Failure","status_code":400,"resources":{"containers":["/1.0/containers/lc3"]},"metadata":null,"may_cancel":false,"err":"Failed to destroy ZFS filesystem: cannot destroy 'lxd/containers/lc3': dataset already exists\n"}}

error: Failed to destroy ZFS filesystem: cannot destroy 'lxd/containers/lc3': dataset already exists

[stgraber]

I fixed that issue already, it will be in LXD 2.0.1.

Until then, you can workaround the issue by manually cleaning things up with zfs destroy.

[stgraber]

Ah, since you mentioned migration, check if you have a remaining "zfs receive" process running, if you do, kill it, that should unblock things.

[tebanep]

Thank you @stgraber I will try that out! And those are great news too, will wait till 2.0.1 then! :)

[stgraber]

Well, if the problem is a stuck zfs receive, I'm not 100% sure that my fix will do the trick.

My fix allows you to remove a container that's broken and also does proper zfs refcounting to not fail deletes. But I don't think it lets you workaround a locked zfs filesystem as would be the case with a leftover zfs receive.

[tebanep]

After rebooting, I could delete the container! Thanks!