Lxd: 'lxc list' huge lags, slow LXD related tasks

Created on 29 Jun 2018 · 13Comments · Source: lxc/lxd

Required information

Distribution: Ubuntu
Distribution version: Ubuntu 18.04 LTS
The output of "lxc info" or if that fails:

config:
core.https_address: 10.240.0.107
core.trust_password: true
api_extensions:

storage_zfs_remove_snapshots
container_host_shutdown_timeout
container_stop_priority
container_syscall_filtering
auth_pki
container_last_used_at
etag
patch
usb_devices
https_allowed_credentials
image_compression_algorithm
directory_manipulation
container_cpu_time
storage_zfs_use_refquota
storage_lvm_mount_options
network
profile_usedby
container_push
container_exec_recording
certificate_update
container_exec_signal_handling
gpu_devices
container_image_properties
migration_progress
id_map
network_firewall_filtering
network_routes
storage
file_delete
file_append
network_dhcp_expiry
storage_lvm_vg_rename
storage_lvm_thinpool_rename
network_vlan
image_create_aliases
container_stateless_copy
container_only_migration
storage_zfs_clone_copy
unix_device_rename
storage_lvm_use_thinpool
storage_rsync_bwlimit
network_vxlan_interface
storage_btrfs_mount_options
entity_description
image_force_refresh
storage_lvm_lv_resizing
id_map_base
file_symlinks
container_push_target
network_vlan_physical
storage_images_delete
container_edit_metadata
container_snapshot_stateful_migration
storage_driver_ceph
storage_ceph_user_name
resource_limits
storage_volatile_initial_source
storage_ceph_force_osd_reuse
storage_block_filesystem_btrfs
resources
kernel_limits
storage_api_volume_rename
macaroon_authentication
network_sriov
console
restrict_devlxd
migration_pre_copy
infiniband
maas_network
devlxd_events
proxy
network_dhcp_gateway
file_get_symlink
network_leases
unix_device_hotplug
storage_api_local_volume_handling
operation_description
clustering
event_lifecycle
storage_api_remote_volume_handling
nvidia_runtime
container_mount_propagation
container_backup
devlxd_images
container_local_cross_pool_handling
proxy_unix
proxy_udp
clustering_join
proxy_tcp_udp_multi_port_handling
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
tls
environment:
addresses:
- 10.240.0.107:8443
  
  architectures:
- x86_64
- i686
  
  certificate: |
  
  -----BEGIN CERTIFICATE-----
  
  ........
  
  -----END CERTIFICATE-----
  
  certificate_fingerprint: ........
  
  driver: lxc
  
  driver_version: 3.0.1
  
  kernel: Linux
  
  kernel_architecture: x86_64
  
  kernel_version: 4.15.0-1009-gcp
  
  server: lxd
  
  server_pid: 128681
  
  server_version: "3.2"
  
  storage: zfs
  
  storage_version: 0.7.9-0york0~18.04
  
  server_clustered: false
  
  server_name: lxd-xxxxxx

Issue description

'lxc list' and 'lxc list --fast' lag considerably (we are aware of issue #4548, we don't use clustering) after lxd runs for some time. It may take few minutes for 'lxc list' and like 30 seconds for 'lxc list --fast' to complete. We've noticed that 'pkill lxd' fixes that temporarily, but on each 'lxc list' run it takes 5-10 seconds more for command to complete. So if we run 'pkill lxd', it may take more than a minute for 'lxc list' command to complete after just several runs. Everything else related to LXD daemon is slow as well. This happens on all Ubuntu 18.04 / LXD 3.x machines.

Steps to reproduce

'lxc list' takes one or several minutes to complete, 'lxc list --fast' around 30 seconds.
'pkill lxd' fixes this without restarting containers, so after that 'lxc list' takes 20 seconds and 'lxc list --fast' 5 seconds.
Unfortunately it takes 5-10 seconds longer for 'lxc list' command to complete on each consecutive run.
'lxc list --fast' holds on much better (there's no 5-10 seconds additional delay after each run) and it takes about 5 seconds to complete.

Information to attach

[ ] Any relevant kernel output (dmesg)
[ ] Container log (lxc info NAME --show-log)
[ ] Container configuration (lxc config show NAME --expanded)
[x] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)

lvl=info msg="LXD 3.2 is starting in normal mode" path=/var/snap/lxd/common/lxd t=2018-06-28T13:17:03+0000
lvl=info msg="Kernel uid/gid map:" t=2018-06-28T13:17:03+0000
lvl=info msg=" - u 0 0 4294967295" t=2018-06-28T13:17:03+0000
lvl=info msg=" - g 0 0 4294967295" t=2018-06-28T13:17:03+0000
lvl=info msg="Configured LXD uid/gid map:" t=2018-06-28T13:17:03+0000
lvl=info msg=" - u 0 1000000 1000000000" t=2018-06-28T13:17:03+0000
lvl=info msg=" - g 0 1000000 1000000000" t=2018-06-28T13:17:03+0000
lvl=info msg="Initializing local database" t=2018-06-28T13:17:03+0000
lvl=info msg="Initializing database gateway" t=2018-06-28T13:17:03+0000
address= id=1 lvl=info msg="Start database node" t=2018-06-28T13:17:03+0000
lvl=info msg="Raft: Restored from snapshot 1-31387-1530191821167" t=2018-06-28T13:17:03+0000
lvl=info msg="Raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:0}]" t=2018-06-28T13:17:03+0000
lvl=info msg="Raft: Node at 0 [Leader] entering Leader state" t=2018-06-28T13:17:03+0000
lvl=info msg="LXD isn't socket activated" t=2018-06-28T13:17:03+0000
lvl=info msg="Starting /dev/lxd handler:" t=2018-06-28T13:17:03+0000
lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock t=2018-06-28T13:17:03+0000
lvl=info msg="REST API daemon:" t=2018-06-28T13:17:03+0000
lvl=info msg=" - binding Unix socket" socket=/var/snap/lxd/common/lxd/unix.socket t=2018-06-28T13:17:03+0000
lvl=info msg=" - binding TCP socket" socket=10.240.0.107:8443 t=2018-06-28T13:17:03+0000
lvl=info msg="Initializing global database" t=2018-06-28T13:17:03+0000
lvl=info msg="Initializing storage pools" t=2018-06-28T13:17:03+0000
lvl=info msg="Initializing networks" t=2018-06-28T13:17:03+0000
lvl=info msg="Loading configuration" t=2018-06-28T13:17:05+0000
lvl=info msg="Connected to MAAS controller" t=2018-06-28T13:17:06+0000
lvl=info msg="Pruning expired images" t=2018-06-28T13:17:06+0000
lvl=info msg="Done pruning expired images" t=2018-06-28T13:17:06+0000
lvl=info msg="Updating instance types" t=2018-06-28T13:17:06+0000
lvl=info msg="Expiring log files" t=2018-06-28T13:17:06+0000
lvl=info msg="Updating images" t=2018-06-28T13:17:07+0000
lvl=info msg="Done updating images" t=2018-06-28T13:17:07+0000
lvl=info msg="Done expiring log files" t=2018-06-28T13:17:07+0000
lvl=info msg="Done updating instance types" t=2018-06-28T13:17:10+0000
ephemeral=false lvl=info msg="Creating container" name=Axj-xxxxxx/xxxxxx-1530192805-autodaily t=2018-06-28T13:33:25+0000
ephemeral=false lvl=info msg="Created container" name=Axj-xxxxxx/xxxxxx-1530192805-autodaily t=2018-06-28T13:33:32+0000

[x] Output of the client with --debug

There are some considerable response lags:

DBUG[06-29|07:38:10] Connecting to a local LXD over a Unix socket
DBUG[06-29|07:38:10] Sending request to LXD etag= method=GET url=http://unix.socket/1.0
DBUG[06-29|07:38:10] Got response struct from LXD
DBUG[06-29|07:38:10]
...
DBUG[06-29|07:38:10] Sending request to LXD etag= method=GET url="http://unix.socket/1.0/containers?recursion=1"
DBUG[06-29|07:38:31] Got response struct from LXD
DBUG[06-29|07:38:31]
...

[ ] Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

Bug

Source

ubojan

👍1

Most helpful comment

If you're not seeing any such slowdowns with current master then we should be good.
I've got stable cherry-picks next on my todo list (first need to unstick a testing issue) so hoping to have stable channels for latest and 3.0 updated by end of week.

stgraber on 12 Jul 2018

🎉3

All 13 comments

Can you run lxc monitor while you do a lxc list and then again for lxc list --fast?

stgraber on 29 Jun 2018

Might also be useful to know how many containers you have on the system so we can better try to reproduce this.

As a point of comparison, one of my own servers with 30 active containers takes half a second with --fast and goes up to 1.8s or so without --fast. Daemon has been running for over 2 days and is rather busy (mostly lxc exec sessions though).

stgraber on 30 Jun 2018

Thanks for your reply. Server which I used to produce results has 190 running containers, others have similar number.

Here is monitor output for lxc list --fast:

metadata:
  context: {}
  level: dbug
  message: 'New event listener: bc0b2907-60bc-49e1-af5b-a5c9873d4fe0'
timestamp: "2018-07-04T13:50:26.537129979Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0
  level: dbug
  message: handling
timestamp: "2018-07-04T13:50:30.207112963Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:50:30.211394825Z"
type: logging

And here is for lxc list (container names are anonymized):

# lxc monitor
metadata:
  context: {}
  level: dbug
  message: 'New event listener: 29a5fb92-2359-41f3-b176-a310f26fdb1b'
timestamp: "2018-07-04T13:51:40.932808713Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:43.495771484Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:43.49982359Z"
type: logging


metadata:
  context:
    k: "0"
  level: dbug
  message: Found cert
timestamp: "2018-07-04T13:51:43.933817085Z"
type: logging


metadata:
  context:
    ip: 10.240.0.75:43464
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots/xxxxxxx-1529543288-autodaily
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:43.933934425Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.627958312Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.628905708Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.629733729Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.633606033Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.630706804Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.631478896Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.632787967Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.635325473Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.636573396Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.638752744Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.640223056Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.641623359Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.643301766Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.644697657Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.647038168Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.650282672Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.65495218Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.662295833Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.659867829Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.772669295Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:47.933744387Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:47.939953809Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.979938131Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:47.985509908Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.01497911Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.022146659Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:48.061698306Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.080506964Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:48.081777159Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.157726178Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:48.163097386Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.167610762Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:48.200544028Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:48.216191105Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.25422237Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.260173632Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.262209502Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:48.308256452Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.509273835Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.514307413Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.600270834Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.600892385Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.613559136Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.618941192Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.681769256Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:48.784510795Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:49.037207182Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:49.039362443Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:49.251458025Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:49.252801962Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:49.275270119Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:49.397805741Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:49.556909601Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:49.872586287Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:49.874465203Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:49.997239212Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:49.998119078Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:50.003764717Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:50.077268814Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:50.203196039Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:50.311324967Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:50.320023371Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:50.350226142Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:50.445027947Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:50.464917184Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:50.61241638Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:50.614953906Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:50.71757031Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:50.720927036Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:50.741488332Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:51.038996498Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.059903269Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.06860544Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.077145618Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.24735946Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.253946028Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.331833913Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:51.342761491Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.364199063Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:51.439597338Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.440574737Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.7327442Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:51.858283115Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.888682879Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.890556766Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:51.988007087Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.195855093Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:52.278244674Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.328386647Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.337528187Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.341209791Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.34629046Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.386062996Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.391757451Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.392240158Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.469604888Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:52.594559374Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:52.636578073Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.888506488Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.895870618Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:52.940698546Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.978112757Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:52.980846116Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:52.988677228Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.163299814Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.168461655Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.190969663Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.193560754Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.234452832Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:53.242073493Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.25270255Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:53.451653495Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:53.466527598Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.498823323Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.502756413Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:53.515268339Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:53.666733678Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:53.672401288Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.726698403Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.748795558Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.76619193Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:53.953224107Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.995712181Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:53.996993743Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:54.099922579Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:54.118572702Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:54.152312966Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:54.163123794Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:54.181556439Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:54.586304414Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:54.590318038Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:54.691338818Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:54.726106548Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:54.727252299Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:54.736055134Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:54.885084741Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:54.903114814Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:55.072222323Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:55.535435944Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:55.795120639Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:55.799955898Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:56.020831767Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:56.023343059Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:56.219500424Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:56.236554673Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:56.253510062Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:56.508322872Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:56.610736067Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:56.611744567Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:56.626154661Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:56.631027605Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:56.64001287Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:56.78573536Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:56.979830787Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:56.980180574Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:56.980733789Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:57.08253072Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:57.086798557Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:57.167949529Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:57.180514847Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:57.184209031Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:57.187465048Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:57.231071428Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:57.234083826Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:57.288168551Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:57.569571609Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:57.575226789Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:57.629170251Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:57.62996027Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:58.10700115Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:58.127236481Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:58.129759022Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:58.390429367Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:58.472822228Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:58.575921405Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:58.585246114Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:58.588980391Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:58.590383785Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:58.597579581Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:58.742229046Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:58.849534872Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:59.04995833Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:59.096244733Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:59.113569949Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:59.115022414Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:59.118278261Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:59.127892074Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:59.423014863Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:59.423891336Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:59.465681768Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:59.477684791Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:59.482904861Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:59.607232542Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:51:59.608828815Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:51:59.616194828Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:00.014419177Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:00.061439066Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:00.155477538Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:00.162028316Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:00.340801076Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:00.827896258Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:00.940314453Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.071695479Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.117477417Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.197164579Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.312868542Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:01.352770795Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.39644311Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.66544154Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.667991046Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.669423313Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.706929892Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.709123769Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.736226707Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.82820086Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.943511508Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:01.944200177Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:02.181979211Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:02.182586336Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:02.188420942Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:02.204793327Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:02.208184718Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:02.212799154Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:02.216909219Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:02.303588934Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:02.314521078Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:02.536509442Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:02.556474834Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:02.562199721Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:02.803265684Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:02.804277186Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:02.805329196Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:02.81081176Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:02.82009879Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:03.104150334Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:03.106908912Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:03.194693744Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:03.203552187Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:03.224901862Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:03.341331508Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:03.351719891Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:03.353723326Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:03.355746066Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:03.455932496Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:03.535386614Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:03.664281729Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:04.107262434Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:04.110433004Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:04.237934767Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:04.251579748Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:04.267826398Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:04.274114612Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:04.29465313Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:04.298445495Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:04.323268788Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:04.332447122Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:04.750516242Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:04.757594772Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:04.863646127Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:04.961533129Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:05.55299839Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:05.599126493Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:05.946090154Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:06.007640968Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:06.008168754Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:06.125740699Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:06.134767321Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:06.137618684Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:06.139068821Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:06.186104675Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:06.187278704Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:06.296561106Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:06.308616333Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:06.410861815Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:06.424598875Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:06.728228186Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:07.149351185Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:07.682238114Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:07.693211366Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:07.698009668Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:07.716534602Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:07.881434999Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:07.885621748Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.079433161Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:08.106550002Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.131990748Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.135847099Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:08.144005974Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:08.250983631Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.29955886Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.391291381Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.428111766Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.429178237Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.430066824Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.43405289Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:08.459246573Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.57215258Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:08.595891562Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.652759654Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.739615421Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:08.747262429Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.759676909Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.761981155Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:08.76906401Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.958798905Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:08.969731188Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:09.000415992Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:09.013377385Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:09.014130533Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:09.024417545Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:09.02522347Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:09.052510931Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:09.238713962Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:09.330380314Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:09.412395476Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:09.413030898Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:09.622274229Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:09.664046576Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:09.667985498Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:09.743156204Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:10.111413552Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:10.128468054Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:10.26157137Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:10.702551839Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:10.703833128Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:10.706476979Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:10.827691866Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:10.829685115Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:10.844893635Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:10.859068205Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:11.293249766Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:11.295455443Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:11.306202859Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:11.310747238Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:11.426451917Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:11.428574897Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:11.460040815Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:11.577315685Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:12.051155333Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:12.056361948Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:12.159763444Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:12.259935719Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:12.428979997Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:12.475234328Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:12.496085804Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:13.09818098Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:13.387361793Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:13.392716468Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:13.399610518Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:13.422463801Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:13.425079373Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:13.531416576Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:13.694562111Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:13.819187706Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:13.823840379Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:13.82905397Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:13.917794248Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:13.923085826Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:14.184082106Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:14.58047197Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:14.69126768Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:14.692010774Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:15.017997572Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:15.258146142Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:15.39742076Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:15.438630462Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:15.448602113Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:15.487017261Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:15.499308902Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:15.643521349Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:15.680874822Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:15.706551564Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:15.707598196Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:15.735956941Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:15.743742994Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.066757579Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:16.201567044Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.203049155Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.205578301Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.33402318Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:16.351783882Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.378344622Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:16.477443334Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.578701937Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.579230401Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.580844112Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.58132029Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.589650997Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.686179205Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.843682958Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:16.856729246Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:16.861304135Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:16.972581484Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:16.996856303Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:17.216680346Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:17.247151322Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:17.294036528Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:17.305098429Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:17.307210724Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:17.308238523Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:17.351312419Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:17.368287485Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:17.594533283Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:17.705666821Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:17.734579487Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:17.737493921Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:17.950364629Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:18.508650863Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:18.556681121Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:18.73183008Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:18.733295387Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:18.967897785Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:19.131244362Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:19.133668114Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:19.140186079Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:19.196923211Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:19.322772977Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:19.463731312Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:19.475259549Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:19.477042181Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:19.872047607Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:19.872614541Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:20.068737826Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:20.069922925Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.347320209Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.366616119Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.407504121Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.505078727Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.538715904Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.546118649Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.585770386Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.59552677Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:20.599301828Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.626269336Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:20.64338476Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:20.660753905Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.692513384Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.702465268Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.715762509Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:20.890395453Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:20.891998062Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:21.517342614Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:21.52380774Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:21.525924551Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:21.639294708Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:22.216108228Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.255219156Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.260404938Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.266111809Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.270271444Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.273071894Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.364414846Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:22.510637447Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.640855436Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:22.651170083Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:22.659985689Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.670584321Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.671209233Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.672162512Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.6727272Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:22.933661239Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:22.954347182Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:22.962412718Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:23.04116756Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:23.054073628Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:23.345143133Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:23.40162685Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:23.527915191Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:23.854292785Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:23.949101723Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:23.951335626Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:23.958756886Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:23.969491033Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:23.986850035Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:24.051858632Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:24.063166243Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.080057098Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.085720074Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.095633889Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.326340367Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.424140454Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.424854971Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.435319101Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.437647735Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.896386071Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:24.910991685Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.930053594Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.932605961Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.985473447Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:24.996659916Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:25.016264772Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:25.018730147Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:25.10790396Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:25.11923655Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:25.125251777Z"
type: logging


metadata:
  context:
    ip: 10.240.0.75:45120
    method: GET
    url: /1.0/containers/xxxxxxxx/snapshots/xxxxxxx-1529334326-autodaily
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:25.144437413Z"
type: logging


metadata:
  context:
    k: "0"
  level: dbug
  message: Found cert
timestamp: "2018-07-04T13:52:25.144394372Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:25.393086876Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:25.393622378Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:25.607115069Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:25.62623499Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:25.632751174Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:25.643776817Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:25.793413729Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:25.799548233Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:25.802579847Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:25.804361305Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:26.231404337Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:26.252052569Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:26.253703176Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:26.381188017Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:26.38438499Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:26.493622746Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:26.502655364Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:26.509989948Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:26.519821903Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:26.523304981Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:26.65341749Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:26.655121208Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:26.964494941Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:26.975966006Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:26.977505471Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:27.082776779Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:27.127249824Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:27.271273315Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:27.416617192Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:27.421020601Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:27.477673723Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:27.586687315Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:27.587541584Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:27.590810666Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:27.651183579Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:28.481155912Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:28.774236598Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:28.774693817Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:28.850272927Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:28.851588062Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:28.89126552Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:28.893512266Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:28.939920616Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:28.951707694Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:28.958775968Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:29.066731529Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:29.080008107Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:29.344542244Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:29.348154646Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:29.608872277Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:29.640426178Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:29.642156728Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:29.658497672Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:29.659615926Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:29.673469892Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:29.88561885Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:29.891102396Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:29.930269258Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:29.987590156Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:30.111014079Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:30.115227574Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:30.121809868Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:30.388093541Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:30.653114799Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:30.769001276Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:30.772368988Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:30.776159849Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:30.881809884Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:30.906630821Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:30.916311485Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:30.918438228Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:30.930644512Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:31.066536527Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:31.090487854Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:31.246891385Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:31.253273892Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:31.254117519Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:31.260483775Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:31.289454955Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:31.405450468Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:31.417020056Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:31.437998761Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:31.610923154Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:31.642733592Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:31.680298421Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:31.687925086Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:31.706674878Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:31.836776997Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:32.095944103Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:32.108987793Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.369238024Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.374122731Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.41613951Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.422719742Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.437586103Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.444865058Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.448819447Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:32.593239326Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:32.630763526Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:32.670184964Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.806757558Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.911259201Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.930410505Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.936154819Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:32.947284948Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:33.446191005Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:33.551992413Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:33.553722386Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:33.554371012Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:33.560068622Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:33.565594597Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:33.881445185Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:33.891362959Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:33.895608089Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:33.90414353Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:33.974874968Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:33.976453786Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:34.104304744Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:34.291067075Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:34.449377466Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:34.450713716Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:34.646214198Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:34.652600539Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:34.680503138Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:34.684175618Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:34.697085722Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:34.808959868Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:34.827733257Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:34.830931083Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.168417159Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.181140133Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.190656916Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.29879284Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.329161073Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.329814494Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.332672428Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.337563083Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.340467918Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/state
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.341120732Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:35.401414966Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:35.514556095Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:35.550892868Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:35.555231918Z"
type: logging


metadata:
  context: {}
  level: dbug
  message: Initializing a ZFS driver
timestamp: "2018-07-04T13:52:35.555623024Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.579712175Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.804185695Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:35.845638888Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:36.125343069Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:36.531190375Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:36.777456533Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:36.783271972Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:37.464229665Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:37.609258052Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:37.626433983Z"
type: logging


metadata:
  context:
    ip: '@'
    method: GET
    url: /1.0/containers/xxxxxxx/snapshots?recursion=1
  level: dbug
  message: handling
timestamp: "2018-07-04T13:52:37.855191768Z"
type: logging

ubojan on 4 Jul 2018

Can you locate the PID of the main LXD process (ps aux | grep lxd.*group usually does the trick) and then do cat /proc/<PID>/maps and attach that output?

We've had a few cases of ongoing leaks due to config parsing issues in liblxc. I thought I cherry-picked the right fixes into the snap a couple of days ago though, so it'd be interesting to see if something wrong is still going on there. Please confirm you're on snap revision 7651 or higher though.

stgraber on 7 Jul 2018

Thanks for your reply.

I can confirm that lxd is on snap revision 7651.

I was able to locate it and here is requested output:

00400000-015ce000 r-xp 00000000 07:06 41                                 /snap/lxd/7651/bin/lxd
017cd000-017ce000 r--p 011cd000 07:06 41                                 /snap/lxd/7651/bin/lxd
017ce000-0182c000 rw-p 011ce000 07:06 41                                 /snap/lxd/7651/bin/lxd
0182c000-01859000 rw-p 00000000 00:00 0 
03034000-0318e000 rw-p 00000000 00:00 0                                  [heap]
c000000000-c0000ce000 rw-p 00000000 00:00 0 
c41e64c000-c420000000 rw-p 00000000 00:00 0 
c420000000-c421600000 rw-p 00000000 00:00 0 
c421600000-c421d00000 rw-p 00000000 00:00 0 
c421d00000-c422100000 rw-p 00000000 00:00 0 
c422100000-c422300000 rw-p 00000000 00:00 0 
c422300000-c422b00000 rw-p 00000000 00:00 0 
c422b00000-c422d00000 rw-p 00000000 00:00 0 
c422d00000-c423600000 rw-p 00000000 00:00 0 
c423600000-c423700000 rw-p 00000000 00:00 0 
c423700000-c423800000 rw-p 00000000 00:00 0 
c423800000-c425100000 rw-p 00000000 00:00 0 
c425100000-c425800000 rw-p 00000000 00:00 0 
c425800000-c427e00000 rw-p 00000000 00:00 0 
c427e00000-c428c00000 rw-p 00000000 00:00 0 
c428c00000-c429d00000 rw-p 00000000 00:00 0 
c429d00000-c42af00000 rw-p 00000000 00:00 0 
c42af00000-c42b000000 rw-p 00000000 00:00 0 
c42b000000-c42bf00000 rw-p 00000000 00:00 0 
c42bf00000-c42fd00000 rw-p 00000000 00:00 0 
c42fd00000-c42ff00000 rw-p 00000000 00:00 0 
c42ff00000-c431800000 rw-p 00000000 00:00 0 
c431800000-c434200000 rw-p 00000000 00:00 0 
c434200000-c435500000 rw-p 00000000 00:00 0 
c435500000-c435900000 rw-p 00000000 00:00 0 
c435900000-c436b00000 rw-p 00000000 00:00 0 
c436b00000-c439700000 rw-p 00000000 00:00 0 
c439700000-c43bc00000 rw-p 00000000 00:00 0 
c43bc00000-c43cd10000 rw-p 00000000 00:00 0 
c43cd10000-c43ea20000 rw-p 00000000 00:00 0 
c43ea20000-c43fa20000 rw-p 00000000 00:00 0 
c43fa20000-c43fc20000 rw-p 00000000 00:00 0 
c43fc20000-c440f20000 rw-p 00000000 00:00 0 
c440f20000-c441040000 rw-p 00000000 00:00 0 
c441040000-c441d40000 rw-p 00000000 00:00 0 
c441d40000-c441e60000 rw-p 00000000 00:00 0 
c441e60000-c443f60000 rw-p 00000000 00:00 0 
c443f60000-c444080000 rw-p 00000000 00:00 0 
c444080000-c444c80000 rw-p 00000000 00:00 0 
c444c80000-c4461b0000 rw-p 00000000 00:00 0 
c4461b0000-c4476b0000 rw-p 00000000 00:00 0 
c4476b0000-c447bb0000 rw-p 00000000 00:00 0 
c447bb0000-c4498e0000 rw-p 00000000 00:00 0 
c4498e0000-c44b410000 rw-p 00000000 00:00 0 
c44b410000-c453680000 rw-p 00000000 00:00 0 
7f168c000000-7f168c021000 rw-p 00000000 00:00 0 
7f168c021000-7f1690000000 ---p 00000000 00:00 0 
7f1690000000-7f1690021000 rw-p 00000000 00:00 0 
7f1690021000-7f1694000000 ---p 00000000 00:00 0 
7f1694000000-7f16942f8000 rw-p 00000000 00:00 0 
7f16942f8000-7f1698000000 ---p 00000000 00:00 0 
7f1698000000-7f1698021000 rw-p 00000000 00:00 0 
7f1698021000-7f169c000000 ---p 00000000 00:00 0 
7f169c000000-7f169c067000 rw-p 00000000 00:00 0 
7f169c067000-7f16a0000000 ---p 00000000 00:00 0 
7f16a0000000-7f16a034e000 rw-p 00000000 00:00 0 
7f16a034e000-7f16a4000000 ---p 00000000 00:00 0 
7f16a4000000-7f16a4021000 rw-p 00000000 00:00 0 
7f16a4021000-7f16a8000000 ---p 00000000 00:00 0 
7f16a8000000-7f16a8074000 rw-p 00000000 00:00 0 
7f16a8074000-7f16ac000000 ---p 00000000 00:00 0 
7f16ae7fd000-7f16ae7fe000 ---p 00000000 00:00 0 
7f16ae7fe000-7f16aeffe000 rw-p 00000000 00:00 0 
7f16aeffe000-7f16aefff000 ---p 00000000 00:00 0 
7f16aefff000-7f16af7ff000 rw-p 00000000 00:00 0 
7f16af7ff000-7f16af800000 ---p 00000000 00:00 0 
7f16af800000-7f16b0000000 rw-p 00000000 00:00 0 
7f16b0000000-7f16b004a000 rw-p 00000000 00:00 0 
7f16b004a000-7f16b4000000 ---p 00000000 00:00 0 
7f16b4000000-7f16b4082000 rw-p 00000000 00:00 0 
7f16b4082000-7f16b8000000 ---p 00000000 00:00 0 
7f16b8000000-7f16b808e000 rw-p 00000000 00:00 0 
7f16b808e000-7f16bc000000 ---p 00000000 00:00 0 
7f16bc7f9000-7f16bc7fa000 ---p 00000000 00:00 0 
7f16bc7fa000-7f16bcffa000 rw-p 00000000 00:00 0 
7f16bcffa000-7f16bcffb000 ---p 00000000 00:00 0 
7f16bcffb000-7f16bd7fb000 rw-p 00000000 00:00 0 
7f16bd7fb000-7f16bd7fc000 ---p 00000000 00:00 0 
7f16bd7fc000-7f16bdffc000 rw-p 00000000 00:00 0 
7f16bdffc000-7f16bdffd000 ---p 00000000 00:00 0 
7f16bdffd000-7f16be7fd000 rw-p 00000000 00:00 0 
7f16be7fd000-7f16be7fe000 ---p 00000000 00:00 0 
7f16be7fe000-7f16beffe000 rw-p 00000000 00:00 0 
7f16beffe000-7f16befff000 ---p 00000000 00:00 0 
7f16befff000-7f16bf7ff000 rw-p 00000000 00:00 0 
7f16bf7ff000-7f16bf800000 ---p 00000000 00:00 0 
7f16bf800000-7f16c0000000 rw-p 00000000 00:00 0 
7f16c0000000-7f16c0021000 rw-p 00000000 00:00 0 
7f16c0021000-7f16c4000000 ---p 00000000 00:00 0 
7f16c4000000-7f16c4021000 rw-p 00000000 00:00 0 
7f16c4021000-7f16c8000000 ---p 00000000 00:00 0 
7f16c8000000-7f16c808a000 rw-p 00000000 00:00 0 
7f16c808a000-7f16cc000000 ---p 00000000 00:00 0 
7f16cc000000-7f16cc035000 rw-p 00000000 00:00 0 
7f16cc035000-7f16d0000000 ---p 00000000 00:00 0 
7f16d0000000-7f16d0021000 rw-p 00000000 00:00 0 
7f16d0021000-7f16d4000000 ---p 00000000 00:00 0 
7f16d4000000-7f16d4021000 rw-p 00000000 00:00 0 
7f16d4021000-7f16d8000000 ---p 00000000 00:00 0 
7f16d87f9000-7f16d87fa000 ---p 00000000 00:00 0 
7f16d87fa000-7f16d8ffa000 rw-p 00000000 00:00 0 
7f16d8ffa000-7f16d8ffb000 ---p 00000000 00:00 0 
7f16d8ffb000-7f16d97fb000 rw-p 00000000 00:00 0 
7f16d97fb000-7f16d97fc000 ---p 00000000 00:00 0 
7f16d97fc000-7f16d9ffc000 rw-p 00000000 00:00 0 
7f16d9ffc000-7f16d9ffd000 ---p 00000000 00:00 0 
7f16d9ffd000-7f16da7fd000 rw-p 00000000 00:00 0 
7f16da7fd000-7f16da7fe000 ---p 00000000 00:00 0 
7f16da7fe000-7f16daffe000 rw-p 00000000 00:00 0 
7f16daffe000-7f16dafff000 ---p 00000000 00:00 0 
7f16dafff000-7f16db7ff000 rw-p 00000000 00:00 0 
7f16db7ff000-7f16db800000 ---p 00000000 00:00 0 
7f16db800000-7f16dc000000 rw-p 00000000 00:00 0 
7f16dc000000-7f16dc2b7000 rw-p 00000000 00:00 0 
7f16dc2b7000-7f16e0000000 ---p 00000000 00:00 0 
7f16e0000000-7f16e0276000 rw-p 00000000 00:00 0 
7f16e0276000-7f16e4000000 ---p 00000000 00:00 0 
7f16e4000000-7f16e42b9000 rw-p 00000000 00:00 0 
7f16e42b9000-7f16e8000000 ---p 00000000 00:00 0 
7f16e8000000-7f16e82e9000 rw-p 00000000 00:00 0 
7f16e82e9000-7f16ec000000 ---p 00000000 00:00 0 
7f16ec000000-7f16ec278000 rw-p 00000000 00:00 0 
7f16ec278000-7f16f0000000 ---p 00000000 00:00 0 
7f16f0000000-7f16f0246000 rw-p 00000000 00:00 0 
7f16f0246000-7f16f4000000 ---p 00000000 00:00 0 
7f16f4000000-7f16f427b000 rw-p 00000000 00:00 0 
7f16f427b000-7f16f8000000 ---p 00000000 00:00 0 
7f16f8000000-7f16f80cc000 rw-p 00000000 00:00 0 
7f16f80cc000-7f16fc000000 ---p 00000000 00:00 0 
7f16fc000000-7f16fc078000 rw-p 00000000 00:00 0 
7f16fc078000-7f1700000000 ---p 00000000 00:00 0 
7f17007f9000-7f17007fa000 ---p 00000000 00:00 0 
7f17007fa000-7f1700ffa000 rw-p 00000000 00:00 0 
7f1700ffa000-7f1700ffb000 ---p 00000000 00:00 0 
7f1700ffb000-7f17017fb000 rw-p 00000000 00:00 0 
7f17017fb000-7f17017fc000 ---p 00000000 00:00 0 
7f17017fc000-7f1701ffc000 rw-p 00000000 00:00 0 
7f1701ffc000-7f1701ffd000 ---p 00000000 00:00 0 
7f1701ffd000-7f17027fd000 rw-p 00000000 00:00 0 
7f17027fd000-7f17027fe000 ---p 00000000 00:00 0 
7f17027fe000-7f1702ffe000 rw-p 00000000 00:00 0 
7f1702ffe000-7f1702fff000 ---p 00000000 00:00 0 
7f1702fff000-7f17037ff000 rw-p 00000000 00:00 0 
7f17037ff000-7f1703800000 ---p 00000000 00:00 0 
7f1703800000-7f1704000000 rw-p 00000000 00:00 0 
7f1704000000-7f1704352000 rw-p 00000000 00:00 0 
7f1704352000-7f1708000000 ---p 00000000 00:00 0 
7f1708000000-7f1708021000 rw-p 00000000 00:00 0 
7f1708021000-7f170c000000 ---p 00000000 00:00 0 
7f170c000000-7f170c0b0000 rw-p 00000000 00:00 0 
7f170c0b0000-7f1710000000 ---p 00000000 00:00 0 
7f1710000000-7f17100df000 rw-p 00000000 00:00 0 
7f17100df000-7f1714000000 ---p 00000000 00:00 0 
7f1714000000-7f1714021000 rw-p 00000000 00:00 0 
7f1714021000-7f1718000000 ---p 00000000 00:00 0 
7f1718000000-7f1718374000 rw-p 00000000 00:00 0 
7f1718374000-7f171c000000 ---p 00000000 00:00 0 
7f171c000000-7f171c268000 rw-p 00000000 00:00 0 
7f171c268000-7f1720000000 ---p 00000000 00:00 0 
7f1720000000-7f17200b5000 rw-p 00000000 00:00 0 
7f17200b5000-7f1724000000 ---p 00000000 00:00 0 
7f1724000000-7f17240dc000 rw-p 00000000 00:00 0 
7f17240dc000-7f1728000000 ---p 00000000 00:00 0 
7f1728000000-7f17282bb000 rw-p 00000000 00:00 0 
7f17282bb000-7f172c000000 ---p 00000000 00:00 0 
7f172c7f9000-7f172c7fa000 ---p 00000000 00:00 0 
7f172c7fa000-7f172cffa000 rw-p 00000000 00:00 0 
7f172cffa000-7f172cffb000 ---p 00000000 00:00 0 
7f172cffb000-7f172d7fb000 rw-p 00000000 00:00 0 
7f172d7fb000-7f172d7fc000 ---p 00000000 00:00 0 
7f172d7fc000-7f172dffc000 rw-p 00000000 00:00 0 
7f172dffc000-7f172dffd000 ---p 00000000 00:00 0 
7f172dffd000-7f172e7fd000 rw-p 00000000 00:00 0 
7f172e7fd000-7f172e7fe000 ---p 00000000 00:00 0 
7f172e7fe000-7f172effe000 rw-p 00000000 00:00 0 
7f172effe000-7f172efff000 ---p 00000000 00:00 0 
7f172efff000-7f172f7ff000 rw-p 00000000 00:00 0 
7f172f7ff000-7f172f800000 ---p 00000000 00:00 0 
7f172f800000-7f1730000000 rw-p 00000000 00:00 0 
7f1730000000-7f1730356000 rw-p 00000000 00:00 0 
7f1730356000-7f1734000000 ---p 00000000 00:00 0 
7f1734000000-7f1734385000 rw-p 00000000 00:00 0 
7f1734385000-7f1738000000 ---p 00000000 00:00 0 
7f1738000000-7f17383b3000 rw-p 00000000 00:00 0 
7f17383b3000-7f173c000000 ---p 00000000 00:00 0 
7f173c000000-7f173c311000 rw-p 00000000 00:00 0 
7f173c311000-7f1740000000 ---p 00000000 00:00 0 
7f1740000000-7f17402a5000 rw-p 00000000 00:00 0 
7f17402a5000-7f1744000000 ---p 00000000 00:00 0 
7f17441f7000-7f17447f9000 rw-p 00000000 00:00 0 
7f17447f9000-7f17447fa000 ---p 00000000 00:00 0 
7f17447fa000-7f1744ffa000 rw-p 00000000 00:00 0 
7f1744ffa000-7f1744ffb000 ---p 00000000 00:00 0 
7f1744ffb000-7f17457fb000 rw-p 00000000 00:00 0 
7f17457fb000-7f17457fc000 ---p 00000000 00:00 0 
7f17457fc000-7f1745ffc000 rw-p 00000000 00:00 0 
7f1745ffc000-7f1745ffd000 ---p 00000000 00:00 0 
7f1745ffd000-7f17467fd000 rw-p 00000000 00:00 0 
7f17467fd000-7f17467fe000 ---p 00000000 00:00 0 
7f17467fe000-7f1746ffe000 rw-p 00000000 00:00 0 
7f1746ffe000-7f1746fff000 ---p 00000000 00:00 0 
7f1746fff000-7f17477ff000 rw-p 00000000 00:00 0 
7f17477ff000-7f1747800000 ---p 00000000 00:00 0 
7f1747800000-7f1748000000 rw-p 00000000 00:00 0 
7f1748000000-7f174809c000 rw-p 00000000 00:00 0 
7f174809c000-7f174c000000 ---p 00000000 00:00 0 
7f174c000000-7f174c386000 rw-p 00000000 00:00 0 
7f174c386000-7f1750000000 ---p 00000000 00:00 0 
7f1750000000-7f17502ec000 rw-p 00000000 00:00 0 
7f17502ec000-7f1754000000 ---p 00000000 00:00 0 
7f1754000000-7f17542e0000 rw-p 00000000 00:00 0 
7f17542e0000-7f1758000000 ---p 00000000 00:00 0 
7f1758000000-7f1758306000 rw-p 00000000 00:00 0 
7f1758306000-7f175c000000 ---p 00000000 00:00 0 
7f175c000000-7f175c0fc000 rw-p 00000000 00:00 0 
7f175c0fc000-7f1760000000 ---p 00000000 00:00 0 
7f1760000000-7f1760385000 rw-p 00000000 00:00 0 
7f1760385000-7f1764000000 ---p 00000000 00:00 0 
7f1764000000-7f1764023000 rw-p 00000000 00:00 0 
7f1764023000-7f1768000000 ---p 00000000 00:00 0 
7f1768002000-7f1768462000 rw-p 00000000 00:00 0 
7f1768469000-7f17687f9000 rw-p 00000000 00:00 0 
7f17687f9000-7f17687fa000 ---p 00000000 00:00 0 
7f17687fa000-7f1768ffa000 rw-p 00000000 00:00 0 
7f1768ffa000-7f1768ffb000 ---p 00000000 00:00 0 
7f1768ffb000-7f17697fb000 rw-p 00000000 00:00 0 
7f17697fb000-7f17697fc000 ---p 00000000 00:00 0 
7f17697fc000-7f1769ffc000 rw-p 00000000 00:00 0 
7f1769ffc000-7f1769ffd000 ---p 00000000 00:00 0 
7f1769ffd000-7f176a7fd000 rw-p 00000000 00:00 0 
7f176a7fd000-7f176a7fe000 ---p 00000000 00:00 0 
7f176a7fe000-7f176affe000 rw-p 00000000 00:00 0 
7f176affe000-7f176afff000 ---p 00000000 00:00 0 
7f176afff000-7f176b7ff000 rw-p 00000000 00:00 0 
7f176b7ff000-7f176b800000 ---p 00000000 00:00 0 
7f176b800000-7f176c000000 rw-p 00000000 00:00 0 
7f176c000000-7f176c31c000 rw-p 00000000 00:00 0 
7f176c31c000-7f1770000000 ---p 00000000 00:00 0 
7f1770000000-7f1770322000 rw-p 00000000 00:00 0 
7f1770322000-7f1774000000 ---p 00000000 00:00 0 
7f1774000000-7f1774084000 rw-p 00000000 00:00 0 
7f1774084000-7f1778000000 ---p 00000000 00:00 0 
7f1778000000-7f17780a6000 rw-p 00000000 00:00 0 
7f17780a6000-7f177c000000 ---p 00000000 00:00 0 
7f177c000000-7f177c282000 rw-p 00000000 00:00 0 
7f177c282000-7f1780000000 ---p 00000000 00:00 0 
7f1780000000-7f1780308000 rw-p 00000000 00:00 0 
7f1780308000-7f1784000000 ---p 00000000 00:00 0 
7f1784008000-7f1784148000 rw-p 00000000 00:00 0 
7f1784152000-7f1784482000 rw-p 00000000 00:00 0 
7f1784483000-7f1784723000 rw-p 00000000 00:00 0 
7f1784729000-7f17847f9000 rw-p 00000000 00:00 0 
7f17847f9000-7f17847fa000 ---p 00000000 00:00 0 
7f17847fa000-7f1784ffa000 rw-p 00000000 00:00 0 
7f1784ffa000-7f1784ffb000 ---p 00000000 00:00 0 
7f1784ffb000-7f17857fb000 rw-p 00000000 00:00 0 
7f17857fb000-7f17857fc000 ---p 00000000 00:00 0 
7f17857fc000-7f1785ffc000 rw-p 00000000 00:00 0 
7f1785ffc000-7f1785ffd000 ---p 00000000 00:00 0 
7f1785ffd000-7f17867fd000 rw-p 00000000 00:00 0 
7f17867fd000-7f17867fe000 ---p 00000000 00:00 0 
7f17867fe000-7f1786ffe000 rw-p 00000000 00:00 0 
7f1786ffe000-7f1786fff000 ---p 00000000 00:00 0 
7f1786fff000-7f17877ff000 rw-p 00000000 00:00 0 
7f17877ff000-7f1787800000 ---p 00000000 00:00 0 
7f1787800000-7f1788000000 rw-p 00000000 00:00 0 
7f1788000000-7f1788247000 rw-p 00000000 00:00 0 
7f1788247000-7f178c000000 ---p 00000000 00:00 0 
7f178c000000-7f178c02e000 rw-p 00000000 00:00 0 
7f178c02e000-7f1790000000 ---p 00000000 00:00 0 
7f1790000000-7f1790021000 rw-p 00000000 00:00 0 
7f1790021000-7f1794000000 ---p 00000000 00:00 0 
7f1794004000-7f1794134000 rw-p 00000000 00:00 0 
7f1794138000-7f1794228000 rw-p 00000000 00:00 0 
7f1794230000-7f17945f0000 rw-p 00000000 00:00 0 
7f17945f0000-7f17945f1000 ---p 00000000 00:00 0 
7f17945f1000-7f1794df1000 rw-p 00000000 00:00 0 
7f1794df1000-7f17955f1000 r--s 00000000 08:01 264910                     /var/snap/lxd/common/lxd/database/global/logs.db
7f17955f1000-7f17955fc000 r-xp 00000000 07:05 2280                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f17955fc000-7f17957fb000 ---p 0000b000 07:05 2280                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f17957fb000-7f17957fc000 r--p 0000a000 07:05 2280                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f17957fc000-7f17957fd000 rw-p 0000b000 07:05 2280                       /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f17957fd000-7f1795803000 rw-p 00000000 00:00 0 
7f1795803000-7f179580e000 r-xp 00000000 07:05 2285                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f179580e000-7f1795a0d000 ---p 0000b000 07:05 2285                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f1795a0d000-7f1795a0e000 r--p 0000a000 07:05 2285                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f1795a0e000-7f1795a0f000 rw-p 0000b000 07:05 2285                       /lib/x86_64-linux-gnu/libnss_nis-2.23.so
7f1795a0f000-7f1795a25000 r-xp 00000000 07:05 2274                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7f1795a25000-7f1795c24000 ---p 00016000 07:05 2274                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7f1795c24000-7f1795c25000 r--p 00015000 07:05 2274                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7f1795c25000-7f1795c26000 rw-p 00016000 07:05 2274                       /lib/x86_64-linux-gnu/libnsl-2.23.so
7f1795c26000-7f1795c28000 rw-p 00000000 00:00 0 
7f1795c28000-7f1795c30000 r-xp 00000000 07:05 2276                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f1795c30000-7f1795e2f000 ---p 00008000 07:05 2276                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f1795e2f000-7f1795e30000 r--p 00007000 07:05 2276                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f1795e30000-7f1795e31000 rw-p 00008000 07:05 2276                       /lib/x86_64-linux-gnu/libnss_compat-2.23.so
7f1795e31000-7f1795f91000 rw-p 00000000 00:00 0 
7f1795f91000-7f1795f92000 ---p 00000000 00:00 0 
7f1795f92000-7f1796792000 rw-p 00000000 00:00 0 
7f1796792000-7f1796793000 ---p 00000000 00:00 0 
7f1796793000-7f1796f93000 rw-p 00000000 00:00 0 
7f1796f93000-7f1796f94000 ---p 00000000 00:00 0 
7f1796f94000-7f1797794000 rw-p 00000000 00:00 0 
7f1797794000-7f1797795000 ---p 00000000 00:00 0 
7f1797795000-7f1797f95000 rw-p 00000000 00:00 0 
7f1797f95000-7f1797f96000 ---p 00000000 00:00 0 
7f1797f96000-7f1798796000 rw-p 00000000 00:00 0 
7f1798796000-7f1798797000 ---p 00000000 00:00 0 
7f1798797000-7f1798f97000 rw-p 00000000 00:00 0 
7f1798f97000-7f1798f98000 ---p 00000000 00:00 0 
7f1798f98000-7f1799798000 rw-p 00000000 00:00 0 
7f1799798000-7f179979f000 r-xp 00000000 07:05 6894                       /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7f179979f000-7f179999e000 ---p 00007000 07:05 6894                       /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7f179999e000-7f179999f000 r--p 00006000 07:05 6894                       /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7f179999f000-7f17999a0000 rw-p 00007000 07:05 6894                       /usr/lib/x86_64-linux-gnu/libffi.so.6.0.4
7f17999a0000-7f1799a0e000 r-xp 00000000 07:05 2299                       /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f1799a0e000-7f1799c0e000 ---p 0006e000 07:05 2299                       /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f1799c0e000-7f1799c0f000 r--p 0006e000 07:05 2299                       /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f1799c0f000-7f1799c10000 rw-p 0006f000 07:05 2299                       /lib/x86_64-linux-gnu/libpcre.so.3.13.2
7f1799c10000-7f1799c8f000 r-xp 00000000 07:05 6912                       /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7f1799c8f000-7f1799e8e000 ---p 0007f000 07:05 6912                       /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7f1799e8e000-7f1799e8f000 r--p 0007e000 07:05 6912                       /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7f1799e8f000-7f1799e90000 rw-p 0007f000 07:05 6912                       /usr/lib/x86_64-linux-gnu/libgmp.so.10.3.0
7f1799e90000-7f1799ec2000 r-xp 00000000 07:05 6924                       /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7f1799ec2000-7f179a0c1000 ---p 00032000 07:05 6924                       /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7f179a0c1000-7f179a0c2000 r--p 00031000 07:05 6924                       /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7f179a0c2000-7f179a0c3000 rw-p 00032000 07:05 6924                       /usr/lib/x86_64-linux-gnu/libhogweed.so.4.2
7f179a0c3000-7f179a0f7000 r-xp 00000000 07:05 6949                       /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7f179a0f7000-7f179a2f6000 ---p 00034000 07:05 6949                       /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7f179a2f6000-7f179a2f8000 r--p 00033000 07:05 6949                       /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7f179a2f8000-7f179a2f9000 rw-p 00035000 07:05 6949                       /usr/lib/x86_64-linux-gnu/libnettle.so.6.2
7f179a2f9000-7f179a30a000 r-xp 00000000 07:05 6973                       /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7f179a30a000-7f179a50a000 ---p 00011000 07:05 6973                       /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7f179a50a000-7f179a50b000 r--p 00011000 07:05 6973                       /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7f179a50b000-7f179a50c000 rw-p 00012000 07:05 6973                       /usr/lib/x86_64-linux-gnu/libtasn1.so.6.5.1
7f179a50c000-7f179a53d000 r-xp 00000000 07:05 6926                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7f179a53d000-7f179a73d000 ---p 00031000 07:05 6926                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7f179a73d000-7f179a73e000 r--p 00031000 07:05 6926                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7f179a73e000-7f179a73f000 rw-p 00032000 07:05 6926                       /usr/lib/x86_64-linux-gnu/libidn.so.11.6.15
7f179a73f000-7f179a798000 r-xp 00000000 07:05 6955                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7f179a798000-7f179a997000 ---p 00059000 07:05 6955                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7f179a997000-7f179a9a1000 r--p 00058000 07:05 6955                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7f179a9a1000-7f179a9a3000 rw-p 00062000 07:05 6955                       /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.1.0
7f179a9a3000-7f179a9bc000 r-xp 00000000 07:05 2348                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7f179a9bc000-7f179abbb000 ---p 00019000 07:05 2348                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7f179abbb000-7f179abbc000 r--p 00018000 07:05 2348                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7f179abbc000-7f179abbd000 rw-p 00019000 07:05 2348                       /lib/x86_64-linux-gnu/libz.so.1.2.8
7f179abbd000-7f179abc0000 r-xp 00000000 07:05 2210                       /lib/x86_64-linux-gnu/libdl-2.23.so
7f179abc0000-7f179adbf000 ---p 00003000 07:05 2210                       /lib/x86_64-linux-gnu/libdl-2.23.so
7f179adbf000-7f179adc0000 r--p 00002000 07:05 2210                       /lib/x86_64-linux-gnu/libdl-2.23.so
7f179adc0000-7f179adc1000 rw-p 00003000 07:05 2210                       /lib/x86_64-linux-gnu/libdl-2.23.so
7f179adc1000-7f179adc5000 r-xp 00000000 07:05 2182                       /lib/x86_64-linux-gnu/libattr.so.1.1.0
7f179adc5000-7f179afc4000 ---p 00004000 07:05 2182                       /lib/x86_64-linux-gnu/libattr.so.1.1.0
7f179afc4000-7f179afc5000 r--p 00003000 07:05 2182                       /lib/x86_64-linux-gnu/libattr.so.1.1.0
7f179afc5000-7f179afc6000 rw-p 00004000 07:05 2182                       /lib/x86_64-linux-gnu/libattr.so.1.1.0
7f179afc6000-7f179afca000 r-xp 00000000 07:05 2195                       /lib/x86_64-linux-gnu/libcap.so.2.24
7f179afca000-7f179b1ca000 ---p 00004000 07:05 2195                       /lib/x86_64-linux-gnu/libcap.so.2.24
7f179b1ca000-7f179b1cb000 r--p 00004000 07:05 2195                       /lib/x86_64-linux-gnu/libcap.so.2.24
7f179b1cb000-7f179b1cc000 rw-p 00005000 07:05 2195                       /lib/x86_64-linux-gnu/libcap.so.2.24
7f179b1cc000-7f179b1fa000 r-xp 00000000 07:05 2317                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f179b1fa000-7f179b3fa000 ---p 0002e000 07:05 2317                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f179b3fa000-7f179b410000 r--p 0002e000 07:05 2317                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f179b410000-7f179b411000 rw-p 00044000 07:05 2317                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f179b411000-7f179b430000 r-xp 00000000 07:05 2318                       /lib/x86_64-linux-gnu/libselinux.so.1
7f179b430000-7f179b62f000 ---p 0001f000 07:05 2318                       /lib/x86_64-linux-gnu/libselinux.so.1
7f179b62f000-7f179b630000 r--p 0001e000 07:05 2318                       /lib/x86_64-linux-gnu/libselinux.so.1
7f179b630000-7f179b631000 rw-p 0001f000 07:05 2318                       /lib/x86_64-linux-gnu/libselinux.so.1
7f179b631000-7f179b633000 rw-p 00000000 00:00 0 
7f179b633000-7f179b756000 r-xp 00000000 07:05 6916                       /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7f179b756000-7f179b955000 ---p 00123000 07:05 6916                       /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7f179b955000-7f179b960000 r--p 00122000 07:05 6916                       /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7f179b960000-7f179b962000 rw-p 0012d000 07:05 6916                       /usr/lib/x86_64-linux-gnu/libgnutls.so.30.6.2
7f179b962000-7f179b963000 rw-p 00000000 00:00 0 
7f179b963000-7f179bb23000 r-xp 00000000 07:05 2192                       /lib/x86_64-linux-gnu/libc-2.23.so
7f179bb23000-7f179bd23000 ---p 001c0000 07:05 2192                       /lib/x86_64-linux-gnu/libc-2.23.so
7f179bd23000-7f179bd27000 r--p 001c0000 07:05 2192                       /lib/x86_64-linux-gnu/libc-2.23.so
7f179bd27000-7f179bd29000 rw-p 001c4000 07:05 2192                       /lib/x86_64-linux-gnu/libc-2.23.so
7f179bd29000-7f179bd2d000 rw-p 00000000 00:00 0 
7f179bd2d000-7f179bdd9000 r-xp 00000000 07:06 191                        /snap/lxd/7651/lib/libsqlite3.so.0.8.6
7f179bdd9000-7f179bfd9000 ---p 000ac000 07:06 191                        /snap/lxd/7651/lib/libsqlite3.so.0.8.6
7f179bfd9000-7f179bfdb000 r--p 000ac000 07:06 191                        /snap/lxd/7651/lib/libsqlite3.so.0.8.6
7f179bfdb000-7f179bfdd000 rw-p 000ae000 07:06 191                        /snap/lxd/7651/lib/libsqlite3.so.0.8.6
7f179bfdd000-7f179bfde000 rw-p 00000000 00:00 0 
7f179bfde000-7f179bfe5000 r-xp 00000000 07:05 2176                       /lib/x86_64-linux-gnu/libacl.so.1.1.0
7f179bfe5000-7f179c1e4000 ---p 00007000 07:05 2176                       /lib/x86_64-linux-gnu/libacl.so.1.1.0
7f179c1e4000-7f179c1e5000 r--p 00006000 07:05 2176                       /lib/x86_64-linux-gnu/libacl.so.1.1.0
7f179c1e5000-7f179c1e6000 rw-p 00007000 07:05 2176                       /lib/x86_64-linux-gnu/libacl.so.1.1.0
7f179c1e6000-7f179c1fe000 r-xp 00000000 07:05 2308                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7f179c1fe000-7f179c3fd000 ---p 00018000 07:05 2308                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7f179c3fd000-7f179c3fe000 r--p 00017000 07:05 2308                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7f179c3fe000-7f179c3ff000 rw-p 00018000 07:05 2308                       /lib/x86_64-linux-gnu/libpthread-2.23.so
7f179c3ff000-7f179c403000 rw-p 00000000 00:00 0 
7f179c403000-7f179c405000 r-xp 00000000 07:05 2339                       /lib/x86_64-linux-gnu/libutil-2.23.so
7f179c405000-7f179c604000 ---p 00002000 07:05 2339                       /lib/x86_64-linux-gnu/libutil-2.23.so
7f179c604000-7f179c605000 r--p 00001000 07:05 2339                       /lib/x86_64-linux-gnu/libutil-2.23.so
7f179c605000-7f179c606000 rw-p 00002000 07:05 2339                       /lib/x86_64-linux-gnu/libutil-2.23.so
7f179c606000-7f179c6b2000 r-xp 00000000 07:06 184                        /snap/lxd/7651/lib/liblxc.so.1.4.0
7f179c6b2000-7f179c8b2000 ---p 000ac000 07:06 184                        /snap/lxd/7651/lib/liblxc.so.1.4.0
7f179c8b2000-7f179c8b4000 r--p 000ac000 07:06 184                        /snap/lxd/7651/lib/liblxc.so.1.4.0
7f179c8b4000-7f179c8b7000 rw-p 000ae000 07:06 184                        /snap/lxd/7651/lib/liblxc.so.1.4.0
7f179c8b7000-7f179c8dd000 r-xp 00000000 07:05 2170                       /lib/x86_64-linux-gnu/ld-2.23.so
7f179c8e4000-7f179c994000 rw-p 00000000 00:00 0 
7f179c994000-7f179c99c000 rw-s 00000000 08:01 264915                     /var/snap/lxd/common/lxd/database/global/db.bin-shm
7f179c99c000-7f179cad7000 rw-p 00000000 00:00 0 
7f179cadb000-7f179cadc000 rw-p 00000000 00:00 0 
7f179cadc000-7f179cadd000 r--p 00025000 07:05 2170                       /lib/x86_64-linux-gnu/ld-2.23.so
7f179cadd000-7f179cade000 rw-p 00026000 07:05 2170                       /lib/x86_64-linux-gnu/ld-2.23.so
7f179cade000-7f179cadf000 rw-p 00000000 00:00 0 
7ffc31fa1000-7ffc31fc2000 rw-p 00000000 00:00 0                          [stack]
7ffc31fd3000-7ffc31fd6000 r--p 00000000 00:00 0                          [vvar]
7ffc31fd6000-7ffc31fd8000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

ubojan on 9 Jul 2018

Running LXD built from git (commit 47bb242731a25291d4400d4660ed0e3efad420c0), I can reproduce slowness after only 40~50 containers deleted + recreated (of about 200 that are running on the machine, all idle Alpine containers). I am running Ubuntu Bionic, with go 1.10.1, no clustering.

By dropping timing Debugf()s into the code, I can observe the slow parts, eg:

func dbQueryRowScan(db *sql.DB, q string, args []interface{}, outargs []interface{}) error {
        start := time.Now().UnixNano()
        ret := query.Retry(func() error {
                return query.Transaction(db, func(tx *sql.Tx) error {
                        return tx.QueryRow(q, args...).Scan(outargs...)
                })
        })
        elapsed := time.Now().UnixNano() - start
        if (elapsed > 5000000) {
                logger.Debugf("dbQueryRowScan() took %d ms", elapsed / 1000000)
        }
        return ret
}

Destroy + recreated ~ 40 containers then:
curl -s --unix-socket /var/lib/lxd/unix.socket http://@/1.0/containers?recursion=1 > /dev/null

Logs show:

DBUG[07-09|08:52:09] Start: doContainersGet() 
DBUG[07-09|08:52:09] db.queryScan() took 88 ms 
DBUG[07-09|08:52:09] db.queryScan() took 85 ms 
DBUG[07-09|08:52:09] db.queryScan() took 84 ms 
DBUG[07-09|08:52:10] db.queryScan() took 85 ms 
DBUG[07-09|08:52:10] db.dbQueryRowScan() took 90 ms 
DBUG[07-09|08:52:10] db.dbQueryRowScan() took 86 ms 
DBUG[07-09|08:52:10] db.queryScan() took 88 ms 
DBUG[07-09|08:52:10] db.queryScan() took 90 ms 
DBUG[07-09|08:52:10] db.dbQueryRowScan() took 84 ms 
DBUG[07-09|08:52:10] db.queryScan() took 86 ms 
DBUG[07-09|08:52:10] db.queryScan() took 86 ms 
DBUG[07-09|08:52:10] db.queryScan() took 85 ms 
DBUG[07-09|08:52:10] db.queryScan() took 83 ms 
DBUG[07-09|08:52:10] db.queryScan() took 84 ms 
DBUG[07-09|08:52:11] db.queryScan() took 85 ms 
DBUG[07-09|08:52:11] db.queryScan() took 87 ms 
DBUG[07-09|08:52:11] db.queryScan() took 86 ms 
DBUG[07-09|08:52:11] db.queryScan() took 83 ms 
DBUG[07-09|08:52:11] db.queryScan() took 88 ms 
DBUG[07-09|08:52:11] db.dbQueryRowScan() took 84 ms 
DBUG[07-09|08:52:11] db.queryScan() took 85 ms 
DBUG[07-09|08:52:11] db.queryScan() took 84 ms 
DBUG[07-09|08:52:11] db.queryScan() took 86 ms 
DBUG[07-09|08:52:11] db.queryScan() took 83 ms 
DBUG[07-09|08:52:12] db.queryScan() took 256 ms 
DBUG[07-09|08:52:12] db.queryScan() took 97 ms 
DBUG[07-09|08:52:12] Inside: doContainersGet() - before sort.Slice() 
DBUG[07-09|08:52:12] End: doContainersGet()

Restart LXD and it's fast again:

DBUG[07-09|08:52:38] Start: doContainersGet() 
DBUG[07-09|08:52:38] db.queryScan() took 6 ms 
DBUG[07-09|08:52:38] db.queryScan() took 6 ms 
DBUG[07-09|08:52:38] db.dbQueryRowScan() took 6 ms 
DBUG[07-09|08:52:38] db.queryScan() took 6 ms 
DBUG[07-09|08:52:39] Inside: doContainersGet() - before sort.Slice() 
DBUG[07-09|08:52:39] End: doContainersGet()

The speed doesn't come back on it's own - if I destroy a bunch of containers, then wait, lxd will show 0% CPU after some seconds, I can wait ages, and the operations are still slow. Restart lxd, they're fast again.

If I log the query strings, it's not a single query that's slow, it's a seemingly random selection (seems to only be about 10%) of all of them, but they get slower over time as more write operations happen to the DB. Again, restart LXD, everything speeds back up again, until about the same amount of containers are destroyed+recreated.

Drilling down a bit deeper, it appears the code in lxd/db/query/transaction.go is the part that gets slow - I put timers around each of the three SQL operations inside Transaction(), db.Begin(), f(tx), and db.Commit(). Interestingly Commit is infrequently slow, but the other two are quite common:

DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 70 ms 
DBUG[07-09|11:50:26] db.queryScan('SELECT key, value FROM containers_config WHERE container_id=?') took 71 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 35 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 36 ms 
DBUG[07-09|11:50:26] query.Transaction() -> Query took 35 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 35 ms 
DBUG[07-09|11:50:26] query.Transaction() -> Query took 35 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 36 ms 
DBUG[07-09|11:50:26] query.Transaction() -> Query took 36 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 37 ms 
DBUG[07-09|11:50:26] query.Transaction() -> Query took 38 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 38 ms 
DBUG[07-09|11:50:26] query.Transaction() -> Query took 36 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 36 ms 
DBUG[07-09|11:50:26] query.Transaction() -> tx.Commit() took 35 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 36 ms 
DBUG[07-09|11:50:26] query.Transaction() -> Query took 35 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 35 ms 
DBUG[07-09|11:50:26] query.Transaction() -> tx.Commit() took 35 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 35 ms 
DBUG[07-09|11:50:26] query.Transaction() -> Query took 35 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 36 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 36 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 36 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 16 ms 
DBUG[07-09|11:50:26] query.Transaction() -> db.Begin() took 43 ms

Currently at a dead end to work out what's causing the slowness though, and I wasn't able to work out profiling go. :(

I hope it helps in some way!

fwaggle on 9 Jul 2018

Thanks, the timing above is pretty useful. @freeekanayaka is coming back to work tomorrow and has been busy working on a big rework of our database layer which I hope will fix that issue too.

@freeekanayaka can you reproduce @fwaggle's issue and then confirm that the new DB codepaths fix the performance problem?

stgraber on 9 Jul 2018

@fwaggle any chance that you can test with current LXD master after making sure you updated all other Go packages (go get -d -u all), this should include a fix to go-grpc-sql which was leaking quite a bit of memory on busy systems.

It's unclear why it'd cause that database slowness for you, but it may be some interaction with the garbage collector that could cause that after a lot of leaked memory.

At minimum you'll want to ensure that you're building LXD with -tags libsqlite3 and that $GOPATH/src/github.com/CanonicalLtd/go-grpc-sql/ is fully up to date before building the LXD binary.

Current master also allows you to run lxc config set core.debug_address 127.0.0.1:8080 which will then run an HTTP server with the pprof API reachable at http://127.0.0.1:8080/debug/pprof which you can access directly or through the pprof tool to get top CPU and memory offenders.

stgraber on 12 Jul 2018

👍1

@stgraber Cheers, I managed to mess up my dev VM and had to blow it away and rebuild from scratch. I've not done much with go before, so I'm not sure what to do with -tags, but just grabbing all master repos via go and building them (I just used make in lxc/lxd), I've rebuilt and destroyed several hundred Alpine containers without restarting LXD and it seems significantly more performant in my dubious testing.

We've got several problems with LXD 3.x, and I think most of them can be tied to this one issue. Do you know how quickly these changes will make it into a snap build?

I'm going to try and do more rigorous testing to make sure I'm not overlooking something, but I didn't want to leave you hanging in case you're still working on it. Do you still need the profile data of the faster version?

fwaggle on 12 Jul 2018

stgraber on 12 Jul 2018

🎉3

I have installed zfs (master: 2e5dc44) and lxd (master: 140c4a6). i.e. the currently latest.
The latest ZFS commits fix the crashes.

Here are some benchmarks.

Latest ZFS and LXD, launching 384 containers in batches of 24, then deleting

In the following, we can see that as containers are created, the creation speed gets slower.
It could be an issue with the containers launching (starting up as soon as they are created).

Launching

ubuntu@myserver:~$ time lxd-benchmark launch --count 384 --parallel 24
Test environment:
  Server backend: lxd
  Server version: 3.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-20-generic
  Storage backend: zfs
  Storage version: 0.7.0-1472_g2e5dc449c
  Container backend: lxc
  Container version: 3.0.1

Test variables:
  Container count: 384
  Container mode: unprivileged
  Startup mode: normal startup
  Image: ubuntu:
  Batches: 16
  Batch size: 24
  Remainder: 0

[Jul 12 19:44:41.326] Found image in local store: f2228450779fee27020d6024af587379b8f51062c32a335327f2b028c924bfa1
[Jul 12 19:44:41.327] Batch processing start
[Jul 12 19:44:54.448] Processed 24 containers in 13.121s (1.829/s)
[Jul 12 19:45:11.758] Processed 48 containers in 30.432s (1.577/s)
[Jul 12 19:45:57.044] Processed 96 containers in 75.718s (1.268/s)
[Jul 12 19:48:14.077] Processed 192 containers in 212.750s (0.902/s)
[Jul 12 19:55:58.700] Processed 384 containers in 677.373s (0.567/s)
[Jul 12 19:55:58.700] Batch processing completed in 677.373s

real    11m17,866s
user    0m6,908s
sys 0m1,208s
ubuntu@myserver:~$

Deleting

We then delete these containers. These are launched containers, therefore they are stopped and then deleted.

ubuntu@myserver:~$ time lxd-benchmark delete
Test environment:
  Server backend: lxd
  Server version: 3.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-20-generic
  Storage backend: zfs
  Storage version: 0.7.0-1472_g2e5dc449c
  Container backend: lxc
  Container version: 3.0.1

[Jul 12 19:57:10.770] Deleting 384 containers
[Jul 12 19:57:10.770] Batch processing start
[Jul 12 19:58:25.626] Processed 48 containers in 74.856s (0.641/s)
[Jul 12 19:59:29.204] Processed 96 containers in 138.434s (0.693/s)
[Jul 12 20:01:07.288] Processed 192 containers in 236.518s (0.812/s)
[Jul 12 20:02:25.931] Processed 384 containers in 315.161s (1.218/s)
[Jul 12 20:02:25.931] Batch processing completed in 315.161s

real    5m16,841s
user    0m12,703s
sys 0m1,053s
ubuntu@myserver:~$

Latest ZFS and LXD, doing `init` of 384 containers in batches of 24, then deleting

Now we try with init which does not start the containers. Obviously it should be faster than with launch. However, still, there is a lag as containers get created. Initially, the speed is 2.4s per container, then it drops to .9s per container.

Performing `init`

ubuntu@myserver:~$ time lxd-benchmark init --count 384 --parallel 24
Test environment:
  Server backend: lxd
  Server version: 3.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-20-generic
  Storage backend: zfs
  Storage version: 0.7.0-1472_g2e5dc449c
  Container backend: lxc
  Container version: 3.0.1

Test variables:
  Container count: 384
  Container mode: unprivileged
  Startup mode: normal startup
  Image: ubuntu:
  Batches: 16
  Batch size: 24
  Remainder: 0

[Jul 12 20:04:07.911] Found image in local store: f2228450779fee27020d6024af587379b8f51062c32a335327f2b028c924bfa1
[Jul 12 20:04:07.911] Batch processing start
[Jul 12 20:04:17.790] Processed 24 containers in 9.879s (2.429/s)
[Jul 12 20:04:29.576] Processed 48 containers in 21.665s (2.216/s)
[Jul 12 20:04:58.617] Processed 96 containers in 50.706s (1.893/s)
[Jul 12 20:06:22.573] Processed 192 containers in 134.662s (1.426/s)
[Jul 12 20:10:58.253] Processed 384 containers in 410.342s (0.936/s)
[Jul 12 20:10:58.253] Batch processing completed in 410.342s

real    6m50,836s
user    0m5,506s
sys 0m0,639s
ubuntu@myserver:~$

Deleting them

Now we delete these containers that are still in the stopped state.

ubuntu@myserver:~$ time lxd-benchmark delete
Test environment:
  Server backend: lxd
  Server version: 3.2
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-20-generic
  Storage backend: zfs
  Storage version: 0.7.0-1472_g2e5dc449c
  Container backend: lxc
  Container version: 3.0.1

[Jul 12 20:42:21.118] Deleting 384 containers
[Jul 12 20:42:21.118] Batch processing start
[Jul 12 20:43:24.088] Processed 48 containers in 62.971s (0.762/s)
[Jul 12 20:44:18.993] Processed 96 containers in 117.875s (0.814/s)
[Jul 12 20:45:43.474] Processed 192 containers in 202.356s (0.949/s)
[Jul 12 20:46:43.905] Processed 384 containers in 262.787s (1.461/s)
[Jul 12 20:46:43.905] Batch processing completed in 262.787s

real    4m24,286s
user    0m5,595s
sys 0m0,530s
ubuntu@myserver:~$

Stock ZFS and LXD from Ubuntu 18.04, demonstrates the problem.

Finally, for comparison, let's look back how LXD performed before these patches were landed. ZFS could work because spl_kmem_cache_slab_limit was set to 0.

Init

ubuntu@myserver:~$ time lxd-benchmark init --count 384 --parallel 24
Test environment:
  Server backend: lxd
  Server version: 3.0.1
  Kernel: Linux
  Kernel architecture: x86_64
  Kernel version: 4.15.0-20-generic
  Storage backend: zfs
  Storage version: 0.7.5-1ubuntu13
  Container backend: lxc
  Container version: 3.0.1

Test variables:
  Container count: 384
  Container mode: unprivileged
  Startup mode: normal startup
  Image: ubuntu:
  Batches: 16
  Batch size: 24
  Remainder: 0

[Jul  9 18:42:23.988] Found image in local store: f2228450779fee27020d6024af587379b8f51062c32a335327f2b028c924bfa1
[Jul  9 18:42:23.988] Batch processing start
[Jul  9 18:42:37.621] Processed 24 containers in 13.633s (1.760/s)
[Jul  9 18:42:57.610] Processed 48 containers in 33.623s (1.428/s)
[Jul  9 18:44:06.108] Processed 96 containers in 102.121s (0.940/s)
[Jul  9 18:50:04.460] Processed 192 containers in 460.472s (0.417/s)
[Jul  9 19:44:05.233] Processed 384 containers in 3701.245s (0.104/s)
[Jul  9 19:44:05.233] Batch processing completed in 3701.245s

real    61m41,756s
user    0m4,690s
sys 0m0,616s

simos on 12 Jul 2018

I just realized that I omitted the lxc list output. In general, lxc list --fast for 384 stopped LXD containers would complete in about 3.5 seconds. lxc list for 384 running LXD containers would complete in about 11 seconds.

simos on 13 Jul 2018

Stable cherry-picks have been pushed, so tentatively closing this one.

stgraber on 25 Jul 2018

Was this page helpful?

0 / 5 - 0 ratings

Related issues

avahi-daemon fails to start inside unpriviledged lxd container

iteco · 3Comments

containers-snapshots: is a directory

purell · 4Comments

lxc delete command fails "directory not empty"

fwaggle · 4Comments

dockerfile and docker-compose equivalent

srkunze · 3Comments

Unable to publish image: Bad type

jsnjack · 3Comments

Lxd: 'lxc list' huge lags, slow LXD related tasks

Required information

Issue description

Steps to reproduce

Information to attach

Most helpful comment

All 13 comments

Latest ZFS and LXD, launching 384 containers in batches of 24, then deleting

Launching

Deleting

Latest ZFS and LXD, doing init of 384 containers in batches of 24, then deleting

Performing init

Deleting them

Stock ZFS and LXD from Ubuntu 18.04, demonstrates the problem.

Init

Related issues

Latest ZFS and LXD, doing `init` of 384 containers in batches of 24, then deleting

Performing `init`