Lxd: Create log directory at service startup

[nikaro]

Required information

• Distribution: Ubuntu
• Distribution version: 17.10
• The output of "lxc info" or if that fails:

config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
api_status: stable
api_version: "1.0"
auth: trusted
public: false
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  driver: lxc
  driver_version: 2.1.0
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.13.0-19-generic
  server: lxd
  server_pid: 3903
  server_version: "2.18"
  storage: zfs
  storage_version: 0.6.5.11-1ubuntu3

Issue description

Because i have my /var/log in RAM, the directory /var/log/lxd does not exist at boot. And LXD cannot start because of this. I have to manually create it and do sudo systemctl restart lxd.service to be able to use LXD.

Steps to reproduce

1. Put /var/log in tmpfs with this in /etc/fstab:

tmpfs    /var/log    tmpfs    defaults,nosuid,nodev,noatime,nodiratime,mode=0755,size=5%    0    0

1. sudo lxd init
2. And do whatever lxc command you like, it will stuck and do nothing forever until you Ctrl+C.
3. Take a look at systemctl status lxd.

Information to attach

$ journalctl -t lxd
-- Logs begin at Mon 2017-12-11 13:37:48 CET, end at Mon 2017-12-11 13:40:21 CET. --
déc. 11 13:40:15 xps lxd[3492]: Log file path doesn't exist: /var/log/lxd
déc. 11 13:40:15 xps lxd[3492]: error: Log file path doesn't exist: /var/log/lxd

[brauner]

There's code to create those directories. This rather looks like an ordering issue on boot where LXD starts before /var/log/ is mounted. Is /var a separate partition?

[nikaro]

/var is not on a separate partition, just /var/log.

[brauner]

Hm, may suspicion is that LXD succeeds in creating the logfile on startup otherwise the daemon wouldn't have started and you wouldn't be able to send commands. So here's what likely happens:

• LXD starts before the tmpfs mount for /var/log/ specified in /etc/fstab is mounted and successfully creates the log directory.
• The tmpfs mount /var/log/ specified in /etc/fstab gets mounted and hides the underlying /var/log/lxd directory.
• LXD is confused when a subcommand is issued that calls GetLogger() and finds that the /var/log/lxd path doesn't exist and hangs.

I could send a fix that basically recreates the /var/log/lxd/ logging directory when a subcommand detects that it is missing. However, that is a nasty fix since the LXD daemon will hold an fd to the old log file in /var/log/lxd/lxd.log and you would loose all startup messages written to that old logfile also it feels rather hacky.

I suggest you add a RequiresMountsFor=/var/log dependency in your lxd.service file. That should cause systemd to only start LXD after the mount for /var/log has been setup.

[stgraber]

We specifically don't want logic in LXD to create /var/log/lxd or /var/lib/lxd/logs as LXD uses one or the other depending on which exists.

[stgraber]

Ah, looks like we have the right logdir logic in shared.LogPath so @brauner's suggested fix would work with this and we'd also be fine with LXD creating that path on startup.

[stgraber]

@nikaro in your case, I'd recommend you override the lxd.service unit (sudo systemctl edit lxd.service) and put:

[Service]
Environment="LXD_DIR=/var/lib/lxd"

Then run:

systemctl daemon-reload
systemctl restart lxd

That should cause LXD to use /var/lib/lxd/logs for log storage, avoiding the entire issue.

[brauner]

Unless, logging to a tmpfs is what is wanted in the first place. :)

[stgraber]

Yeah, in which case tweaking the units dependency is probably the best way to go here.
But LXD does pretty severe log rotation and expiry on its own, specifically because we know people are running very large amounts of containers and don't want to run out of space because of it.

[nikaro]

Thanks for your answers. It's a laptop with SSD and i don't really care about logs, so that's why i have /var/log in tmps. It's wanted :-) So i will go for units dependency tweaking.

[nikaro]

I confirm that it works. For futures googlers that may come here for the same issue, i did sudo systemctl edit lxd.service and put:

[Unit]
RequiresMountsFor=/var/log

And sudo systemctl daemon-reload && sudo systemctl restart lxd to finish.

[tdimitrov]

I hit the exactly the same problem. I use lxd on a laptop, and /var/log is tmpfs. However @nikaro 's solution didn't work for me out of the box. I added a line which creates the lxd dir. Is lxd still supposed to create its log dir?

My complete dropin:

[Service]
ExecStartPre=/bin/mkdir -p /var/log/lxd

[Unit]
RequiresMountsFor=/var/log