Lxd: lxc/lxd not working on xenial error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied

[mbruzek]

Required information

• Distribution: Ubuntu
• Distribution version: xenial (daily build)
• The output of "lxc info" or if that fails: "error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied"
  
  
  
  • Kernel version: 4.4.0-6-generic
  
  
  • LXC version: 2.0.0.beta3
  
  
  • LXD version: 2.0.0.beta3
  
  
  • Storage backend in use: dir
    
    

    Issue description

  
  

I am trying to follow the instructions on linuxcontainers.org and many of the lxc commands fail getting the error no the unix.socket with permission denied.

$ lxc info
error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied
$ lxc list
error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied
$ lxc image list
error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied

Steps to reproduce

I followed the instructions on linuxcontainers.org to install. The deviation was I tried to use the ppa for xenial and that failed because xenial is not released yet. The lxd package was available in the xenial archive, see versions above.

Information to attach

The lxd daemon is running.

$ ps -ef | grep lxd
root      4179     1  0 08:28 ?        00:00:06 /usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log
$ sudo service lxd status
● lxd.service - Container hypervisor based on LXC
   Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
   Active: active (running) since Tue 2016-02-23 08:28:28 CST; 1h 19min ago
 Main PID: 4179 (lxd)
   CGroup: /system.slice/lxd.service
           ‣ 4179 /usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log

Feb 23 08:28:22 warhorse systemd[1]: Starting Container hypervisor based on LXC...
Feb 23 08:28:22 warhorse lxd[4179]: t=2016-02-23T08:28:22-0600 lvl=warn msg="CGroup memory swap accounting is disabl
Feb 23 08:28:28 warhorse systemd[1]: Started Container hypervisor based on LXC.

And some lxc commands work, such as: lxc image list images: but I am not able to run many of the other lxc commands on this system.

I attached the lxd log file, please let me know if you need anything more information.

[tych0]

Did you do newgrp lxd? Can you post the output of id?

[mbruzek]

Hello @tych0, yes I did run newgrp lxd, and here is the output of id:

$ id
uid=1000(mbruzek) gid=1000(mbruzek) groups=1000(mbruzek),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),114(lpadmin),130(sambashare),133(docker),134(libvirtd),136(lxd)

[stgraber]

And you've been running the lxc command from this same shell?

Can you paste the output of "ls -lh /var/lib/lxd/unix.socket" too?

[tych0]

What's the output of ls -alh /var/lib/lxd/?

[stgraber]

good to see we agree @tych0 :)

[mbruzek]

$ ls -alh /var/lib/lxd
total 88K
drwxr-xr-x  9 root root 4.0K Feb 23 08:28 .
drwxr-xr-x 90 root root 4.0K Feb 23 08:23 ..
drwx--x--x  2 root root 4.0K Feb 23 08:28 containers
drwx--x--x  2 root root 4.0K Feb 23 08:28 devices
drwxr-xr-x  2 root root 4.0K Feb 23 08:28 devlxd
drwx------  2 root root 4.0K Feb 23 08:28 images
-rw-r--r--  1 root root  38K Feb 23 08:23 lxd.db
drwx------  2 root root 4.0K Feb 23 08:28 security
-rw-r--r--  1 root root 2.2K Feb 23 08:28 server.crt
-rw-------  1 root root 3.2K Feb 23 08:28 server.key
drwx--x--x  2 root root 4.0K Feb 23 08:28 shmounts
drwx------  2 root root 4.0K Feb 23 08:28 snapshots
srw-rw----  1 root root    0 Feb 23 08:23 unix.socket

I have also read #275 looking for an answer to my problems, but no luck yet.

[stgraber]

Okay, well, that's clearly wrong :)

Can you try:

systemctl stop lxd
systemctl stop lxd.socket
systemctl start lxd.socket

See if that fixes that mess somehow?

Also, the exact version of the lxd package would be helpful.

[mbruzek]

mbruzek@warhorse:~$ systemctl stop lxd 
Warning: Stopping lxd.service, but it can still be activated by:
  lxd.socket
mbruzek@warhorse:~$ systemctl stop lxd.socket
mbruzek@warhorse:~$ systemctl start lxd.socket
mbruzek@warhorse:~$ systemctl start lxd
mbruzek@warhorse:~$ systemctl status lxd
● lxd.service - Container hypervisor based on LXC
   Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
   Active: active (running) since Tue 2016-02-23 09:58:01 CST; 8s ago
  Process: 18133 ExecStartPost=/usr/bin/lxd waitready --timeout=600 (code=exited, status=0/SUCCESS)
  Process: 18122 ExecStartPre=/usr/lib/x86_64-linux-gnu/lxc/lxc-apparmor-load (code=exited, status=0/SUCCESS)
 Main PID: 18132 (lxd)
    Tasks: 0 (limit: 512)
   Memory: 424.0K
      CPU: 19ms
   CGroup: /system.slice/lxd.service
           ‣ 18132 /usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log

Feb 23 09:58:01 warhorse systemd[1]: Starting Container hypervisor based on LXC...
Feb 23 09:58:01 warhorse lxd[18132]: t=2016-02-23T09:58:01-0600 lvl=warn msg="CGroup memory swap accounting is disab
Feb 23 09:58:01 warhorse systemd[1]: Started Container hypervisor based on LXC.
mbruzek@warhorse:~$ systemctl status lxd.socket
● lxd.socket
   Loaded: loaded (/lib/systemd/system/lxd.socket; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2016-02-23 09:57:48 CST; 33s ago
   Listen: /var/lib/lxd/unix.socket (Stream)

Feb 23 09:57:48 warhorse systemd[1]: Starting lxd.socket.
Feb 23 09:57:48 warhorse systemd[1]: Listening on lxd.socket.
mbruzek@warhorse:~$ ls -alh /var/lib/lxd
total 88K
drwxr-xr-x  9 root root 4.0K Feb 23 09:57 .
drwxr-xr-x 90 root root 4.0K Feb 23 08:23 ..
drwx--x--x  2 root root 4.0K Feb 23 08:28 containers
drwx--x--x  2 root root 4.0K Feb 23 08:28 devices
drwxr-xr-x  2 root root 4.0K Feb 23 09:58 devlxd
drwx------  2 root root 4.0K Feb 23 08:28 images
-rw-r--r--  1 root root  38K Feb 23 08:23 lxd.db
drwx------  2 root root 4.0K Feb 23 08:28 security
-rw-r--r--  1 root root 2.2K Feb 23 08:28 server.crt
-rw-------  1 root root 3.2K Feb 23 08:28 server.key
drwx--x--x  2 root root 4.0K Feb 23 08:28 shmounts
drwx------  2 root root 4.0K Feb 23 08:28 snapshots
srw-rw----  1 root lxd     0 Feb 23 09:57 unix.socket

I installed lxd from the package archive in xenial, I did not change these permissions.

$ dpkg -l | grep lxd
ii  lxd                                                  2.0.0~beta3-0ubuntu4                        amd64        Container hypervisor based on LXC - daemon
ii  lxd-client                                           2.0.0~beta3-0ubuntu4                        amd64        Container hypervisor based on LXC - client
ii  lxd-tools                                            2.0.0~beta3-0ubuntu4                        amd64        Container hypervisor based on LXC - extra tools

[stgraber]

Ok, so that fixed it.

"dpkg -l lxd" please

I need the exact package version because I've been uploading a bunch of lxd packages to the archive so I need to know which is broken exactly :)

[mbruzek]

$ dpkg -l | grep lxd
ii  lxd                                                  2.0.0~beta3-0ubuntu4                        amd64        Container hypervisor based on LXC - daemon
ii  lxd-client                                           2.0.0~beta3-0ubuntu4                        amd64        Container hypervisor based on LXC - client
ii  lxd-tools                                            2.0.0~beta3-0ubuntu4                        amd64        Container hypervisor based on LXC - extra tools

[mbruzek]

@stgraber I don't see what was fixed. Are the permissions to that directory changed? What was fixed?

[stgraber]

unix.socket is now owned by the right group which should let you talk to it.

[stgraber]

Anyway, I've got another similar maintainer script bug to investigate for upstart, I'll take a look at the source of this problem today.

[nskaggs]

I had the same problem as above, with the same installed package. After going through what you've noted above (removing the socket files, ensuring I'm in lxd group, etc), I now get an upstart error.

I don't really want to restart my session, but I wasn't in the lxd group when I checked, so I did add myself. Shouldn't the package does this step for me?

sudo status lxdstatus: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

I noticed I was using the older version of upstart, so I updated just to be sure, but the error remains.

Unpacking upstart (1.13.2-0ubuntu19) over (1.13.2-0ubuntu18) ...

$ ls -alh /var/lib/lxd/
total 88K
drwxr-xr-x  9 root root 4.0K Feb 23 17:36 .
drwxr-xr-x 96 root root 4.0K Feb 23 17:16 ..
drwx--x--x  3 root root 4.0K Feb 23 17:31 containers
drwx--x--x  3 root root 4.0K Feb 23 17:31 devices
drwxr-xr-x  2 root root 4.0K Feb 23 17:36 devlxd
drwx------  2 root root 4.0K Feb 23 17:19 images
-rw-r--r--  1 root root  38K Feb 23 17:36 lxd.db
drwx------  4 root root 4.0K Feb 23 17:31 security
-rw-r--r--  1 root root 2.0K Feb 23 17:19 server.crt
-rw-------  1 root root 3.2K Feb 23 17:19 server.key
drwx--x--x  3 root root 4.0K Feb 23 17:31 shmounts
drwx------  2 root root 4.0K Feb 23 17:19 snapshots
srw-rw----  1 root lxd     0 Feb 23 17:36 unix.socket

$ dpkg -l lxd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name              Version       Architecture  Description
+++-=================-=============-=============-=======================================
ii  lxd               2.0.0~beta3-0 amd64         Container hypervisor based on LXC - dae

[stgraber]

Above was for Xenial, are you on Xenial?

If so, the status command isn't going to work with systemd :)

[stgraber]

And you don't need to restart your session to get into the lxd group, just run "newgrp lxd" in the shell you're interacting with LXD from.

[stgraber]

The package does put you in the lxd group, but it can't do anything retroactively for existing sessions, so you need to either use newgrp or restart your session.

[nskaggs]

Yes, running xenial, sorry I should have confirmed that.

[stgraber]

ok, then don't use the "status" command, use "systemctl status" :)

[nskaggs]

Brillant, thanks. I purged, wiped /var/lib/lxd completely, then re-installed. Got the same issue again, and then was able to fix by setting the permission properly again, and doing newgrp lxd (again). Playing with adapt.

$adapt list
error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied
error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied
$ ls -alh /var/lib/lxd/
total 52K
drwxr-xr-x  2 root root 4.0K Feb 23 18:34 .
drwxr-xr-x 96 root root 4.0K Feb 23 18:34 ..
-rw-r--r--  1 root root  38K Feb 23 18:34 lxd.db
srw-rw----  1 root root    0 Feb 23 18:34 unix.socket
$ systemctl stop lxd
Warning: Stopping lxd.service, but it can still be activated by:
  lxd.socket
$ systemctl stop lxd.socket
$ systemctl start lxd.socket
$ ls -alh /var/lib/lxd/
total 52K
drwxr-xr-x  2 root root 4.0K Feb 23 18:37 .
drwxr-xr-x 96 root root 4.0K Feb 23 18:34 ..
-rw-r--r--  1 root root  38K Feb 23 18:34 lxd.db
srw-rw----  1 root lxd     0 Feb 23 18:37 unix.socket
$ adapt list
error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied
$ newgrp lxd
$ adapt list
works ;-)

[stgraber]

I think this got fixed with recent packaging updates.

[ghost]

I confirm that I experienced the exact same scenario on 15.10 (Wily) x86_64, today, with the following version of the packages from the PPA :

ii  lxd               2.0.0~rc5-0ubuntu1~ubuntu15.10.1~ppa1  amd64
ii  lxd-client        2.0.0~rc5-0ubuntu1~ubuntu15.10.1~ppa1  amd64

The following fixed the socket permissions issue :

sudo systemctl stop lxd
sudo systemctl stop lxd.socket
sudo systemctl start lxd.socket

Looks like this could still be an issue for new PPA users on 15.10.

[stgraber]

I guess if we get more reports of this we'll file a systemd bug, because there's nothing that LXD or our packaging does which would explain this behavior...

[fenris]

1 fenris@x240:~⟫ cat /etc/issue
Ubuntu 16.04 LTS n l

fenris@x240:~⟫ uname -a
Linux x240 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
fenris@x240:~⟫ lxc --version
2.0.0
fenris@x240:~⟫ lxc list
error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied
1 fenris@x240:~⟫ sudo lxc list
+--------+---------+------+------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+--------+---------+------+------+------------+-----------+
| ubuntu | STOPPED | | | PERSISTENT | 0 |
+--------+---------+------+------+------------+-----------+
fenris@x240:~⟫ sudo lxc start ubuntu
error: Error calling 'lxd forkstart ubuntu /var/lib/lxd/containers /var/log/lxd/ubuntu/lxc.conf': err='exit status 1'
Try lxc info --show-log ubuntu for more info
1 fenris@x240:~⟫ lxc info --show-log
error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied

[patademahesh]

I think your LXD daemon is not running. try to start LXD service using

sudo service lxd restart

Mahesh

[stgraber]

sudo chown root:lxd /var/lib/lxd/unix.socket

That should fix it, assuming that you already made sure that you are indeed in the right group and ran "newgrp lxd" or restarted your session.

There is some kind of systemd race occasionally causing wrong ownership of the socket, it seems to be very rare and only happen on initial installation...

[danmac-uk]

+1 just seen this on 16.04 i386 installed from release ISO, stopping lxd and restarting lxd.socket as above worked for me. Did not need to manually set with chown.

[dankegel]

Confirmed, still happening, the following still works around it:

sudo systemctl stop lxd.socket
sudo systemctl start lxd.socket

or alternately the chown.

It's happened every time I tried it tonight, very repeatable.

It happens quite repeatably here. I was able to use lxd the other day, but today
every time I install it, the socket is owned by root with group root rather than lxd.

It happens during postinst during the call to deb-systemd-helper enable lxd.service.

dank@server:~$ id
uid=1000(dank) gid=131(lxd) groups=131(lxd),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),113(lpadmin),128(sambashare),130(docker),1000(dank)
dank@server:~$ ls -l /var/lib/lxd/unix.socket
ls: cannot access '/var/lib/lxd/unix.socket': No such file or directory
dank@server:~$ apt-cache policy lxd
lxd:
  Installed: (none)
  Candidate: 2.0.0-0ubuntu4
  Version table:
     2.0.0-0ubuntu4 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
dank@server:~$ sudo apt-get install lxd
...
dank@server:~$ ls -l /var/lib/lxd/unix.socket
srw-rw---- 1 root root 0 Apr 29 20:28 /var/lib/lxd/unix.socket

[dankegel]

Filed https://bugs.launchpad.net/ubuntu/+source/lxd/+bug/1577001

[SimplySeth]

Ubuntu 16.04.2 LTS

lxd 2.0.9

lxd-client 2.0.9

I did the newgrp lxd command BTW

I get this when doing sudo lxd init

error: Unable to talk to LXD: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: connection refused

/var/lib/lxd looks like this

drwx--x--x 2 root lxd 6 Mar 15 15:43 containers drwx--x--x 2 root lxd 6 Mar 15 15:43 devices drwxr-xr-x 2 root lxd 6 Mar 15 15:44 devlxd drwx------ 2 root lxd 6 Mar 15 15:43 images -rw-r--r-- 1 root lxd 41K Mar 15 15:44 lxd.db drwx------ 2 root lxd 6 Mar 15 15:43 security -rw-r--r-- 1 root lxd 1.9K Mar 15 15:43 server.crt -rw------- 1 root lxd 3.2K Mar 15 15:43 server.key drwx--x--x 2 root lxd 6 Mar 15 15:43 shmounts drwx------ 2 root lxd 6 Mar 15 15:43 snapshots srw-rw---- 1 root lxd 0 Mar 15 16:01 unix.socket -rw------- 1 root lxd 2.0G Mar 15 16:01 zfs.img

[Gemstone123]

i'm facing this issue:
$ lxc network get lxdbr0 ipv4.address
Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied

even on this:
$ lxc list
Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: permission denied

can anyone help me out??

[stgraber]

@Gemstone123 sounds like you're either not in in the lxd group and so don't have access to it.

[dankegel]

I've found on 18.04 that I have to reboot (not just log out and back in) after installing lxd and adding my user to the lxd group.

Gemstone, verify that the output of 'id' shows your session knows you're in the lxd group.

[Gemstone123]

@stgraber oh i see, but i added myself to users. don't know why facing this issue then.

[Gemstone123]

@dankegel the output of "id" is:
uid=1000(munir) gid=1000(munir) groups=1000(munir),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),116(lpadmin),126(sambashare),999(lxd)

[dankegel]

You probably want to post a question at https://discuss.linuxcontainers.org/
Be sure to give your operating system and lxd versions.

[adrian5]

Just want to report back, that I just ran into this on a fresh 18.04 (VPS) with the 3.21 snap. Rebooting made no difference.

With Ubuntu packages, I expect them to "just work" upon installation, without requiring even small manual tasks like user/group changes.

[stgraber]

The snap does not and will not add user to the group, so that's expected behavior.
Any user we add to that group effectively gets root access on the system, so we're not going to start adding random users on installation :)

For the deb, it was a bit different as it was Ubuntu-specific and so could assume the behavior of the sudoers group for example. The snap is installed by many distros and we don't want to accidentally create a giant security issue.

[adrian5]

I see, that's fair enough.