Lxd: Unable to make docker work correctly

[antitoine]

Required information

• Distribution: Ubuntu
• Distribution version: 16.04
• The output of "lxc info":

config: {}
api_extensions:
- id_map
api_status: stable
api_version: "1.0"
auth: trusted
public: false
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIFszCCA5ugAwIBAgIRAJ6hd9vWkZGyscGWVXxvnt8wDQYJKoZIhvcNAQELBQAw
...
    rnQ+3UXFd0ZuUXVYxOfj4r9hr2/HFKw=
    -----END CERTIFICATE-----
  certificate_fingerprint: 573e522c90917438f6ae50c04f500b1906dcf5b0b59c6b25aa60a2ff30eb72b9
  driver: lxc
  driver_version: 2.0.8
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.4.0-87-generic
  server: lxd
  server_pid: 4995
  server_version: 2.0.10
  storage: dir
  storage_version: ""

Issue description

Docker crash when the image pull process is in progress. Before (maybe an update) I didn't have the issue.

Steps to reproduce

1. lxc launch images:ubuntu/xenial/amd64 test -p default -p docker
2. lxc exec test -- apt update
3. lxc exec test -- apt install docker.io
4. lxc exec test -- docker pull hyperledger/fabric-peer:x86_64-1.0.0

Result:

lxc exec test -- docker pull hyperledger/fabric-peer:x86_64-1.0.0
x86_64-1.0.0: Pulling from hyperledger/fabric-peer
aafe6b5e13de: Pull complete 
0a2b43a72660: Pull complete 
18bdd1e546d2: Pull complete 
8198342c3e05: Pull complete 
f56970a44fd4: Pull complete 
e32b597e7839: Pull complete 
a6e362fc71c4: Extracting [==================================================>] 17.48 MB/17.48 MB
f107ea6d90f4: Download complete 
72c8e84de237: Download complete 
776cc74c9f73: Download complete 
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: operation not permitted

[Kramerican]

I can confirm I see the same following the exact steps @antitoine posted here.

Same deal if you do lxc launch ubuntu:xenial test -p default -p docker

[brauner]

Well, that doesn't necessarily look like a Docker problem. It depends what happens in one of the layers. Maybe it is trying set file capabilities. If that's the case then this won't work with this kernel. If you feel comfortable doing this you could try to make the container privileged, restart the lxd container and then try the install again. If it succeeds then this is very likely something like this.

[stgraber]

2953  <... lsetxattr resumed> )         = -1 EPERM (Operation not permitted)

[stgraber]

So it's indeed something attempting to set a file system xattr which isn't going to work in an unprivileged container.

[stgraber]

going to test with the latest upstream Docker but I expect the same to happen. The good news is that the latest upstream Docker should work properly in privileged containers, so that'd be a way around this particular kernel limitation.

[stgraber]

Exact same result with latest Docker CE in an unprivileged container. Trying Docker CE in a privileged container now.

[stgraber]

stgraber@castiana:~$ lxc launch ubuntu:16.04 test -c security.privileged=true -c security.nesting=true
Creating test
Starting test

stgraber@castiana:~$ lxc exec test bash
root@test:~# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK

root@test:~# add-apt-repository \
>    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
>    $(lsb_release -cs) \
>    stable"

root@test:~# apt update
Hit:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu xenial-security InRelease
Get:5 https://download.docker.com/linux/ubuntu xenial InRelease [38.9 kB]
Get:6 https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages [1966 B]
Fetched 40.9 kB in 0s (77.2 kB/s)   
Reading package lists... Done
Building dependency tree       
Reading state information... Done
6 packages can be upgraded. Run 'apt list --upgradable' to see them.

root@test:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  libdrm2 python3-distupgrade python3-update-manager sudo ubuntu-release-upgrader-core update-manager-core
6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 591 kB of archives.
After this operation, 9216 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 sudo amd64 1.8.16-0ubuntu1.5 [390 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 libdrm2 amd64 2.4.76-1~ubuntu16.04.1 [30.1 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 ubuntu-release-upgrader-core all 1:16.04.22 [29.5 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3-distupgrade all 1:16.04.22 [104 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python3-update-manager all 1:16.04.7 [31.7 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 update-manager-core all 1:16.04.7 [5324 B]
Fetched 591 kB in 0s (9713 kB/s)             
(Reading database ... 25785 files and directories currently installed.)
Preparing to unpack .../sudo_1.8.16-0ubuntu1.5_amd64.deb ...
Unpacking sudo (1.8.16-0ubuntu1.5) over (1.8.16-0ubuntu1.4) ...
Preparing to unpack .../libdrm2_2.4.76-1~ubuntu16.04.1_amd64.deb ...
Unpacking libdrm2:amd64 (2.4.76-1~ubuntu16.04.1) over (2.4.70-1~ubuntu16.04.1) ...
Preparing to unpack .../ubuntu-release-upgrader-core_1%3a16.04.22_all.deb ...
Unpacking ubuntu-release-upgrader-core (1:16.04.22) over (1:16.04.21) ...
Preparing to unpack .../python3-distupgrade_1%3a16.04.22_all.deb ...
Unpacking python3-distupgrade (1:16.04.22) over (1:16.04.21) ...
Preparing to unpack .../python3-update-manager_1%3a16.04.7_all.deb ...
Unpacking python3-update-manager (1:16.04.7) over (1:16.04.6) ...
Preparing to unpack .../update-manager-core_1%3a16.04.7_all.deb ...
Unpacking update-manager-core (1:16.04.7) over (1:16.04.6) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for libc-bin (2.23-0ubuntu9) ...
Setting up sudo (1.8.16-0ubuntu1.5) ...
Setting up libdrm2:amd64 (2.4.76-1~ubuntu16.04.1) ...
Setting up python3-distupgrade (1:16.04.22) ...
Setting up python3-update-manager (1:16.04.7) ...
Setting up ubuntu-release-upgrader-core (1:16.04.22) ...
Setting up update-manager-core (1:16.04.7) ...
Processing triggers for libc-bin (2.23-0ubuntu9) ...

root@test:~# apt install docker-ce
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  aufs-tools cgroupfs-mount libltdl7
Suggested packages:
  mountall
The following NEW packages will be installed:
  aufs-tools cgroupfs-mount docker-ce libltdl7
0 upgraded, 4 newly installed, 0 to remove and 6 not upgraded.
Need to get 20.6 MB of archives.
After this operation, 96.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 aufs-tools amd64 1:3.2+20130722-1.1ubuntu1 [92.9 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 cgroupfs-mount all 1.2 [4970 B]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 libltdl7 amd64 2.4.6-0.1 [38.3 kB]
Get:4 https://download.docker.com/linux/ubuntu xenial/stable amd64 docker-ce amd64 17.06.0~ce-0~ubuntu [20.5 MB]
Fetched 20.6 MB in 3s (5603 kB/s)    
Selecting previously unselected package aufs-tools.
(Reading database ... 25504 files and directories currently installed.)
Preparing to unpack .../aufs-tools_1%3a3.2+20130722-1.1ubuntu1_amd64.deb ...
Unpacking aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
Selecting previously unselected package cgroupfs-mount.
Preparing to unpack .../cgroupfs-mount_1.2_all.deb ...
Unpacking cgroupfs-mount (1.2) ...
Selecting previously unselected package libltdl7:amd64.
Preparing to unpack .../libltdl7_2.4.6-0.1_amd64.deb ...
Unpacking libltdl7:amd64 (2.4.6-0.1) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../docker-ce_17.06.0~ce-0~ubuntu_amd64.deb ...
Unpacking docker-ce (17.06.0~ce-0~ubuntu) ...
Processing triggers for libc-bin (2.23-0ubuntu9) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu19) ...
Setting up aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
Setting up cgroupfs-mount (1.2) ...
Setting up libltdl7:amd64 (2.4.6-0.1) ...
Setting up docker-ce (17.06.0~ce-0~ubuntu) ...
Processing triggers for libc-bin (2.23-0ubuntu9) ...
Processing triggers for systemd (229-4ubuntu19) ...
Processing triggers for ureadahead (0.100.0-19) ...

root@test:~# docker pull hyperledger/fabric-peer:x86_64-1.0.0
x86_64-1.0.0: Pulling from hyperledger/fabric-peer
aafe6b5e13de: Pull complete 
0a2b43a72660: Pull complete 
18bdd1e546d2: Pull complete 
8198342c3e05: Pull complete 
f56970a44fd4: Pull complete 
e32b597e7839: Pull complete 
a6e362fc71c4: Pull complete 
f107ea6d90f4: Pull complete 
72c8e84de237: Pull complete 
776cc74c9f73: Pull complete 
Digest: sha256:b7c1c2a6b356996c3dbe2b9554055cd2b63194cd7a492a83de2dbabf7f7e3c65
Status: Downloaded newer image for hyperledger/fabric-peer:x86_64-1.0.0

Closing the issue since this is a kernel limitation of unprivileged containers and not something that LXD can do much about. There is ongoing work by @hallyn to fix this kind of issue for fscaps, but I don't think that this is the issue here as it looked like a raw xattr rather than a fs capability xattr.

[vikalyan]

@stgraber This makes LXD not as useful especially in CI/CD scenarios where providing access to the Docker daemon on the host is not very safe and running privileged LXD containers is not safe either. What exactly can be done to enable this scenario without compromising on security?

[stgraber]

@vikalyan this specific failure was because of filesystem capabilities not working with unprivileged container. The kernel work by @hallyn has been merged in the upstream kernel as of 3 weeks ago and will be part of the 4.14 kernel release. So that particular issue should go away once you upgrade your system to a 4.14 kernel or newer.

[wuxxin]

@stgraber i tried with kernel 4.14 (from http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14.6/ ) on a ubuntu xenial, docker version 17.09.1-ce , lxd container started with security.nesting=true, the issue persists. same setup but with security.privileged=true it works.

[truelai]

@vikalyan this specific failure was because of filesystem capabilities not working with unprivileged container. The kernel work by @hallyn has been merged in the upstream kernel as of 3 weeks ago and will be part of the 4.14 kernel release. So that particular issue should go away once you upgrade your system to a 4.14 kernel or newer.

@stgraber I'm on 4.4.0-122-generic and this issue still exists. Is there any update on this?

root@nextcloud:/etc/apache2/sites-available# docker pull collabora/code
Using default tag: latest
latest: Pulling from collabora/code
bd97b43c27e3: Pull complete 
6960dc1aba18: Pull complete 
2b61829b0db5: Pull complete 
1f88dc826b14: Pull complete 
73b3859b1e43: Pull complete 
d1d89cabd406: Pull complete 
c02fcc31752b: Pull complete 
4594265498c5: Extracting [==================================================>] 488.5 MB/488.5 MB
failed to register layer: ApplyLayer exit status 1 stdout:  stderr: operation not permitted