Runtime: Does Kata-conatiner support aufs/overlay filesystem?

Created on 24 Feb 2020 · 22Comments · Source: kata-containers/runtime

Description of problem

I run a kata-container and install docker inside it , found that default filesystem in docker is vfs, which is slow and want to use aufs/overlay

Expected result

Support aufs/overlay

Actual result

root@c43d1b58e8e1:/# docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.6
 Storage Driver: vfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.75-44.container
 Operating System: Ubuntu 18.04.3 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.95GiB
 Name: c43d1b58e8e1
 ID: UCXF:FA7D:VQKK:QPBW:2SIZ:TPHB:FWMZ:ODY3:AWEE:5GVB:HX2R:XDPE
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

When I change it to overlay by setting daemon.json to

{
  "storage-driver": "overlay2"
}

the docker error is

time="2020-02-24T06:48:29.687283268Z" level=error msg="'overlay2' is not supported over overlayfs" storage-driver=overlay2
time="2020-02-24T06:48:29.692526734Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
time="2020-02-24T06:48:29.692853627Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
failed to start daemon: error initializing graphdriver: backing file system is unsupported for this graph driver

bug needs-review

Source

YongHaoWu

Most helpful comment

@YongHaoWu If you want to restart containers, you need something persistent to store the data. Maybe you can try to add a block device to your container and mount it to /var/lib/docker in your init script inside the DinD container, with the --device option.

bergwolf on 17 Mar 2020

😄1 👍1

All 22 comments

To be clear then, you are running docker inside a Kata container :-)
Looking at our kernel config (fragments), it looks like OVERLAY is enabled, specifically to allow docker to run:
https://github.com/kata-containers/packaging/blob/master/kernel/configs/fragments/common/fs.conf#L48
That code was added over on this PR, which might be worth a read:
https://github.com/kata-containers/packaging/pull/716

@egernst - do you know the current state of running Docker inside Kata?

grahamwhaley on 24 Feb 2020

@grahamwhaley Hi, I had ready tried overlay before and still got the error. Just paste the overlay2 for bug report. Wait me update the storage driver. Sry for the mistake. Thanks @grahamwhaley

YongHaoWu on 24 Feb 2020

root@bde3fec11d66:/# cat /etc/docker/daemon.json
{
  "storage-driver": "overlay"
}
root@bde3fec11d66:/# service docker start
 * Starting Docker: docker                                                                          [ OK ]
root@bde3fec11d66:/# docker info
Client:
 Debug Mode: false

Server:
ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
errors pretty printing info

And the error is failed to start daemon: error initializing graphdriver: backing file system is unsupported for this graph driver

Full docker log is:

time="2020-02-24T10:38:57.875084586Z" level=info msg="containerd successfully booted in 0.064627s"
time="2020-02-24T10:38:57.912231782Z" level=warning msg="unable to modify root key limit, number of containers could be limited by this quota: open /proc/sys/kernel/keys/root_maxkeys: no such file or directory"
time="2020-02-24T10:38:57.927444606Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2020-02-24T10:38:57.927794591Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2020-02-24T10:38:57.927990871Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
time="2020-02-24T10:38:57.928522377Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2020-02-24T10:38:57.934537817Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2020-02-24T10:38:57.934648967Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2020-02-24T10:38:57.934786659Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
time="2020-02-24T10:38:57.934895038Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2020-02-24T10:38:57.947041899Z" level=warning msg="[graphdriver] WARNING: the overlay storage-driver is deprecated, and will be removed in a future release"
time="2020-02-24T10:38:57.950995568Z" level=error msg="'overlay' is not supported over overlayfs" storage-driver=overlay
time="2020-02-24T10:38:57.955319706Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
time="2020-02-24T10:38:57.955835037Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
failed to start daemon: error initializing graphdriver: backing file system is unsupported for this graph driver

YongHaoWu on 24 Feb 2020

I wish we really did have DinK (Docker-In-Kata) documented - we have an Issue open for it, with some useful discussion (you may wish to reference if you did not already @YongHaoWu ) over at https://github.com/kata-containers/runtime/issues/1429#issuecomment-477385283

Any takers for making a formal markdown doc for this in the Kata docs repo?

grahamwhaley on 24 Feb 2020

@gabibeyer Seems #1429 (comment) is just mount kata container's host's /var/run to the kata container, which is not perfect as containers all share the same /var/run.

YongHaoWu on 24 Feb 2020

@YongHaoWu We had added support for overlay as far as I remember, let me try to reproduce your issue.

@grahamwhaley Agree we need docs for DinK, I suppose I can work on that since I will be taking a look at this issue.

amshinde on 26 Feb 2020

@amshinde hi, any feedback? Can u reproduce the issue? thanks.

YongHaoWu on 2 Mar 2020

The issue you're seeing is running overlay on top of virtio-fs. We are working to add support for this, but today there is a limitation in the checking that is done by the overlayfs driver.

As a reference, please see this discussion on the kata-dev mailing list: http://lists.katacontainers.io/pipermail/kata-dev/2020-January/001209.html

See this issue for more details:
https://github.com/kata-containers/runtime/issues/1888

I think it would make sense to close this as a duplicate of #1888 and continue with comments/details on there? WDYT @YongHaoWu?

egernst on 2 Mar 2020

@egernst Hi, how can I ensure that I was running on top of virtio-fs?

I did run kata container on VM, does it matter?

YongHaoWu on 3 Mar 2020

It depends on how you have setup your snapshotter/graph driver (if you are using overlayfs, for example, for docker/CRI), and then it depends on your Kata configuration.toml.

See https://github.com/kata-containers/runtime/blob/master/cli/config/configuration-qemu.toml.in#L106-L109 for the toml setting I am referring to.

egernst on 3 Mar 2020

@egernst Hi, I had checked my configure file and it just used the default value virtio-9p of shared_fs. So is virtio-9p had the same issue as virtio-fs?

YongHaoWu on 4 Mar 2020

@egernst @amshinde Hi, sry to bother u. Any resp?

YongHaoWu on 8 Mar 2020

@YongHaoWu yes virtio-9p fails the same reason as virtio-fs -- they cannot be used as overlayfs top layers. There is ongoing work in upstream kernel community to make it work for virtio-fs but it requires some time to be there.

bergwolf on 10 Mar 2020

@YongHaoWu Any chance if you can give https://github.com/kata-containers/runtime/issues/1429#issuecomment-477385283 a try? It puts guest /var/run to guest memory tmpfs and should fix the issue. Nothing is mounted on the host.

bergwolf on 12 Mar 2020

@bergwolf Aah, sry for my misunderstanding. I don't quite know the method @grahamwhaley provide before.

#1429 (comment) seems a beautiful solution. Thanks.

Will try later.

YongHaoWu on 12 Mar 2020

@bergwolf Thanks, it works.

root@localhost:~# docker run  --privileged     --mount type=tmpfs,destination=/var/lib/docker,tmpfs-mode=1770     -it --memory 5g --cpus 8  harbor.shopeemobile.com/cloud/virtualdev:ubuntu1604

root@c3408a616395:/#
root@c3408a616395:/# service docker start
 * Starting Docker: docker                                                                                                                                      [ OK ]
root@c3408a616395:/# docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.6
 Storage Driver: overlay2
  Backing Filesystem: tmpfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 35bd7a5f69c13e1563af8a93431411cd9ecf5021
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.15-45.container
 Operating System: Ubuntu 16.04.6 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 6.95GiB
 Name: c3408a616395
 ID: Z5IL:N2FW:36UX:XKAH:D72O:ZQVC:AAQN:LWT6:ME2H:JRMU:RA7B:RUBD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
root@c3408a616395:/# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:f9dfddf63636d84ef479d645ab5885156ae030f611a56f3a7ac7f2fdd86d7e4e
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

root@c3408a616395:/# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@c3408a616395:/# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
ef62f7872b6d        hello-world         "/hello"            13 seconds ago      Exited (0) 11 seconds ago                       peaceful_tesla

YongHaoWu on 17 Mar 2020

@bergwolf @grahamwhaley But mount /var/lib/docker as a tmpfs filesystem has a issue, if the container was restarted, all the docker container data were lost.

Any good idea to solve this?

YongHaoWu on 17 Mar 2020

bergwolf on 17 Mar 2020

😄1 👍1

@bergwolf I do use this method in runc container before. Just wonder whether there are better methods in Kata runtime. Thanks.

YongHaoWu on 17 Mar 2020

Given that we have a workaround and nothing to fix we are closing this issue.