Runtime: Run docker in kata container

Created on 1 Jun 2018 · 111Comments · Source: kata-containers/runtime

Description of problem

Tried using the https://hub.docker.com/r/hyperhq/docker-in-hyper/ image and instructions, however, it seems to fail due to a missing filesystem driver in the kernel (don't have aufs, overlay, or overlay2 in the container)

Expected result

I believe that a "docker-in-kata" use case should function -- ideally without having to supply the privileged flag -- similar to hyper containers. This is really critical for running docker in our CI/CD, etc.

Actual result

All of these are running with kata-runtime 1.0:

root@katadev-01:~# docker run -it --rm -P --privileged hyperhq/docker-in-hyper
WARN[0000] [!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting -tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!] 
INFO[0000] libcontainerd: new containerd process, pid: 26 
ERRO[0001] devmapper: Udev sync is not supported. This will lead to data loss and unexpected behavior. Install a more recent version of libdevmapper or select a different storage driver. For more information, see https://docs.docker.com/engine/reference/commandline/daemon/#daemon-storage-driver-option 
ERRO[0001] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded. 
FATA[0001] Error starting daemon: invalid argument      
root@katadev-01:~#

AND

root@katadev-01:~# docker run -it --privileged docker:dind
WARN[2018-06-01T02:04:45.951057616Z] could not change group /var/run/docker.sock to docker: group docker not found 
WARN[2018-06-01T02:04:45.957701238Z] [!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!] 
INFO[2018-06-01T02:04:45.976487211Z] libcontainerd: started new docker-containerd process  pid=13
INFO[0000] starting containerd                           module=containerd revision=773c489c9c1b21a6d78b5c538cd395416ec50f88 version=v1.0.3
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
WARN[0000] failed to load plugin io.containerd.metadata.v1.bolt  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1
WARN[0000] failed to load plugin io.containerd.differ.v1.walking  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"...  module=containerd type=io.containerd.gc.v1
WARN[0000] failed to load plugin io.containerd.gc.v1.scheduler  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.containers  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.content"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.content  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.diff  error="could not load required differ due plugin init error: walking: invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.events"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.images  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.leases  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.namespaces  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.snapshots  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"...  module=containerd type=io.containerd.monitor.v1
INFO[0000] loading plugin "io.containerd.runtime.v1.linux"...  module=containerd type=io.containerd.runtime.v1
WARN[0000] failed to load plugin io.containerd.runtime.v1.linux  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.tasks  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
INFO[0000] containerd successfully booted in 0.078145s   module=containerd
ERRO[2018-06-01T02:04:46.237294234Z] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.  storage-driver=overlay2
ERRO[2018-06-01T02:04:46.300917461Z] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.  storage-driver=overlay
ERRO[2018-06-01T02:04:46.301453650Z] Failed to built-in GetDriver graph devicemapper /var/lib/docker 
Error starting daemon: error while opening volume store metadata database: invalid argument
root@katadev-01:~#

I can't find the kata-collect-data.sh script on my machine and running it directly from github source with bash fails... I installed kata via the debian package for Ubuntu 16.04

feature

Source

svarlamov

👍1

Most helpful comment

Thanks @grahamwhaley, I appreciate that.

If it is useful I would be happy to do a more complete write up / walkthrough on both docker-in-kata and kata-in-docker. Not sure when I can have that done but I think that kata-in-docker in particular is a cool way to test out kata-containers without the need to mess with (and potentially break) a new user's containerization setup. I imagine that a three line "try it out risk free" could help the project grow.

That said, I should likely wait for a few people to try out the current approach. It works on my machine but that doesn't mean much. Also, I am not sure how amenable the kata-containers project would be to enabling any or all of the kernel features I turned on to make docker-in-kata work. I really respect how minimal the current kata kernel build is. If any changes derived from my work were to be accepted I assume I would need to

get this working on the 4.19 kernel (should be easy),
find and document the minimum set of kernel features strictly necessary to run nested docker (I didn't want to rebuild a million times so I just turned on basically all of the nf_tables / nat / netfilter features),
determine if we actually want to include btrfs support (I found docker build to be completely broken with the 9p / overlay approach but that doesn't mean we must use btrfs),
write some real docs (my stream of thought notes are likely not quite good enough :nerd_face:),
write some tests.

I am happy to do any and all of that, I would just want to confirm that there is interest first :smile_cat:.

I also just wrote up a quick future plans section with some details of other things I would like to try. I would be interested to know of other people's use cases for docker-in-kata or kata-in-docker. My focus is on networking and distributed systems but I am sure other people have different goals we might want to account for.

Again, thanks to everyone for all their hard work on kata-containers. I have really enjoyed playing with this project :smiley:

daniel-noland on 26 Nov 2019

👍4

All 111 comments

Thanks for the issue, @svarlamov. In this case, its s because the default kernel provided by docker doesn't have these configs enabled, AFAICT. We do provide a method for building your own and specifying a custom kernel. It seems to me that this should meet your needs. Or at least get you past this issue!

We don't enable "everything" by default, as for most the current config works and added features just impact potential density for all.

egernst on 1 Jun 2018

@egernst Thank you for the explanation -- will build my own kernel then. Thanks!

svarlamov on 1 Jun 2018

I had/have the same issue. I've been able to start dockerd in a kata container after doing the following:

Rebuild the guest kernel with overlayfs and a bunch of iptables/netfilter modules, following https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#install-guest-kernel-images. Otherwise, besides the overlay issue, docker complains that it cannot find some Netfilter NAT modules.
Start the host/kata-enabled Docker daemon with Devicemapper storage, by adding this to my /etc/docker/daemon.json:

"storage-driver": "devicemapper",
    "storage-opts": [
      "dm.directlvm_device=/dev/data/katana",
      "dm.thinp_percent=95",
      "dm.thinp_metapercent=2",
      "dm.thinp_autoextend_threshold=80",
      "dm.thinp_autoextend_percent=20",
      "dm.directlvm_device_force=false"
  ]

Start my container hosting the nested Docker with specific options. In particular, for the Netfilter stuff, iptables calls the modprobe command, even if I compiled the modules statically. modprobe requires access to lib/modules/kernel_release/modules.builtin generated during custom kernel build. Also, I had to grant more capabilities to the container:

docker run -it --cap-add=NET_ADMIN --cap-add=SYS_ADMIN --cap-add=SYS_RESOURCE --security-opt apparmor=unconfined -v ./modules.builtin.4.14.22:/lib/modules/4.14.22/modules.builtin alpine sh

Within the container, generate modules.dep from the mounted modules.builtin:

depmod -ae

(This step can probably be done from the host where the kernel was compiled, and modules.dep be mounted like modules.builtin).

Still within the container:

mount -o remount,rw /proc/sys
umount /proc/keys /proc/timer_list /proc/sched_debug /proc/sched_debug /proc/scsi /proc/irq /proc/sys /proc/bus /proc/fs

Then install Docker-ce, and run the daemon:

Now, it starts. But I cannot run any container, it always fail with

docker: Error response from daemon: cgroups: cannot find cgroup mount destination: unknown.

Would it be possible to have an option in kata-runtime for skipping any custom setup it does related to procfs and cgroups? It would boot the kernel and leave these untouched, which should make the kata container behave more like a regular VM.

m-barthelemy on 3 Jun 2018

@m-barthelemy Thank you for your explanation! I haven't had a chance to dig much deeper into this yet -- have you gotten the procfs and cgroups stuff sorted?

svarlamov on 11 Jun 2018

Also wondering here -- it seems that we're trying to really docker in docker, as opposed to connecting to the daemon at the VM level via a shared sock/tcp. Is there any possibility of that? Does it even make sense?

svarlamov on 11 Jun 2018

I really would like this function out of mainline too. One of the long outstanding issues in the community is integrating container building in a secure way. You can see all sorts of docs like: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html where there are lots of different ways to deploy it, each with their major drawbacks. There just isn't a good solution currently.

But some way to give a kata container to a user while giving them privilege would allow it to start fast, run fast, and get torn down quickly while still providing security via vm level isolation would be awesome.

Alternately, buildah seems to be maturing rapidly and might not need as much from the runtime as docker may. So maybe at least getting that tool up to snuff in kata might work too.

kfox1111 on 12 Jun 2018

👍2

@egernst Since you have a bit more background on this proj, are you able to share a brief outline of what changes are likely required to make this happen?

svarlamov on 25 Jun 2018

hey @svarlamov @egernst , I'd be really interested in a little more information on how to get this use-case to work!
I attempted to get this to work by building my own gust kernel with overlay-fs support enabled - but still no luck, dockerd does not start... (I used the default `docker/dind`` images)
So any Information on what I'm missing is dearly welcome :-)

enzian on 26 Jun 2018

Hi @enzian - Could you paste the error you get?

It seems the issue you are all seeing is that the guest environment is not providing sufficient functionality to run docker. That isn't surprising given we try to keep it as minimal as possible. But that said, we're considering ways to provide for the myriad features/services folk need.

A thought has occurred to me: lxc (like docker) creates Linux containers. But lxc (unlike docker afaik) provides a really useful command (lxc-checkconfig) you can run to determine if your system is capable of creating an lxc container.

As such, although it's not guaranteed to help your run a docker container, trying something like the following might be enlightening:

$ sudo docker run -ti ubuntu sh
# apt-get update && apt-get -y install lxc
# lxc-checkconfig

ftr, I just tried this, but the command needs to be able to find the kernel config. That either means you'll have to make /proc/config.gz available to the Ubuntu environment, or you'll have to enable a debug console to run that lxc command in the root level guest namespace:

https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#set-up-a-debug-console

jodh-intel on 26 Jun 2018

currently this is where I am:

WARN[2018-06-26T11:21:07.176131843Z] could not change group /var/run/docker.sock to docker: group docker not found
INFO[2018-06-26T11:21:07.183109507Z] libcontainerd: started new docker-containerd process  pid=10
INFO[0000] starting containerd                           module=containerd revision=773c489c9c1b21a6d78b5c538cd395416ec50f88 version=v1.0.3
ERRO[0000] failed to change OOM score to -500            error="write /proc/10/oom_score_adj: permission denied" module=containerd
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containe
rd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io
.containerd.metadata.v1.bolt"
WARN[0000] failed to load plugin io.containerd.metadata.v1.bolt  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1
WARN[0000] failed to load plugin io.containerd.differ.v1.walking  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"...  module=containerd type=io.containerd.gc.v1
WARN[0000] failed to load plugin io.containerd.gc.v1.scheduler  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.containers  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.content"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.content  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.diff  error="could not load required differ due plugin init error: walking: invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.events"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.images  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.leases  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.namespaces  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.snapshots  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"...  module=containerd type=io.containerd.monitor.v1
INFO[0000] loading plugin "io.containerd.runtime.v1.linux"...  module=containerd type=io.containerd.runtime.v1
WARN[0000] failed to load plugin io.containerd.runtime.v1.linux  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.tasks  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
INFO[0000] containerd successfully booted in 0.060983s   module=containerd
Error starting daemon: error while opening volume store metadata database: invalid argument

enzian on 26 Jun 2018

Thanks @enzian - a few things jump out:

you either need to add support for btrfs, or stop docker trying to use it ;)
you need to create a docker group.
writing to /proc/self/oom_score_adj works, but not if the number is negative as docker seems to be trying to do.

jodh-intel on 26 Jun 2018

By the way guys, are you running docker in debug mode? I don't see any debug messages in the log snippets so I suspect you aren't. Enabling might give more clues :)

https://success.docker.com/article/how-do-i-enable-debug-logging-of-the-docker-daemon

jodh-intel on 26 Jun 2018

ok, patched in btrfs, mounted modules.builtin added a group named docker ahead of running dockerd --oom-score-adjust 500 and got:

INFO[2018-06-26T13:11:19.705657148Z] libcontainerd: started new docker-containerd process  pid=56
INFO[0000] starting containerd                           module=containerd revision=773c489c9c1b21a6d78b5c538cd395416ec50f88 version=v1.0.3
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containe
rd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io
.containerd.metadata.v1.bolt"
WARN[0000] failed to load plugin io.containerd.metadata.v1.bolt  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1
WARN[0000] failed to load plugin io.containerd.differ.v1.walking  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"...  module=containerd type=io.containerd.gc.v1
WARN[0000] failed to load plugin io.containerd.gc.v1.scheduler  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.containers  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.content"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.content  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.diff  error="could not load required differ due plugin init error: walking: invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.events"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.images  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.leases  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.namespaces  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.snapshots  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"...  module=containerd type=io.containerd.monitor.v1
INFO[0000] loading plugin "io.containerd.runtime.v1.linux"...  module=containerd type=io.containerd.runtime.v1
WARN[0000] failed to load plugin io.containerd.runtime.v1.linux  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.tasks  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
INFO[0000] containerd successfully booted in 0.060515s   module=containerd
INFO[2018-06-26T13:11:19.813457318Z] [graphdriver] using prior storage driver: overlay2
Error starting daemon: error while opening volume store metadata database: invalid argument

enzian on 26 Jun 2018

Hi @enzian - great - definite progress there :) It might be worth keeping a gist of how you did what you did.

Did you enable debug as I still don't see any DEBUG-level messages in the above?

jodh-intel on 26 Jun 2018

sorry, for that @jodh-intel one more with debug logs:

# dockerd --oom-score-adjust 500 --log-level debug
DEBU[2018-06-26T13:56:29.055959879Z] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[2018-06-26T13:56:29.060197970Z] libcontainerd: started new docker-containerd process  pid=45
INFO[0000] starting containerd                           module=containerd revision=773c489c9c1b21a6d78b5c538cd395416ec50f88 version=v1.0.3
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containe
rd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io
.containerd.metadata.v1.bolt"
WARN[0000] failed to load plugin io.containerd.metadata.v1.bolt  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1
WARN[0000] failed to load plugin io.containerd.differ.v1.walking  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"...  module=containerd type=io.containerd.gc.v1
WARN[0000] failed to load plugin io.containerd.gc.v1.scheduler  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.containers  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.content"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.content  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.diff  error="could not load required differ due plugin init error: walking: invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.events"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.images  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.leases  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.namespaces  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.snapshots  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"...  module=containerd type=io.containerd.monitor.v1
INFO[0000] loading plugin "io.containerd.runtime.v1.linux"...  module=containerd type=io.containerd.runtime.v1
WARN[0000] failed to load plugin io.containerd.runtime.v1.linux  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.tasks  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
INFO[0000] containerd successfully booted in 0.024696s   module=containerd
DEBU[2018-06-26T13:56:29.105001716Z] Golang's threads limit set to 14310
DEBU[2018-06-26T13:56:29.109639616Z] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2018-06-26T13:56:29.111562933Z] Using default logging driver json-file
DEBU[2018-06-26T13:56:29.112648560Z] [graphdriver] priority list: [btrfs zfs overlay2 aufs overlay devicemapper vfs]
DEBU[2018-06-26T13:56:29.119353124Z] backingFs=extfs,  projectQuotaSupported=false  storage-driver=overlay2
INFO[2018-06-26T13:56:29.119389092Z] [graphdriver] using prior storage driver: overlay2
DEBU[2018-06-26T13:56:29.119402452Z] Initialized graph driver overlay2
DEBU[2018-06-26T13:56:29.128428824Z] Cleaning up old mountid : start.
Error starting daemon: error while opening volume store metadata database: invalid argument

It does not leave me any smarted ... ;-)

enzian on 26 Jun 2018

oh, and a gist of how I got it this situation :p

enzian on 26 Jun 2018

Thanks @enzian. The error at the end is coming from boltdb:

https://github.com/moby/moby/blob/master/volume/service/store.go#L93

A quick test shows that bolt.Open() is failing on an mmap(MAP_SHARED):

255   mmap(NULL, 32768, PROT_READ, MAP_SHARED, 3, 0) = -1 EINVAL (Invalid argument)

That I think is:

https://github.com/clearcontainers/runtime/issues/852

What might be worth exploring is a rootPath set to "":

https://github.com/moby/moby/blob/master/volume/service/store.go#L83

/cc @sboeuf, @grahamwhaley.

jodh-intel on 26 Jun 2018

Ah, we've seen a BoltDB issue under VM containers before - so, this may not be specific to docker-under-kata. We wanted to add it to our 'popular images test', but it looks like there was no image.
Ah, see also: https://github.com/clearcontainers/runtime/issues/523

I think really @mcastelino and @egernst are the folks to ask, but both are unavailable right now I think for comment.

grahamwhaley on 26 Jun 2018

Thanks @grahamwhaley.

@GabyCT - did you try https://hub.docker.com/r/rossriley/docker-bolt/?

jodh-intel on 26 Jun 2018

@jodh-intel no, I think that we were looking for boltdb official release at docker hub

GabyCT on 26 Jun 2018

@jodh-intel, I tried the image from https://hub.docker.com/r/rossriley/docker-bolt/ but I do not think that is what we are looking for as a replacement for boltdb. The image https://hub.docker.com/r/rossriley/docker-bolt/ does not have go , bolt or boltd.

GabyCT on 26 Jun 2018

interesting... https://github.com/clearcontainers/runtime/issues/852 implies its a lack of feature in 9p.

Is there any workarounds we could do? emptyDir maybe? Is every file system passed through 9p?

kfox1111 on 26 Jun 2018

buildah-bud might be a workaround here too, as it may not use boltdb.

kfox1111 on 26 Jun 2018

Hi @kfox1111 . All filesystem maps from the VM to the host are via 9p unless you use a graph driver on the host that supports block device mode, and then Kata can map that block mount directly into the VM. I thought we had an Issue open asking for us to document how you enable that (@amshinde - are you willing to write that up - how to set up and use block devices with kata?) Using a block backed graph driver might work @kfox1111 . You'll need I believe to use devicemapper as the graph driver on the host.
Another route we have used inside containers is to use a RAMFS/TMPFS inside the container. We used to just mount a TMPFS, but we have locked down the containers a bit more now, so you are not normally allowed to mount inside the container - one workaround you can see here was to set the TMPDIR to the /dev/shm path for instance ;-) . That will also depend on if dind can run with that data held only inside the container, and not mapped back onto the host?

grahamwhaley on 26 Jun 2018

@jodh-intel, @egernst @WeiZhang555 @sboeuf - I think we should add any known 9p limitations to the limitations doc - wdyt?

grahamwhaley on 26 Jun 2018

@grahamwhaley I had added some documentation for block storage for virtcontainers library itself:
https://github.com/kata-containers/runtime/tree/master/virtcontainers#storage.
But I think that needs to be moved somewhere more accessible, with some more details added in.
I'll get that done today.

amshinde on 26 Jun 2018

@grahamwhaley I tried your approach to use devicemapper as dockers storage driver:

Append "storage-driver": "devicemapper" to /etc/docker/daemon.json.
Run the container: sudo docker run --rm -it -v ${tmpdir}/linux-${kernel_version}/modules.builtin:/lib/modules/4.14.22/modules.builtin docker:dind sh
From within the container, add the docker group: addgroup -S -g 1000 docker
Start the docker daemon: dockerd --oom-score-adjust 500 --log-level debug

but nothing changed:

DEBU[2018-06-27T06:42:04.879906182Z] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[2018-06-27T06:42:04.884528932Z] libcontainerd: started new docker-containerd process  pid=11
INFO[0000] starting containerd                           module=containerd revision=773c489c9c1b21a6d78b5c538cd395416ec50f88 version=v1.0.3
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containe
rd.metadata.v1.bolt"
WARN[0000] failed to load plugin io.containerd.metadata.v1.bolt  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1
WARN[0000] failed to load plugin io.containerd.differ.v1.walking  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"...  module=containerd type=io.containerd.gc.v1
WARN[0000] failed to load plugin io.containerd.gc.v1.scheduler  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.containers  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.content"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.content  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.diff  error="could not load required differ due plugin init error: walking: invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.events"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.images  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.leases  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.namespaces  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.snapshots  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"...  module=containerd type=io.containerd.monitor.v1
INFO[0000] loading plugin "io.containerd.runtime.v1.linux"...  module=containerd type=io.containerd.runtime.v1
WARN[0000] failed to load plugin io.containerd.runtime.v1.linux  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.tasks  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
INFO[0000] containerd successfully booted in 0.081603s   module=containerd
DEBU[2018-06-27T06:42:05.027178985Z] Golang's threads limit set to 14310
DEBU[2018-06-27T06:42:05.046806952Z] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2018-06-27T06:42:05.057989341Z] Using default logging driver json-file
DEBU[2018-06-27T06:42:05.062461103Z] [graphdriver] priority list: [btrfs zfs overlay2 aufs overlay devicemapper vfs]
DEBU[2018-06-27T06:42:05.065507881Z] cannot open /dev/zfs: open /dev/zfs: no such file or directory  storage-driver=zfs
DEBU[2018-06-27T06:42:05.084589779Z] backingFs=extfs,  projectQuotaSupported=false  storage-driver=overlay2
DEBU[2018-06-27T06:42:05.085792363Z] Initialized graph driver overlay2
DEBU[2018-06-27T06:42:05.112173884Z] Cleaning up old mountid : start.
Error starting daemon: error while opening volume store metadata database: invalid argument

enzian on 27 Jun 2018

I'm not set up here to try out devicemapper - @amshinde , can I ask you to have a look at this output to see if there is anything obvious?
@enzian - can you run a mount inside the container and paste the output here - I believe we should see the filesystems mounted as block devices (when using devicemapper), rather than as 9p mounts - let's check that as a first step to ensure the host side devicemapper got enabled and picked up by the Kata runtime...

grahamwhaley on 27 Jun 2018

@grahamwhaley - agreed - go for it! ;)

jodh-intel on 27 Jun 2018

@grahamwhaley

/ # mount
/dev/sda on / type xfs (rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
kataShared on /etc/resolv.conf type 9p (rw,sync,dirsync,nodev,relatime,access=client,trans=virtio)
kataShared on /etc/hostname type 9p (rw,sync,dirsync,nodev,relatime,access=client,trans=virtio)
kataShared on /etc/hosts type 9p (rw,sync,dirsync,nodev,relatime,access=client,trans=virtio)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
kataShared on /var/lib/docker type 9p (rw,sync,dirsync,nodev,relatime,access=client,trans=virtio)
kataShared on /lib/modules/4.14.22/modules.builtin type 9p (rw,sync,dirsync,nodev,relatime,access=client,trans=virtio)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
proc on /proc/bus type proc (ro,relatime)
proc on /proc/fs type proc (ro,relatime)
proc on /proc/irq type proc (ro,relatime)
proc on /proc/sys type proc (ro,relatime)
tmpfs on /proc/keys type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/timer_list type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/sched_debug type tmpfs (rw,nosuid,size=65536k,mode=755)
tmpfs on /proc/scsi type tmpfs (ro,relatime)
tmpfs on /sys/firmware type tmpfs (ro,relatime)

enzian on 27 Jun 2018

Thanks @enzian I think the line:

/dev/sda on / type xfs (rw,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota)

shows you have a block mount as your primary container mount point via devicemapper, correct @amshinde ?

Looking at your failure case, it would seem from:

DEBU[2018-06-27T06:42:05.062461103Z] [graphdriver] priority list: [btrfs zfs overlay2 aufs overlay devicemapper vfs]
DEBU[2018-06-27T06:42:05.065507881Z] cannot open /dev/zfs: open /dev/zfs: no such file or directory  storage-driver=zfs
DEBU[2018-06-27T06:42:05.084589779Z] backingFs=extfs,  projectQuotaSupported=false  storage-driver=overlay2
DEBU[2018-06-27T06:42:05.085792363Z] Initialized graph driver overlay2
DEBU[2018-06-27T06:42:05.112173884Z] Cleaning up old mountid : start.
Error starting daemon: error while opening volume store metadata database: invalid argument

that docker (in the container?) has tried btrfs and then zfs, and failed on zfs. Why it does not then continue onto overlay2 etc. I don't know. I'm thinking there might be more things needing to be enabled in the VM kernel here.

grahamwhaley on 27 Jun 2018

@grahamwhaley thanks for the feedback - If you can give me a hint as to what feature toggles to enable in the kernel, then I'd like to try this and report back here - I just have absolutely no clue what it could be!

enzian on 27 Jun 2018

@enzian, @m-barthelemy, @svarlamov - could you confirm which version of docker you are using in the container btw?

jodh-intel on 27 Jun 2018

I would be tempted to try the lxc hack above as it might provide some clues.

jodh-intel on 27 Jun 2018

yeah, devicemapper isn't really an option for us. been down that path and ended up with corrupted volumes. Rather not go down that path again.

Ramfs might be a bit tight for some container builds too.

Any possibility of making temp space available as a disk volume? Maybe detect k8s emptyDirs and write a qcow2 file into them instead and pass that through as a disk?

kfox1111 on 27 Jun 2018

Here's a pastebin of a run of buildah in a container (not kata). Can someone with overlay in kernel try a variant of it in kata?
https://pastebin.com/vALfxGU1

kfox1111 on 27 Jun 2018

@enzian You have devicemapper set up correctly on the host since your rootfs is mounted on /dev/sda. But looks like docker ends up using overlay2 inside the guest.
Can you check that output of sudo docker info | grep Storage inside Kata Container.

You may need to setup libdevmapper and dmsetup inside the Kata container. After you do that and start docker, can you check the output of dmsetup ls to see if devicemapper pool is set up.

amshinde on 27 Jun 2018

@kfox1111 What corruption have you seen with devicemapper? Can you explain detailed steps?

amshinde on 27 Jun 2018

unfortunately not. I had it happen to me on two separate machines in close succession but was unable to reproduce it enough to file a bug report. Its been long enough ago now that I don't want to misremember and state some behaviour it had that may be incorrect. I switched to overlay2 at that point just to work around it and haven't seen the issue since. But am very weary of devicemapper with containers now.

kfox1111 on 27 Jun 2018

hey @amshinde, I'd love to run docker info but, not having dockerd running it gives me attitude about not being able to access docker.sock ;-)
I managed to install device-mapper in the containers but launching dockerd with the corresponding configuration:

# dockerd --oom-score-adjust 500 --log-level debug
WARN[0000] Running experimental build
DEBU[2018-07-02T06:32:31.770153424Z] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[2018-07-02T06:32:31.773967934Z] libcontainerd: started new docker-containerd process  pid=220
INFO[0000] starting containerd                           module=containerd revision=773c489c9c1b21a6d78b5c538cd395416ec50f88 version=v1.0.3
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containe
rd.metadata.v1.bolt"
WARN[0000] failed to load plugin io.containerd.metadata.v1.bolt  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1
WARN[0000] failed to load plugin io.containerd.differ.v1.walking  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"...  module=containerd type=io.containerd.gc.v1
WARN[0000] failed to load plugin io.containerd.gc.v1.scheduler  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.containers  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.content"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.content  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.diff  error="could not load required differ due plugin init error: walking: invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.events"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.images  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.leases  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.namespaces  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.snapshots  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"...  module=containerd type=io.containerd.monitor.v1
INFO[0000] loading plugin "io.containerd.runtime.v1.linux"...  module=containerd type=io.containerd.runtime.v1
WARN[0000] failed to load plugin io.containerd.runtime.v1.linux  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.tasks  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
INFO[0000] containerd successfully booted in 0.047492s   module=containerd
DEBU[2018-07-02T06:32:31.835245814Z] Golang's threads limit set to 14310
DEBU[2018-07-02T06:32:31.847837557Z] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2018-07-02T06:32:31.851672227Z] Using default logging driver json-file
DEBU[2018-07-02T06:32:31.852598655Z] [graphdriver] trying provided driver: devicemapper
WARN[2018-07-02T06:32:31.854073950Z] Unable to locate plugin: devicemapper, retrying in 1s
WARN[2018-07-02T06:32:32.855076214Z] Unable to locate plugin: devicemapper, retrying in 2s
WARN[2018-07-02T06:32:34.856382681Z] Unable to locate plugin: devicemapper, retrying in 4s
WARN[2018-07-02T06:32:38.857287651Z] Unable to locate plugin: devicemapper, retrying in 8s
^CINFO[2018-07-02T06:32:41.520027316Z] Processing signal 'interrupt'
ERRO[2018-07-02T06:32:46.857994381Z] Failed to GetDriver graph                     driver=devicemapper error="Error looking up graphdriver plugin devicemapper: plugin \"devicemapper\" not found" home-dir=/var/lib/docker
DEBU[2018-07-02T06:32:46.859716729Z] Cleaning up old mountid : start.
Error starting daemon: error initializing graphdriver: driver not supported

enzian on 2 Jul 2018

@enzian can you please try the vfs driver? Its the slowest, but has the highest chance of working. https://docs.docker.com/storage/storagedriver/vfs-driver/

kfox1111 on 2 Jul 2018

@kfox1111 switched to vfs inside the container:

DEBU[2018-07-06T07:19:48.642112275Z] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[2018-07-06T07:19:48.649073528Z] libcontainerd: started new docker-containerd process  pid=14
INFO[0000] starting containerd                           module=containerd revision=773c489c9c1b21a6d78b5c538cd395416ec50f88 version=v1.0.3
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
WARN[0000] failed to load plugin io.containerd.metadata.v1.bolt  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1
WARN[0000] failed to load plugin io.containerd.differ.v1.walking  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"...  module=containerd type=io.containerd.gc.v1
WARN[0000] failed to load plugin io.containerd.gc.v1.scheduler  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.containers  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.content"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.content  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.diff  error="could not load required differ due plugin init error: walking: invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.events"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.images  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.leases  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.namespaces  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.snapshots  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"...  module=containerd type=io.containerd.monitor.v1
INFO[0000] loading plugin "io.containerd.runtime.v1.linux"...  module=containerd type=io.containerd.runtime.v1
WARN[0000] failed to load plugin io.containerd.runtime.v1.linux  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"...  module=containerd type=io.containerd.grpc.v1
WARN[0000] failed to load plugin io.containerd.grpc.v1.tasks  error="invalid argument" module=containerd
INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
INFO[0000] containerd successfully booted in 0.059385s   module=containerd
DEBU[2018-07-06T07:19:48.822054127Z] Golang's threads limit set to 14310
DEBU[2018-07-06T07:19:48.833974987Z] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2018-07-06T07:19:48.839030867Z] Using default logging driver json-file
DEBU[2018-07-06T07:19:48.839578302Z] [graphdriver] trying provided driver: vfs
DEBU[2018-07-06T07:19:48.842562174Z] Initialized graph driver vfs
DEBU[2018-07-06T07:19:48.869836976Z] Cleaning up old mountid : start.
Error starting daemon: error while opening volume store metadata database: invalid argument

enzian on 6 Jul 2018

Am I missing more kernel features?

enzian on 6 Jul 2018

Hi @enzian - well, we're still stuck on the original boltdb issue. This looks like a known 9p FS + mmap option issue so until 9p can either be fixed/replaced, I think your own options are:

Explore that docker root option I mentioned above.
Try forcing docker to run in a non-9p mount.

Something like:
```
# mount -t ramfs -o size=10M ramfs /var/lib/docker/volumes
# mount -t ramfs -o size=10M ramfs /var/lib/docker/builder
```
See https://github.com/intel/cc-oci-runtime/issues/669#issuecomment-281280979

jodh-intel on 6 Jul 2018

@enzian ok, thanks for trying.

Anyone know if buildah uses boltdb?

kfox1111 on 6 Jul 2018

@jodh-intel I'm wondering how Hyper containers deal with this issue. Is there any way to loop one of those folks into this discussion?

svarlamov on 7 Jul 2018

Sure we can @svarlamov - @bergwolf, any thoughts or insight on this?

grahamwhaley on 9 Jul 2018

@svarlamov @grahamwhaley right now, docker in hyper is supported, I thought we had added some kernel modules to support them. Here are some issues in hyperstart:

https://github.com/hyperhq/hyperstart/issues/80
https://github.com/hyperhq/hyperstart/issues/72 (https://github.com/hyperhq/hyperstart/pull/79)
https://github.com/hyperhq/hyperstart/pull/225

gnawux on 9 Jul 2018

Getting closer here (see the updated gist):

Took a load of modules (mostly NF) and added them to the kernel config
Recompiled the kernel
Run the docker:dind image with SYS-ADMIN capabilities ()
Mount /var/lib/docker with a block-device as @jodh-intel suggested (which seems to work quite fine)
Run dockerd which goes so much further now:

INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
INFO[0000] containerd successfully booted in 0.027018s   module=containerd
DEBU[2018-07-13T06:47:09.531482853Z] Golang's threads limit set to 14310
DEBU[2018-07-13T06:47:09.537972946Z] Using default logging driver json-file
DEBU[2018-07-13T06:47:09.538286817Z] [graphdriver] priority list: [btrfs zfs overlay2 aufs overlay devicemapper vfs]
DEBU[2018-07-13T06:47:09.539349538Z] cannot open /dev/zfs: open /dev/zfs: no such file or directory  storage-driver=zfs
DEBU[2018-07-13T06:47:09.541229271Z] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2018-07-13T06:47:09.550576484Z] backingFs=ramfs,  projectQuotaSupported=false  storage-driver=overlay2
DEBU[2018-07-13T06:47:09.550810204Z] Initialized graph driver overlay2
INFO[2018-07-13T06:47:09.621527090Z] Graph migration to content-addressability took 0.00 seconds
WARN[2018-07-13T06:47:09.622755518Z] Your kernel does not support swap memory limit
WARN[2018-07-13T06:47:09.622946704Z] Your kernel does not support cgroup rt period
WARN[2018-07-13T06:47:09.622963083Z] Your kernel does not support cgroup rt runtime
WARN[2018-07-13T06:47:09.623049676Z] Your kernel does not support cgroup blkio weight
WARN[2018-07-13T06:47:09.623223364Z] Your kernel does not support cgroup blkio weight_device
WARN[2018-07-13T06:47:09.623390384Z] Your kernel does not support cgroup blkio throttle.read_bps_device
WARN[2018-07-13T06:47:09.623403047Z] Your kernel does not support cgroup blkio throttle.write_bps_device
WARN[2018-07-13T06:47:09.623415221Z] Your kernel does not support cgroup blkio throttle.read_iops_device
WARN[2018-07-13T06:47:09.623466557Z] Your kernel does not support cgroup blkio throttle.write_iops_device
DEBU[2018-07-13T06:47:09.625593013Z] Max Concurrent Downloads: 3
DEBU[2018-07-13T06:47:09.625756973Z] Max Concurrent Uploads: 5
INFO[2018-07-13T06:47:09.627967783Z] Loading containers: start.
DEBU[2018-07-13T06:47:09.628179309Z] Option Experimental: false
DEBU[2018-07-13T06:47:09.628264577Z] Option DefaultDriver: bridge
DEBU[2018-07-13T06:47:09.628276518Z] Option DefaultNetwork: bridge
DEBU[2018-07-13T06:47:09.628284065Z] Network Control Plane MTU: 1500
DEBU[2018-07-13T06:47:09.632997932Z] processing event stream                       module=libcontainerd namespace=moby
WARN[2018-07-13T06:47:09.639083921Z] Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'
ip: can't find device 'br_netfilter'
modprobe: module bridge not found in modules.dep
modprobe: module br_netfilter not found in modules.dep
, error: exit status 1
WARN[2018-07-13T06:47:09.645567994Z] Running modprobe nf_nat failed with message: `ip: can't find device 'nf_nat'
modprobe: module nf_nat not found in modules.dep`, error: exit status 1
WARN[2018-07-13T06:47:09.653101747Z] Running modprobe xt_conntrack failed with message: `ip: can't find device 'xt_conntrack'
modprobe: module xt_conntrack not found in modules.dep`, error: exit status 1
DEBU[2018-07-13T06:47:09.654061904Z] Fail to initialize firewalld: Failed to connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory, using raw iptables instead
DEBU[2018-07-13T06:47:09.660296171Z] /sbin/iptables, [-t filter -C FORWARD -j DOCKER-ISOLATION]
DEBU[2018-07-13T06:47:09.664465139Z] /sbin/iptables, [-t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2018-07-13T06:47:09.667547265Z] /sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[2018-07-13T06:47:09.669072394Z] /sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2018-07-13T06:47:09.670798284Z] /sbin/iptables, [-t nat -D PREROUTING]
DEBU[2018-07-13T06:47:09.674591078Z] /sbin/iptables, [-t nat -D OUTPUT]
DEBU[2018-07-13T06:47:09.676467037Z] /sbin/iptables, [-t nat -F DOCKER]
DEBU[2018-07-13T06:47:09.678155960Z] /sbin/iptables, [-t nat -X DOCKER]
DEBU[2018-07-13T06:47:09.679489806Z] /sbin/iptables, [-t filter -F DOCKER]
DEBU[2018-07-13T06:47:09.680933665Z] /sbin/iptables, [-t filter -X DOCKER]
DEBU[2018-07-13T06:47:09.684677424Z] /sbin/iptables, [-t filter -F DOCKER-ISOLATION-STAGE-1]
DEBU[2018-07-13T06:47:09.686065086Z] /sbin/iptables, [-t filter -X DOCKER-ISOLATION-STAGE-1]
DEBU[2018-07-13T06:47:09.688605360Z] /sbin/iptables, [-t filter -F DOCKER-ISOLATION-STAGE-2]
DEBU[2018-07-13T06:47:09.689990247Z] /sbin/iptables, [-t filter -X DOCKER-ISOLATION-STAGE-2]
DEBU[2018-07-13T06:47:09.693836543Z] /sbin/iptables, [-t filter -F DOCKER-ISOLATION]
DEBU[2018-07-13T06:47:09.695340101Z] /sbin/iptables, [-t filter -X DOCKER-ISOLATION]
DEBU[2018-07-13T06:47:09.696342424Z] /sbin/iptables, [-t nat -n -L DOCKER]
DEBU[2018-07-13T06:47:09.697088819Z] /sbin/iptables, [-t nat -N DOCKER]
DEBU[2018-07-13T06:47:09.697747869Z] daemon configured with a 15 seconds minimum shutdown timeout
DEBU[2018-07-13T06:47:09.697758917Z] start clean shutdown of all containers with a 15 seconds timeout...
DEBU[2018-07-13T06:47:09.698840331Z] Cleaning up old mountid : start.
DEBU[2018-07-13T06:47:09.699094248Z] Cleaning up old mountid : done.
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.1: can't initialize iptables table `nat': Table
does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

Which says to me, that I messed up in the kernel config because assuming I've added all the NF*IPTABLE modules... Does anyone have any pointers?

enzian on 13 Jul 2018

looks you need more netfilter related modules

gnawux on 13 Jul 2018

Do you have these items: CONFIG_BRIDGE_NETFILTER, CONFIG_NF_NAT, CONFIG_NETFILTER_XT_MATCH_CONNTRACK? looks they were mentioned in the log messages.

gnawux on 13 Jul 2018

I finally had some time to get back to this: I managed to enable the kernel modules as described by @gnawux:

CONFIG_BRIDGE_NETFILTER=y
CONFIG_NF_NAT=y
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y

full .config file here, but dockerds start I still get similar errors about iptables being missing:

INFO[0000] Start snapshots syncer
INFO[0000] Start streaming server
DEBU[2018-07-27T10:51:08.586193506Z] Golang's threads limit set to 14310
INFO[2018-07-27T10:51:08.593459822Z] parsed scheme: "unix"                         module=grpc
INFO[2018-07-27T10:51:08.595175121Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2018-07-27T10:51:08.595902125Z] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0  <nil>}]  module=grpc
INFO[2018-07-27T10:51:08.596158424Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2018-07-27T10:51:08.596777835Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420180400, CONNECTING  module=grpc
INFO[2018-07-27T10:51:08.600335064Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420180400, READY  module=grpc
DEBU[2018-07-27T10:51:08.602593198Z] Using default logging driver json-file
DEBU[2018-07-27T10:51:08.605071843Z] [graphdriver] priority list: [btrfs zfs overlay2 aufs overlay devicemapper vfs]
DEBU[2018-07-27T10:51:08.608032893Z] cannot open /dev/zfs: open /dev/zfs: no such file or directory  storage-driver=zfs
DEBU[2018-07-27T10:51:08.610534551Z] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2018-07-27T10:51:08.628725721Z] backingFs=ramfs,  projectQuotaSupported=false  storage-driver=overlay2
DEBU[2018-07-27T10:51:08.629500403Z] Initialized graph driver overlay2
INFO[2018-07-27T10:51:08.665734795Z] Graph migration to content-addressability took 0.00 seconds
WARN[2018-07-27T10:51:08.667214778Z] Your kernel does not support swap memory limit
WARN[2018-07-27T10:51:08.667377176Z] Your kernel does not support cgroup rt period
WARN[2018-07-27T10:51:08.667435627Z] Your kernel does not support cgroup rt runtime
WARN[2018-07-27T10:51:08.667449712Z] Your kernel does not support cgroup blkio weight
WARN[2018-07-27T10:51:08.667459918Z] Your kernel does not support cgroup blkio weight_device
WARN[2018-07-27T10:51:08.667469622Z] Your kernel does not support cgroup blkio throttle.read_bps_device
WARN[2018-07-27T10:51:08.667483770Z] Your kernel does not support cgroup blkio throttle.write_bps_device
WARN[2018-07-27T10:51:08.667495822Z] Your kernel does not support cgroup blkio throttle.read_iops_device
WARN[2018-07-27T10:51:08.667507108Z] Your kernel does not support cgroup blkio throttle.write_iops_device
DEBU[2018-07-27T10:51:08.671575992Z] Max Concurrent Downloads: 3
DEBU[2018-07-27T10:51:08.671873638Z] Max Concurrent Uploads: 5
INFO[2018-07-27T10:51:08.673290366Z] parsed scheme: "unix"                         module=grpc
INFO[2018-07-27T10:51:08.673584821Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2018-07-27T10:51:08.674438487Z] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0  <nil>}]  module=grpc
INFO[2018-07-27T10:51:08.674460036Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2018-07-27T10:51:08.674738917Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420181570, CONNECTING  module=grpc
INFO[2018-07-27T10:51:08.676941537Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420181570, READY  module=grpc
INFO[2018-07-27T10:51:08.679796523Z] Loading containers: start.
DEBU[2018-07-27T10:51:08.682193952Z] Option Experimental: false
DEBU[2018-07-27T10:51:08.682346081Z] Option DefaultDriver: bridge
DEBU[2018-07-27T10:51:08.682363063Z] Option DefaultNetwork: bridge
DEBU[2018-07-27T10:51:08.682371733Z] Network Control Plane MTU: 1500
DEBU[2018-07-27T10:51:08.701616705Z] processing event stream                       module=libcontainerd namespace=moby
WARN[2018-07-27T10:51:08.709271923Z] Running modprobe nf_nat failed with message: `ip: can't find device 'nf_nat'
modprobe: module nf_nat not found in modules.dep`, error: exit status 1
WARN[2018-07-27T10:51:08.729028521Z] Running modprobe xt_conntrack failed with message: `ip: can't find device 'xt_conntrack'
modprobe: module xt_conntrack not found in modules.dep`, error: exit status 1
DEBU[2018-07-27T10:51:08.732745943Z] Fail to initialize firewalld: Failed to connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory, using raw iptables instead
DEBU[2018-07-27T10:51:08.751954433Z] /sbin/iptables, [-t filter -C FORWARD -j DOCKER-ISOLATION]
DEBU[2018-07-27T10:51:08.762499411Z] /sbin/iptables, [-t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2018-07-27T10:51:08.772759882Z] /sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[2018-07-27T10:51:08.783451874Z] /sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2018-07-27T10:51:08.794465186Z] /sbin/iptables, [-t nat -D PREROUTING]
DEBU[2018-07-27T10:51:08.811103951Z] /sbin/iptables, [-t nat -D OUTPUT]
DEBU[2018-07-27T10:51:08.820874051Z] /sbin/iptables, [-t nat -F DOCKER]
DEBU[2018-07-27T10:51:08.830884780Z] /sbin/iptables, [-t nat -X DOCKER]
DEBU[2018-07-27T10:51:08.841733347Z] /sbin/iptables, [-t filter -F DOCKER]
DEBU[2018-07-27T10:51:08.850840873Z] /sbin/iptables, [-t filter -X DOCKER]
DEBU[2018-07-27T10:51:08.862800442Z] /sbin/iptables, [-t filter -F DOCKER-ISOLATION-STAGE-1]
DEBU[2018-07-27T10:51:08.872978487Z] /sbin/iptables, [-t filter -X DOCKER-ISOLATION-STAGE-1]
DEBU[2018-07-27T10:51:08.880716019Z] /sbin/iptables, [-t filter -F DOCKER-ISOLATION-STAGE-2]
DEBU[2018-07-27T10:51:08.892484987Z] /sbin/iptables, [-t filter -X DOCKER-ISOLATION-STAGE-2]
DEBU[2018-07-27T10:51:08.901295087Z] /sbin/iptables, [-t filter -F DOCKER-ISOLATION]
DEBU[2018-07-27T10:51:08.911585780Z] /sbin/iptables, [-t filter -X DOCKER-ISOLATION]
DEBU[2018-07-27T10:51:08.922621000Z] /sbin/iptables, [-t nat -n -L DOCKER]
DEBU[2018-07-27T10:51:08.932371665Z] /sbin/iptables, [-t nat -N DOCKER]
DEBU[2018-07-27T10:51:08.943944919Z] daemon configured with a 15 seconds minimum shutdown timeout
DEBU[2018-07-27T10:51:08.945088467Z] start clean shutdown of all containers with a 15 seconds timeout...
DEBU[2018-07-27T10:51:08.948254460Z] Cleaning up old mountid : start.
DEBU[2018-07-27T10:51:08.949412294Z] Cleaning up old mountid : done.
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.1: can't initialize iptables table `nat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

enzian on 27 Jul 2018

Hi @enzian - Are you specifying docker run -cap-add=CAP_NET_ADMIN --privileged when you start the Kata container?

jodh-intel on 27 Jul 2018

hey @jodh-intel, not exactly but almost:

sudo docker run --rm -it --cap-add=SYS_ADMIN --cap-add=NET_ADMIN -v ${tmpdir}/linux-${kernel_version}/modules.builtin:/lib/modules/4.14.22/modules.builtin docker:dind sh

but if I add the privileged flag I get an OCI error:

docker: Error response from daemon: OCI runtime create failed: QMP command failed: unknown.

PS: I also switched to 1.1.0 version of kata!

enzian on 27 Jul 2018

Hi @enzian - oops - sorry, yes we don't (cannot) support privileged mode (https://github.com/kata-containers/documentation/blob/master/Limitations.md#docker-run---privileged).

Can you review/manipulate the firewall tables manually in the Kata container? If not, it sounds like you might need further kernel config options.

@amshinde, @sboeuf, @bergwolf - any other ideas?

jodh-intel on 27 Jul 2018

Well I switch all my modules from being compiled into the kernel to being compiled to modules and that helped - the errors concerning failures when loading the modules are gone and it looks like the modules have actually been loaded correctly. But when docker tries to interact with iptables to create the docker NAT - this happens:

time="2018-07-27T13:42:50.987364618Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
time="2018-07-27T13:42:50.987564793Z" level=debug msg="Initialized graph driver overlay2"
time="2018-07-27T13:42:51.001629435Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2018-07-27T13:42:51.002095371Z" level=warning msg="Your kernel does not support swap memory limit"
time="2018-07-27T13:42:51.002196730Z" level=warning msg="Your kernel does not support cgroup rt period"
time="2018-07-27T13:42:51.002281170Z" level=warning msg="Your kernel does not support cgroup rt runtime"
time="2018-07-27T13:42:51.002363398Z" level=warning msg="Your kernel does not support cgroup blkio weight"
time="2018-07-27T13:42:51.002440118Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
time="2018-07-27T13:42:51.002516874Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_bps_device"
time="2018-07-27T13:42:51.002600616Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_bps_device"
time="2018-07-27T13:42:51.002675754Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_iops_device"
time="2018-07-27T13:42:51.002748171Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_iops_device"
time="2018-07-27T13:42:51.004155494Z" level=debug msg="Max Concurrent Downloads: 3"
time="2018-07-27T13:42:51.004241190Z" level=debug msg="Max Concurrent Uploads: 5"
time="2018-07-27T13:42:51.004552695Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2018-07-27T13:42:51.004628758Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2018-07-27T13:42:51.004730882Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0  <nil>}]" module=grpc
time="2018-07-27T13:42:51.004848142Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2018-07-27T13:42:51.004968696Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420181660, CONNECTING" module=grpc
time="2018-07-27T13:42:51.006128094Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420181660, READY" module=grpc
time="2018-07-27T13:42:51.006278687Z" level=info msg="Loading containers: start."
time="2018-07-27T13:42:51.006733601Z" level=debug msg="Option Experimental: false"
time="2018-07-27T13:42:51.006861099Z" level=debug msg="Option DefaultDriver: bridge"
time="2018-07-27T13:42:51.006934421Z" level=debug msg="Option DefaultNetwork: bridge"
time="2018-07-27T13:42:51.007003522Z" level=debug msg="Network Control Plane MTU: 1500"
time="2018-07-27T13:42:51.035651920Z" level=debug msg="Fail to initialize firewalld: Failed to connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory, using raw iptables instead"
time="2018-07-27T13:42:51.052397555Z" level=debug msg="/sbin/iptables, [-t filter -C FORWARD -j DOCKER-ISOLATION]"
time="2018-07-27T13:42:51.061158496Z" level=debug msg="/sbin/iptables, [-t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]"
time="2018-07-27T13:42:51.062510373Z" level=debug msg="processing event stream" module=libcontainerd namespace=moby
time="2018-07-27T13:42:51.071662203Z" level=debug msg="/sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]"
time="2018-07-27T13:42:51.081948632Z" level=debug msg="/sbin/iptables, [-t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]"
time="2018-07-27T13:42:51.091521717Z" level=debug msg="/sbin/iptables, [-t nat -D PREROUTING]"
time="2018-07-27T13:42:51.099841861Z" level=debug msg="/sbin/iptables, [-t nat -D OUTPUT]"
time="2018-07-27T13:42:51.107887889Z" level=debug msg="/sbin/iptables, [-t nat -F DOCKER]"
time="2018-07-27T13:42:51.114537168Z" level=debug msg="/sbin/iptables, [-t nat -X DOCKER]"
time="2018-07-27T13:42:51.120206454Z" level=debug msg="/sbin/iptables, [-t filter -F DOCKER]"
time="2018-07-27T13:42:51.126592345Z" level=debug msg="/sbin/iptables, [-t filter -X DOCKER]"
time="2018-07-27T13:42:51.132509204Z" level=debug msg="/sbin/iptables, [-t filter -F DOCKER-ISOLATION-STAGE-1]"
time="2018-07-27T13:42:51.138113287Z" level=debug msg="/sbin/iptables, [-t filter -X DOCKER-ISOLATION-STAGE-1]"
time="2018-07-27T13:42:51.144228992Z" level=debug msg="/sbin/iptables, [-t filter -F DOCKER-ISOLATION-STAGE-2]"
time="2018-07-27T13:42:51.150091477Z" level=debug msg="/sbin/iptables, [-t filter -X DOCKER-ISOLATION-STAGE-2]"
time="2018-07-27T13:42:51.155618543Z" level=debug msg="/sbin/iptables, [-t filter -F DOCKER-ISOLATION]"
time="2018-07-27T13:42:51.161882148Z" level=debug msg="/sbin/iptables, [-t filter -X DOCKER-ISOLATION]"
time="2018-07-27T13:42:51.169052640Z" level=debug msg="/sbin/iptables, [-t nat -n -L DOCKER]"
time="2018-07-27T13:42:51.176485116Z" level=debug msg="/sbin/iptables, [-t nat -N DOCKER]"
time="2018-07-27T13:42:51.183955595Z" level=debug msg="daemon configured with a 15 seconds minimum shutdown timeout"
time="2018-07-27T13:42:51.184108256Z" level=debug msg="start clean shutdown of all containers with a 15 seconds timeout..."
time="2018-07-27T13:42:51.185256314Z" level=debug msg="Cleaning up old mountid : start."
time="2018-07-27T13:42:51.185477500Z" level=debug msg="Cleaning up old mountid : done."
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.1: can't initialize iptables table `nat': Table
does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

The same behavior appears when interacting with iptables manually...

enzian on 27 Jul 2018

Hi @enzian - I've got a bunch of nat kernel modules on my system shown by lsmod so I'd guess you don't have them enabled at kernel build / available somehow?

jodh-intel on 27 Jul 2018

so, quick checkin: I added some additional Kernel modules and I'm now running with --cap-add=All and there is no errors from failed iptables setups anymore:

DEBU[2018-07-30T14:52:20.530391699Z] backingFs=ramfs,  projectQuotaSupported=false  storage-driver=overlay2
INFO[2018-07-30T14:52:20.531246587Z] [graphdriver] using prior storage driver: overlay2
DEBU[2018-07-30T14:52:20.532577602Z] Initialized graph driver overlay2
INFO[2018-07-30T14:52:20.545229480Z] Graph migration to content-addressability took 0.00 seconds
WARN[2018-07-30T14:52:20.547728701Z] Your kernel does not support swap memory limit
WARN[2018-07-30T14:52:20.548987033Z] Your kernel does not support cgroup rt period
WARN[2018-07-30T14:52:20.549793169Z] Your kernel does not support cgroup rt runtime
WARN[2018-07-30T14:52:20.550060141Z] Your kernel does not support cgroup blkio weight
WARN[2018-07-30T14:52:20.550415706Z] Your kernel does not support cgroup blkio weight_device
WARN[2018-07-30T14:52:20.550435111Z] Your kernel does not support cgroup blkio throttle.read_bps_device
WARN[2018-07-30T14:52:20.550446330Z] Your kernel does not support cgroup blkio throttle.write_bps_device
WARN[2018-07-30T14:52:20.551145673Z] Your kernel does not support cgroup blkio throttle.read_iops_device
WARN[2018-07-30T14:52:20.551425902Z] Your kernel does not support cgroup blkio throttle.write_iops_device
DEBU[2018-07-30T14:52:20.557940003Z] Max Concurrent Downloads: 3
DEBU[2018-07-30T14:52:20.557967149Z] Max Concurrent Uploads: 5
INFO[2018-07-30T14:52:20.559013376Z] parsed scheme: "unix"                         module=grpc
INFO[2018-07-30T14:52:20.559136339Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2018-07-30T14:52:20.559602017Z] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0  <nil>}]  module=grpc
INFO[2018-07-30T14:52:20.559624672Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2018-07-30T14:52:20.559738359Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420181390, CONNECTING  module=grpc
INFO[2018-07-30T14:52:20.562271990Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420181390, READY  module=grpc
INFO[2018-07-30T14:52:20.566101250Z] Loading containers: start.
DEBU[2018-07-30T14:52:20.567423343Z] Option Experimental: false
DEBU[2018-07-30T14:52:20.567440376Z] Option DefaultDriver: bridge
DEBU[2018-07-30T14:52:20.567446231Z] Option DefaultNetwork: bridge
DEBU[2018-07-30T14:52:20.567493384Z] Network Control Plane MTU: 1500
WARN[2018-07-30T14:52:20.588367377Z] Running modprobe nf_nat failed with message: `ip: can't find device 'nf_nat'
modprobe: can't load module nf_nat (net/netfilter/nf_nat.ko): invalid module format`, error: exit status 1
DEBU[2018-07-30T14:52:20.603021870Z] Fail to initialize firewalld: Failed to connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory, using raw iptables instead
DEBU[2018-07-30T14:52:20.620923004Z] /sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION]
DEBU[2018-07-30T14:52:20.630546045Z] /sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2018-07-30T14:52:20.634581752Z] processing event stream                       module=libcontainerd namespace=moby
DEBU[2018-07-30T14:52:20.645776744Z] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[2018-07-30T14:52:20.657295454Z] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2018-07-30T14:52:20.669528142Z] /sbin/iptables, [--wait -t nat -D PREROUTING]
DEBU[2018-07-30T14:52:20.681051547Z] /sbin/iptables, [--wait -t nat -D OUTPUT]
DEBU[2018-07-30T14:52:20.691087630Z] /sbin/iptables, [--wait -t nat -F DOCKER]
DEBU[2018-07-30T14:52:20.698390176Z] /sbin/iptables, [--wait -t nat -X DOCKER]
DEBU[2018-07-30T14:52:20.707937265Z] /sbin/iptables, [--wait -t filter -F DOCKER]
DEBU[2018-07-30T14:52:20.717181706Z] /sbin/iptables, [--wait -t filter -X DOCKER]
DEBU[2018-07-30T14:52:20.726568643Z] /sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION-STAGE-1]
DEBU[2018-07-30T14:52:20.735424185Z] /sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION-STAGE-1]
DEBU[2018-07-30T14:52:20.745125754Z] /sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION-STAGE-2]
DEBU[2018-07-30T14:52:20.753914601Z] /sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION-STAGE-2]
DEBU[2018-07-30T14:52:20.763943208Z] /sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION]
DEBU[2018-07-30T14:52:20.773346297Z] /sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION]
DEBU[2018-07-30T14:52:20.782430608Z] /sbin/iptables, [--wait -t nat -n -L DOCKER]
DEBU[2018-07-30T14:52:20.791284424Z] /sbin/iptables, [--wait -t nat -N DOCKER]
DEBU[2018-07-30T14:52:20.804641287Z] /sbin/iptables, [--wait -t filter -n -L DOCKER]
DEBU[2018-07-30T14:52:20.815512090Z] /sbin/iptables, [--wait -t filter -N DOCKER]
DEBU[2018-07-30T14:52:20.826533999Z] /sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION-STAGE-1]
DEBU[2018-07-30T14:52:20.836109880Z] /sbin/iptables, [--wait -t filter -N DOCKER-ISOLATION-STAGE-1]
DEBU[2018-07-30T14:52:20.845768048Z] /sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION-STAGE-2]
DEBU[2018-07-30T14:52:20.854715435Z] /sbin/iptables, [--wait -t filter -N DOCKER-ISOLATION-STAGE-2]
DEBU[2018-07-30T14:52:20.862936864Z] /sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-1 -j RETURN]
DEBU[2018-07-30T14:52:20.873216678Z] /sbin/iptables, [--wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN]
DEBU[2018-07-30T14:52:20.883676081Z] /sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-2 -j RETURN]
DEBU[2018-07-30T14:52:20.894006740Z] /sbin/iptables, [--wait -A DOCKER-ISOLATION-STAGE-2 -j RETURN]
WARN[2018-07-30T14:52:20.904388739Z] Enabling IP forwarding failed: open /proc/sys/net/ipv4/ip_forward: read-only file system
DEBU[2018-07-30T14:52:20.905172303Z] daemon configured with a 15 seconds minimum shutdown timeout
DEBU[2018-07-30T14:52:20.906342335Z] start clean shutdown of all containers with a 15 seconds timeout...
DEBU[2018-07-30T14:52:20.908555695Z] Cleaning up old mountid : start.
DEBU[2018-07-30T14:52:20.909443211Z] Cleaning up old mountid : done.
Error starting daemon: Error initializing network controller: error obtaining controller instance: Enabling IP forwarding failed: open /proc/sys/net/ipv4/ip_forward: read-only file system

This probably means I still don't have all the kernel modules I need or I have a permission problem!
I'll the kernel config into a PR once I`ve got this working

enzian on 30 Jul 2018

Hi @enzian - great progress. Regarding the kernel config, have you compared the config you are using with https://github.com/hyperhq/hyperstart/blob/master/build/arch/x86_64/kernel_config?

jodh-intel on 30 Jul 2018

Is there an easy way to slip in init scripts into the initrd?
echo 1 > /proc/sys/net/ipv4/ip_forward
Might do the trick.

kfox1111 on 30 Jul 2018

All you'd need to do:

1) Create a rootfs image using the osbuilder tool:

https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#create-a-rootfs-image

2) Modify that rootfs directory to create a script / systemd unit to enable that setting on boot.

3) Build and install your image:

https://github.com/kata-containers/documentation/blob/master/Developer-Guide.md#build-a-rootfs-image

jodh-intel on 30 Jul 2018

@jodh-intel I did base my config on the the latest tagged version 1.1.0, not on master since I notice quite some changes and I'd like to base my work on more stable grounds ;-)!
I made a gist with my current kernel config!

enzian on 31 Jul 2018

@jodh-intel is there any docs on how the kata project builds default images?

I made a new root file system based on the centos template - put net.ipv4.ip_forward = 1 in /etc/sysctl.conf built the image and:

INFO[2018-07-31T08:50:35.705289559Z] Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address
DEBU[2018-07-31T08:50:35.706669827Z] Allocating IPv4 pools for network bridge (6650c7d4ed18c43c9b5a812bea28bda207a006a530c904425bdfc35d5194291d)
DEBU[2018-07-31T08:50:35.708078542Z] RequestPool(LocalDefault, 172.18.0.0/16, , map[], false)
DEBU[2018-07-31T08:50:35.710763821Z] RequestAddress(LocalDefault/172.18.0.0/16, <nil>, map[RequestAddressType:com.docker.network.gateway])
DEBU[2018-07-31T08:50:35.711698379Z] Request address PoolID:172.18.0.0/16 App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:<nil>
DEBU[2018-07-31T08:50:35.713843582Z] Did not find any interface with name docker0: Link not found
DEBU[2018-07-31T08:50:35.722500600Z] Setting bridge mac address to 02:42:d1:1c:65:6c
DEBU[2018-07-31T08:50:35.723810910Z] Assigning address to bridge interface docker0: 172.18.0.1/16
DEBU[2018-07-31T08:50:35.724853713Z] /sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.18.0.0/16 ! -o docker0 -j MASQUERADE]
DEBU[2018-07-31T08:50:35.734985888Z] /sbin/iptables, [--wait -t nat -I POSTROUTING -s 172.18.0.0/16 ! -o docker0 -j MASQUERADE]
DEBU[2018-07-31T08:50:35.745888315Z] /sbin/iptables, [--wait -t nat -C DOCKER -i docker0 -j RETURN]
DEBU[2018-07-31T08:50:35.755898577Z] /sbin/iptables, [--wait -t nat -I DOCKER -i docker0 -j RETURN]
DEBU[2018-07-31T08:50:35.766734710Z] /sbin/iptables, [--wait -D FORWARD -i docker0 -o docker0 -j DROP]
DEBU[2018-07-31T08:50:35.777456642Z] /sbin/iptables, [--wait -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT]
DEBU[2018-07-31T08:50:35.787150417Z] /sbin/iptables, [--wait -I FORWARD -i docker0 -o docker0 -j ACCEPT]
DEBU[2018-07-31T08:50:35.797764321Z] /sbin/iptables, [--wait -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT]
DEBU[2018-07-31T08:50:35.809889512Z] /sbin/iptables, [--wait -I FORWARD -i docker0 ! -o docker0 -j ACCEPT]
DEBU[2018-07-31T08:50:35.821134780Z] /sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2018-07-31T08:50:35.834194386Z] /sbin/iptables, [--wait -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2018-07-31T08:50:35.849569298Z] releasing IPv4 pools from network bridge (6650c7d4ed18c43c9b5a812bea28bda207a006a530c904425bdfc35d5194291d)
DEBU[2018-07-31T08:50:35.849592058Z] ReleaseAddress(LocalDefault/172.18.0.0/16, 172.18.0.1)
DEBU[2018-07-31T08:50:35.849610236Z] Released address PoolID:LocalDefault/172.18.0.0/16, Address:172.18.0.1 Sequence:App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:2
DEBU[2018-07-31T08:50:35.849615130Z] ReleasePool(LocalDefault/172.18.0.0/16)
DEBU[2018-07-31T08:50:35.849625499Z] daemon configured with a 15 seconds minimum shutdown timeout
DEBU[2018-07-31T08:50:35.851021349Z] start clean shutdown of all containers with a 15 seconds timeout...
DEBU[2018-07-31T08:50:35.854409036Z] Cleaning up old mountid : start.
DEBU[2018-07-31T08:50:35.855576479Z] Cleaning up old mountid : done.
Error starting daemon: Error initializing network controller: Error creating default "bridge" network: Failed to program NAT chain: Failed to inject DOCKER in PREROUTING chain: iptables failed: iptables --wait -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER: iptables: No chain/target/match by that name.
 (exit status 1)

enzian on 31 Jul 2018

Hi @enzian - that kernel config link I pasted above is for hyper, not Kata. And hyper has support for running under docker apparently...

Regarding info on building images, again, I pasted links above :) Further info:

jodh-intel on 31 Jul 2018

Huge step forward today - changed some stuff the kernel .config (see in the gist) and dockerd is actually starting! Running a container fails though:

DEBU[2018-07-31T13:15:33.767216460Z] Calling POST /v1.38/containers/create
DEBU[2018-07-31T13:15:33.768017956Z] form data: {"AttachStderr":true,"AttachStdin":false,"AttachStdout":true,"Cmd":null,"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"AutoRemove":false,"Binds":null,"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":[],"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"Devices":[],"DiskQuota":0,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"MaskedPaths":null,"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyPaths":null,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"nginx","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"Tty":false,"User":"","Volumes":{},"WorkingDir":""}
DEBU[2018-07-31T13:15:33.770732742Z] Calling GET /v1.38/info
DEBU[2018-07-31T13:15:33.778002903Z] Calling POST /v1.38/images/create?fromImage=nginx&tag=latest
DEBU[2018-07-31T13:15:33.778611399Z] Trying to pull nginx from https://registry-1.docker.io v2
DEBU[2018-07-31T13:15:35.325095421Z] Pulling ref from V2 registry: nginx:latest
DEBU[2018-07-31T13:15:35.325448219Z] docker.io/library/nginx:latest resolved to a manifestList object with 6 entries; looking for a unknown/amd64 match
DEBU[2018-07-31T13:15:35.325890217Z] found match for linux/amd64 with media type application/vnd.docker.distribution.manifest.v2+json, digest sha256:4ffd9758ea9ea360fd87d0cee7a2d1cf9dba630bb57ca36b3108dcd3708dc189
DEBU[2018-07-31T13:15:40.800946313Z] pulling blob "sha256:be8881be8156e4068e611fe956aba2b9593ebd953be14fb7feea6d0659aa3abe"
DEBU[2018-07-31T13:15:40.801412011Z] pulling blob "sha256:32d9726baeef1d8d626fa618fdc76200248b83e6453d642c25bddee92326d5ab"
DEBU[2018-07-31T13:15:40.801419211Z] pulling blob "sha256:87e5e6f71297704d2e6724248ededfb71e994dcf2fd9568b309b2d294e6ee455"
DEBU[2018-07-31T13:15:41.888341194Z] Downloaded be8881be8156 to tempfile /var/lib/docker/tmp/GetImageBlob812237261
DEBU[2018-07-31T13:15:42.123925733Z] Downloaded 32d9726baeef to tempfile /var/lib/docker/tmp/GetImageBlob794999496
DEBU[2018-07-31T13:15:42.383523844Z] Downloaded 87e5e6f71297 to tempfile /var/lib/docker/tmp/GetImageBlob946216839
DEBU[2018-07-31T13:15:42.711773788Z] Cleaning up layer d769a8bc1c7929095290a6c45f590faf2add42a1447dc335f64658bf62a7da22: Error processing tar file(exit status 1): permission denied
INFO[2018-07-31T13:15:42.716548862Z] Attempting next endpoint for pull after error: failed to register layer: Error processing tar file(exit status 1): permission denied

now could this be due to the dockerd not having insufficent permissions ... but where?

enzian on 31 Jul 2018

Hi @enzian - great! Clearly, we need to understand the EPERM issue, but does a simple(r) "docker run -ti busybox sh" work?

Regarding the perm issue, I wrote a tool that is designed to help with this sort of question. Simply apt-get/dnf install it and you can then place it "before" dockerd:

/usr/bin/procenv --file=/tmp/procenv.log --exec -- /usr/bin/dockerd ...

If you do that on a working system and again in the Kata Container, you can then simply diff the /tmp/procenv.log files and you should get a pointer to the issue. You might want/need to play around with the --output / --format options to make life easier for yourself.

jodh-intel on 31 Jul 2018

ok this is embarrassing - My success was based on me switching the default container runtime to runc so I could simply use docker to build my custom OS images :p
BUT I still have some progress: all modules are loaded successfully so far.

I'm currently failing to set iptable rules:

DEBU[2018-08-02T07:12:36.376260393Z] Max Concurrent Downloads: 3
DEBU[2018-08-02T07:12:36.376378442Z] Max Concurrent Uploads: 5
INFO[2018-08-02T07:12:36.377474600Z] parsed scheme: "unix"                         module=grpc
INFO[2018-08-02T07:12:36.377589216Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2018-08-02T07:12:36.377961298Z] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/docker-containerd.sock 0  <nil>}]  module=grpc
INFO[2018-08-02T07:12:36.377979443Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2018-08-02T07:12:36.381937429Z] pickfirstBalancer: HandleSubConnStateChange: 0xc42017f3e0, CONNECTING  module=grpc
INFO[2018-08-02T07:12:36.383964012Z] pickfirstBalancer: HandleSubConnStateChange: 0xc42017f3e0, READY  module=grpc
INFO[2018-08-02T07:12:36.384542863Z] Loading containers: start.
DEBU[2018-08-02T07:12:36.385750952Z] Option Experimental: false
DEBU[2018-08-02T07:12:36.386491709Z] Option DefaultDriver: bridge
DEBU[2018-08-02T07:12:36.386563132Z] Option DefaultNetwork: bridge
DEBU[2018-08-02T07:12:36.386572733Z] Network Control Plane MTU: 1500
DEBU[2018-08-02T07:12:36.403845778Z] processing event stream                       module=libcontainerd namespace=moby
DEBU[2018-08-02T07:12:36.421967540Z] Fail to initialize firewalld: Failed to connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory, using raw iptables instead
DEBU[2018-08-02T07:12:36.443395542Z] /sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION]
DEBU[2018-08-02T07:12:36.456091606Z] /sbin/iptables, [--wait -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2018-08-02T07:12:36.471062339Z] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL ! --dst 127.0.0.0/8 -j DOCKER]
DEBU[2018-08-02T07:12:36.484904671Z] /sbin/iptables, [--wait -t nat -D OUTPUT -m addrtype --dst-type LOCAL -j DOCKER]
DEBU[2018-08-02T07:12:36.498821064Z] /sbin/iptables, [--wait -t nat -D PREROUTING]
DEBU[2018-08-02T07:12:36.511529639Z] /sbin/iptables, [--wait -t nat -D OUTPUT]
DEBU[2018-08-02T07:12:36.523790695Z] /sbin/iptables, [--wait -t nat -F DOCKER]
DEBU[2018-08-02T07:12:36.534015526Z] /sbin/iptables, [--wait -t nat -X DOCKER]
DEBU[2018-08-02T07:12:36.545258483Z] /sbin/iptables, [--wait -t filter -F DOCKER]
DEBU[2018-08-02T07:12:36.555932728Z] /sbin/iptables, [--wait -t filter -X DOCKER]
DEBU[2018-08-02T07:12:36.565882452Z] /sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION-STAGE-1]
DEBU[2018-08-02T07:12:36.576385761Z] /sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION-STAGE-1]
DEBU[2018-08-02T07:12:36.586749906Z] /sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION-STAGE-2]
DEBU[2018-08-02T07:12:36.598908999Z] /sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION-STAGE-2]
DEBU[2018-08-02T07:12:36.609175476Z] /sbin/iptables, [--wait -t filter -F DOCKER-ISOLATION]
DEBU[2018-08-02T07:12:36.619063920Z] /sbin/iptables, [--wait -t filter -X DOCKER-ISOLATION]
DEBU[2018-08-02T07:12:36.630705866Z] /sbin/iptables, [--wait -t nat -n -L DOCKER]
DEBU[2018-08-02T07:12:36.641409713Z] /sbin/iptables, [--wait -t nat -N DOCKER]
DEBU[2018-08-02T07:12:36.652235006Z] /sbin/iptables, [--wait -t filter -n -L DOCKER]
DEBU[2018-08-02T07:12:36.662115302Z] /sbin/iptables, [--wait -t filter -N DOCKER]
DEBU[2018-08-02T07:12:36.673351212Z] /sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION-STAGE-1]
DEBU[2018-08-02T07:12:36.683074097Z] /sbin/iptables, [--wait -t filter -N DOCKER-ISOLATION-STAGE-1]
DEBU[2018-08-02T07:12:36.694355142Z] /sbin/iptables, [--wait -t filter -n -L DOCKER-ISOLATION-STAGE-2]
DEBU[2018-08-02T07:12:36.705798300Z] /sbin/iptables, [--wait -t filter -N DOCKER-ISOLATION-STAGE-2]
DEBU[2018-08-02T07:12:36.715729049Z] /sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-1 -j RETURN]
DEBU[2018-08-02T07:12:36.728453435Z] /sbin/iptables, [--wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN]
DEBU[2018-08-02T07:12:36.741588156Z] /sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION-STAGE-2 -j RETURN]
DEBU[2018-08-02T07:12:36.753855332Z] /sbin/iptables, [--wait -A DOCKER-ISOLATION-STAGE-2 -j RETURN]
INFO[2018-08-02T07:12:36.837951339Z] Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address
DEBU[2018-08-02T07:12:36.840720338Z] Allocating IPv4 pools for network bridge (a81da86b6a65c2da6a6f6d81369f1acc2c47440b1e61a36098f99b6b8abf3599)
DEBU[2018-08-02T07:12:36.841788682Z] RequestPool(LocalDefault, 172.18.0.0/16, , map[], false)
DEBU[2018-08-02T07:12:36.843539148Z] RequestAddress(LocalDefault/172.18.0.0/16, 172.18.0.1, map[RequestAddressType:com.docker.network.gateway])
DEBU[2018-08-02T07:12:36.844465065Z] Request address PoolID:172.18.0.0/16 App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65534, Sequence: (0x80000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0 Serial:false PrefAddress:172.18.0.1
DEBU[2018-08-02T07:12:36.846810080Z] /sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.18.0.0/16 ! -o docker0 -j MASQUERADE]
DEBU[2018-08-02T07:12:36.856475378Z] /sbin/iptables, [--wait -t nat -I POSTROUTING -s 172.18.0.0/16 ! -o docker0 -j MASQUERADE]
DEBU[2018-08-02T07:12:36.866538508Z] releasing IPv4 pools from network bridge (a81da86b6a65c2da6a6f6d81369f1acc2c47440b1e61a36098f99b6b8abf3599)
DEBU[2018-08-02T07:12:36.866560339Z] ReleaseAddress(LocalDefault/172.18.0.0/16, 172.18.0.1)
DEBU[2018-08-02T07:12:36.866579862Z] Released address PoolID:LocalDefault/172.18.0.0/16, Address:172.18.0.1 Sequence:App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:0
DEBU[2018-08-02T07:12:36.866590839Z] ReleasePool(LocalDefault/172.18.0.0/16)
DEBU[2018-08-02T07:12:36.866603404Z] daemon configured with a 15 seconds minimum shutdown timeout
DEBU[2018-08-02T07:12:36.867551317Z] start clean shutdown of all containers with a 15 seconds timeout...
DEBU[2018-08-02T07:12:36.870347503Z] Cleaning up old mountid : start.
DEBU[2018-08-02T07:12:36.871432718Z] Cleaning up old mountid : done.
Error starting daemon: Error initializing network controller: Error creating default "bridge" network: Failed to Setup IP tables: Unable to enable NAT rule:  (iptables failed: iptables --wait -t nat -I POSTROUTING -s 172.18.0.0/16 ! -o docker0 -j MASQUERADE: iptables: No chain/target/match by that name.

Any tips are dearly welcome - as some of you may have noticed I've rarely taken a dive that deep into the underbelly of an OS :p

enzian on 2 Aug 2018

That error is a bit odd as the iptables command it claims has failed seems to have succeeded a few lines above? Maybe docker is doing some sort of async network setup though?

Assuming you've enabled all the kernel config, I think we need a docker/network expert to look at this.

/cc @jessfraz, @mcastelino.

jodh-intel on 2 Aug 2018

well it is correct - iptables is missing those chains:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
DOCKER-USER  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (0 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (0 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (0 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

... my question is why ;-)

enzian on 2 Aug 2018

What does the following show inside the container?

# iptables -t nat -L -n

jodh-intel on 2 Aug 2018

that:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
DOCKER-USER  all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (0 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION-STAGE-1 (0 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (0 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
/ # iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.18.0.0/16        0.0.0.0/0

Chain DOCKER (0 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            0.0.0.0/0

... seems like what I'm looking for. And here is the updated kernel config... Then I use the docker/docker:18.04.0-dind image and run

depmod -ae
modprobe iptable_nat
modprobe iptable_filter
modprobe ipt_MASQUERADE
umount /var/lib/docker/
mount -t ramfs -o size=10M ramfs /var/lib/docker/
dockerd --oom-score-adjust 500 --log-level debug

enzian on 2 Aug 2018

It looks like there were still modules missing... I added all nf_tables functionality (not as modules) and now dockerd comes um and i can launch a container: docker run -ti busybox sh. The image is pulled but starting it fails due to some kind of cgroup issue:

DEBU[2018-08-02T12:34:14.425220486Z] Calling GET /v1.38/info
DEBU[2018-08-02T12:34:14.492906272Z] Calling POST /v1.38/images/create?fromImage=busybox&tag=latest
DEBU[2018-08-02T12:34:14.498493035Z] Trying to pull busybox from https://registry-1.docker.io v2
DEBU[2018-08-02T12:34:21.577046142Z] Pulling ref from V2 registry: busybox:latest
DEBU[2018-08-02T12:34:21.578408928Z] docker.io/library/busybox:latest resolved to a manifestList object with 8 entries; looking for a unknown/amd64 match
DEBU[2018-08-02T12:34:21.580416763Z] found match for linux/amd64 with media type application/vnd.docker.distribution.manifest.v2+json, digest sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd
DEBU[2018-08-02T12:34:22.108951972Z] pulling blob "sha256:8c5a7da1afbc602695fcb2cd6445743cec5ff32053ea589ea9bd8773b7068185"
DEBU[2018-08-02T12:34:22.734425502Z] Downloaded 8c5a7da1afbc to tempfile /var/lib/docker/tmp/GetImageBlob271722718
DEBU[2018-08-02T12:34:22.744642475Z] Applying tar in /var/lib/docker/overlay2/336eb482ceea3ce34b680a3bf93870ac8c05dfbac40a1f26302505d0b43851a7/diff  storage-driver=overlay2
DEBU[2018-08-02T12:34:22.828439063Z] Applied tar sha256:f9d9e4e6e2f0689cd752390e14ade48b0ec6f2a488a05af5ab2f9ccaf54c299d to 336eb482ceea3ce34b680a3bf93870ac8c05dfbac40a1f26302505d0b43851a7, size: 1162769
DEBU[2018-08-02T12:34:22.836538392Z] Calling POST /v1.38/containers/create
DEBU[2018-08-02T12:34:22.837370245Z] form data: {"AttachStderr":true,"AttachStdin":true,"AttachStdout":true,"Cmd":["sh"],"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"AutoRemove":false,"Binds":null,"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":[],"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"Devices":[],"DiskQuota":0,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"MaskedPaths":null,"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyPaths":null,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"busybox","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":true,"StdinOnce":true,"Tty":true,"User":"","Volumes":{},"WorkingDir":""}
DEBU[2018-08-02T12:34:22.847593840Z] container mounted via layerStore: &{/var/lib/docker/overlay2/1311d55ac501ff916058abf6d42005b93e086ef8a3595a2843c4c5680320c5c4/merged 0x31d3280 0x31d3280}
DEBU[2018-08-02T12:34:22.881061541Z] Calling POST /v1.38/containers/6c3184c9c44d500806978573e925285b92c3ddac17b6d27668666cf7f5970270/attach?stderr=1&stdin=1&stdout=1&stream=1
DEBU[2018-08-02T12:34:22.885817914Z] attach: stdin: begin
DEBU[2018-08-02T12:34:22.887363489Z] attach: stdout: begin
DEBU[2018-08-02T12:34:22.887876149Z] attach: stderr: begin
DEBU[2018-08-02T12:34:22.888736053Z] Calling POST /v1.38/containers/6c3184c9c44d500806978573e925285b92c3ddac17b6d27668666cf7f5970270/wait?condition=next-exit
DEBU[2018-08-02T12:34:22.893448089Z] Calling POST /v1.38/containers/6c3184c9c44d500806978573e925285b92c3ddac17b6d27668666cf7f5970270/start
DEBU[2018-08-02T12:34:22.896785338Z] container mounted via layerStore: &{/var/lib/docker/overlay2/1311d55ac501ff916058abf6d42005b93e086ef8a3595a2843c4c5680320c5c4/merged 0x31d3280 0x31d3280}
DEBU[2018-08-02T12:34:22.900390382Z] Assigning addresses for endpoint gracious_hermann's interface on network bridge
DEBU[2018-08-02T12:34:22.900894012Z] RequestAddress(LocalDefault/172.18.0.0/16, <nil>, map[])
DEBU[2018-08-02T12:34:22.900922100Z] Request address PoolID:172.18.0.0/16 App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:2 Serial:false PrefAddress:<nil>
DEBU[2018-08-02T12:34:22.910694854Z] Assigning addresses for endpoint gracious_hermann's interface on network bridge
DEBU[2018-08-02T12:34:22.915383129Z] Programming external connectivity on endpoint gracious_hermann (93848102883eb35b511c9185f49fa501f1c9b70890a879d5c54986a4fcc3ddc9)
DEBU[2018-08-02T12:34:22.917035936Z] EnableService 6c3184c9c44d500806978573e925285b92c3ddac17b6d27668666cf7f5970270 START
DEBU[2018-08-02T12:34:22.917452052Z] EnableService 6c3184c9c44d500806978573e925285b92c3ddac17b6d27668666cf7f5970270 DONE
DEBU[2018-08-02T12:34:22.925253032Z] bundle dir created                            bundle=/var/run/docker/containerd/6c3184c9c44d500806978573e925285b92c3ddac17b6d27668666cf7f5970270 module=libcontainerd namespace=moby root=/var/lib/docker/overlay2/1311d55ac501ff916058abf6d42005b93e086ef8a3595a2843c4c5680320c5c4/merged
INFO[0056] shim docker-containerd-shim started           address="/containerd-shim/moby/6c3184c9c44d500806978573e925285b92c3ddac17b6d27668666cf7f5970270/shim.sock" debug=false pid=150
INFO[0057] shim reaped                                   id=6c3184c9c44d500806978573e925285b92c3ddac17b6d27668666cf7f5970270
ERRO[2018-08-02T12:34:23.117122671Z] stream copy error: reading from a closed fifo
DEBU[2018-08-02T12:34:23.120757471Z] attach: stdout: end
DEBU[2018-08-02T12:34:23.120931992Z] attach: stdin: end
DEBU[2018-08-02T12:34:23.120951432Z] attach: stderr: end
DEBU[2018-08-02T12:34:23.121434346Z] attach done
DEBU[2018-08-02T12:34:23.121737430Z] Closing buffered stdin pipe
DEBU[2018-08-02T12:34:23.132886949Z] Revoking external connectivity on endpoint gracious_hermann (93848102883eb35b511c9185f49fa501f1c9b70890a879d5c54986a4fcc3ddc9)
WARN[2018-08-02T12:34:23.135326011Z] Failed to delete conntrack state for 172.18.0.2: invalid argument
DEBU[2018-08-02T12:34:23.135343222Z] DeleteConntrackEntries purged ipv4:0, ipv6:0
DEBU[2018-08-02T12:34:23.141425341Z] Releasing addresses for endpoint gracious_hermann's interface on network bridge
DEBU[2018-08-02T12:34:23.141454529Z] ReleaseAddress(LocalDefault/172.18.0.0/16, 172.18.0.2)
DEBU[2018-08-02T12:34:23.141530378Z] Released address PoolID:LocalDefault/172.18.0.0/16, Address:172.18.0.2 Sequence:App: ipam/default/data, ID: LocalDefault/172.18.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65532, Sequence: (0xe0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3
ERRO[2018-08-02T12:34:23.143851876Z] 6c3184c9c44d500806978573e925285b92c3ddac17b6d27668666cf7f5970270 cleanup: failed to delete container from containerd: no such container
ERRO[2018-08-02T12:34:23.144443578Z] Handler for POST /v1.38/containers/6c3184c9c44d500806978573e925285b92c3ddac17b6d27668666cf7f5970270/start returned error: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"mkdir /sys/fs/cgroup/cpuset/docker: read-only file system\"": unknown

(I think I see light at the end of the tunnel ;-) )

enzian on 2 Aug 2018

I think you'd need to specify --privileged or --cap-add SYS_ADMIN -v /sys/fs/cgroup:/sys/fs/cgroup:fw to allow the cgroups to be modified.

jodh-intel on 2 Aug 2018

Nope, that does not help :p

enzian on 2 Aug 2018

We're trying to run docker in a kata container here, yes? In which case we can't use --privileged on the kata container (unsupported) - so, going to have to try the cap-add (and I see that that may not have helped....)

grahamwhaley on 2 Aug 2018

@grahamwhaley nope we're already at the stage where we use runc to run a container within a kata container!

enzian on 2 Aug 2018

hmm on my host /sys/fs/cgroup/* are mounted as rw but in my kata-container they are ro

cgroup on /sys/fs/cgroup/systemd type cgroup (ro,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (ro,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (ro,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (ro,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (ro,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (ro,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (ro,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (ro,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory)

enzian on 2 Aug 2018

@enzian - the mounts and mount options (ro) are coming from docker, so you have to run --privileged or use the hack above to have these appear as rw fwics.

jodh-intel on 2 Aug 2018

@jodh-intel so this means I have to mount the cgroups from my host into the kata-container which would then be able to modify cgroups on my host? This seems counter intuitive when someone decides to run virtual machines under it's containers... Or am I getting something wrong here?

@jodh-intel Here's what I did:

HOST $>sudo docker run --runtime kata-runtime --rm -it --cap-add=All -v ${PWD}:/lib/modules/4.14.22 -v /sys/fs/cgroup:/sys/fs/cgroup:rw docker:18.06.0-dind sh

CONTAINER $> umount /var/lib/docker/
CONTAINER $> mount -t ramfs -o size=10M ramfs /var/lib/docker/
CONTAINER $> dockerd --oom-score-adjust 500 --log-level debug

At this point the cgroup information is mounted from the host and I can start my container inception:

CONTAINER $> docker run -ti busybox sh

which works perfectly - but how could I now enforce resource constraints on the VM while still giving docker within the VM the freedom to manage it's own cgroups?

enzian on 2 Aug 2018

@enzian - I'm certainly not recommending it, I was just trying to point out what docker does.

The Hyper guys might be able to offer some insight if this is supported in that environment.

/cc @bergwolf, @gnawux.

jodh-intel on 2 Aug 2018

Hmm, we'd need the ability to mount not only from the host machine but also from the VM, which is not something accounted for in the OCI specs, I'd assume!

enzian on 3 Aug 2018

There was some talk on the storage list about supporting emptyDir in CRI so that kata could implement it vm side rather then over 9p.

kfox1111 on 6 Aug 2018

wow, just reading through this issue now - @enzian you've been on quite a ride - thank you.

A couple of things:
1) You can run a container as privileged - give it a try with latest release of kata (1.2.0). Note, though, you won't be accessing host ip tables, sysfs, etc, but will have access for the virtual machines' structures.
2) I don't think you need to play with cgroups on the host.
3) This is a bit confusing, since there are a few layers here. Inside the VM we constrain the container itself (the original container image in which you want to run docker), which you point out above. I haven't dove in much yet, but as far as I understand you should be able to nest cgroups, and they're handled in a hierarchical manner. @mcastelino, you know?

Me marking this as a good first issue was pretty sadistic, in hindsight.

egernst on 16 Aug 2018

hey @egernst: thanks for removing that flag ;-) I'll give 1.20 a shot and check back in here!

you've been on quite a ride

Yes it was a wild ride - I'd be lying if I said it wasn't fun though ;-)

enzian on 17 Aug 2018

well it has been quite a while but, I updated to the newest kata version and attempted to run a privileged container:

$ sudo docker run --rm -it --privileged docker:dind sh
docker: Error response from daemon: OCI runtime create failed: QMP command failed: unknown.

enzian on 28 Sep 2018

I suspect this might be a QEMU version thing. What version did you upgrade from/to - and did you upgrade the QEMU at the same time. And how did you upgrade?
@jodh-intel @jcvenegas - did we have a niggle whereby the QEMU was not auto-updating via packaging recently?

grahamwhaley on 28 Sep 2018

Hi @enzian - welcome back! :smile:

@grahamwhaley - yep - that was https://github.com/kata-containers/packaging/issues/193.

If you can re-run kata-collect-data.sh we can see if you were affected by that @enzian.

jodh-intel on 28 Sep 2018

hey, here's the kafa-collect dump

@grahamwhaley: I upgraded from 1.1 to 1.3 directly used apt to do so.

enzian on 4 Oct 2018

Hi @enzian . Thanks for the info. I checked the qemu version from your file: qemu-lite 2.11.0+git.f886228056-51, and that is the same commit as I have installed.
So I ran your above priv dind docker command, and I get a prompt!

My install is slightly behind yours - I'm on a 1.3.0-rc1 pretty much (this is my dev box though, so some parts may be a little 'in-flight').
Calling on others to see if they have seen this QMP issue before then - /cc @sboeuf @amshinde @devimc @egernst

grahamwhaley on 4 Oct 2018

I reinstalled kata on a clean ubuntu 1803 machine on hyper-v using kata-manager.sh. It still gives me the same error:

docker: Error response from daemon: OCI runtime create failed: QMP command failed: unknown.
ERRO[0002] error waiting for container: context canceled

enzian on 2 Nov 2018

@enzian You need a machine with KVM capability. With hyper-v, you need to enable nested VT on the host OS:
Could not access KVM kernel module: No such file or directory

bergwolf on 2 Nov 2018

@enzian You can run kata-runtime kata-check to check for proper capability in your hyper-v guest.

bergwolf on 2 Nov 2018

ok:

INFO[0000] CPU property found                            arch=amd64 description="Intel Architecture CPU" name=GenuineIntel pid=3431 source=runtime type=attribute
INFO[0000] CPU property found                            arch=amd64 description="Virtualization support" name=vmx pid=3431 source=runtime type=flag
INFO[0000] CPU property found                            arch=amd64 description="64Bit CPU" name=lm pid=3431 source=runtime type=flag
INFO[0000] CPU property found                            arch=amd64 description=SSE4.1 name=sse4_1 pid=3431 source=runtime type=flag
INFO[0000] kernel property found                         arch=amd64 description="Kernel-based Virtual Machine" name=kvm pid=3431 source=runtime type=module
INFO[0000] kernel property found                         arch=amd64 description="Host kernel accelerator for virtio" name=vhost pid=3431 source=runtime type=module
INFO[0000] kernel property found                         arch=amd64 description="Host kernel accelerator for virtio network" name=vhost_net pid=3431 source=runtime type=module
INFO[0000] kernel property found                         arch=amd64 description="Intel KVM" name=kvm_intel pid=3431 source=runtime type=module
WARN[0000] kernel module parameter has unexpected value  arch=amd64 description="Intel KVM" expected=Y name=kvm_intel parameter=nested pid=3431 source=runtime type=module value=N
INFO[0000] Kernel property value correct                 arch=amd64 description="Intel KVM" expected=Y name=kvm_intel parameter=unrestricted_guest pid=3431 source=runtime type=module value=Y
INFO[0000] System is capable of running Kata Containers  arch=amd64 name=kata-runtime pid=3431 source=runtime
INFO[0000] device available                              arch=amd64 check-type=full device=/dev/kvm name=kata-runtime pid=3431 source=runtime
INFO[0000] feature available                             arch=amd64 check-type=full feature=create-vm name=kata-runtime pid=3431 source=runtime
INFO[0000] System can currently create Kata Containers   arch=amd64 name=kata-runtime pid=3431 source=runtime

enzian on 2 Nov 2018

might it be my processor being too old?

WARN[0000] kernel module parameter has unexpected value  arch=amd64 description="Intel KVM" expected=Y name=kvm_intel parameter=nested pid=1599 source=runtime type=module value=N

enzian on 16 Nov 2018

might it be my processor being too old?

WARN[0000] kernel module parameter has unexpected value  arch=amd64 description="Intel KVM" expected=Y name=kvm_intel parameter=nested pid=1599 source=runtime type=module value=N

Nested mode is only needed if you want to get yet another level of nesting, and it is not required on the system where you have katacontainers installed (i.e., on the system where you run docker run --runtime=kata-runtime ...).
Given your config, you should be able to run containers using kata.

marcov on 16 Nov 2018

@enzian - please could you:

Enable full debug,
Re-run the failing docker command.
Run sudo kata-collect-data.sh.
Review the output of the above script and if it doesn't contain anything sensitive, paste the output into this issue as a comment. Part of the report will show errors which might give us more of a clue.

jodh-intel on 16 Nov 2018

@enzian - it would also obviously be useful if you could try running on a baremetal system without hyper-v for comparative purposes ;)

jodh-intel on 16 Nov 2018

ok, well we found our problem. The ESX-Cluster we had did not support unrestricted guests - migrated the hosts and it works!

enzian on 20 Nov 2018

Cheers!

caoruidong on 20 Nov 2018

I think we can close this issue now, right?

enzian on 22 Nov 2018

@enzian - nice! :smile:

@svarlamov - have you managed to recreate @enzian's results?

It might be useful to summarise what is required and document this somewhere (since this is a very long issue :)

jodh-intel on 22 Nov 2018

@jodh-intel - I will try to run a kublet in combination with a DinD container. This I will document and contribute the how-to to the kata-project. Is this something that would be welcome?

enzian on 23 Nov 2018

❤3 🎉1

@enzian looking forward to your docs

scher200 on 22 Feb 2019

Hello, and good work with all this. I'm curious to see the documentation. I have a docker:dind image running with the kata-runtime, but it hangs when building dockerfiles. When starting the dockerd, I also get errors about the overlay storage-driver:
ERRO[2019-02-26T23:42:57.805555497Z] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded. storage-driver=overlay
I'm guessing I need to build the kernel image with non-default flags. Is there any progress on the documentation around this issue?

Thank you!!

bajacondor on 27 Feb 2019

@bajacondor you mean documentation on how to build your kernel?

jcvenegas on 27 Feb 2019

Hi @jcvenegas, I was hoping for documentation around running docker in docker or building with docker within a kata container.

bajacondor on 27 Feb 2019

👍3

Just as an FYI here, I managed to make docker in kata work. I have written up some basic documentation on how this was done here. I also provide a containerized build for the whole thing (which I tried to make portable, even if you don't have kata installed).

I would appreciate any feedback on my approach (curious to know if it works / is useful for anyone else). I will improve the quick docs and add a few other features tomorrow if I get time after work.

Thanks to everyone for their work on this issue :)

daniel-noland on 26 Nov 2019

👍1

Wow @daniel-noland - triple nested!!! (kata-in-docker and docker-in-kata in the same stackup!).
I had a quick skim through your link, and it looked OK to me. Yes, it'd be great if we could get some official docs PR'd into the kata repos - ideally for both kata-in-docker and docker-in-kata setups.

grahamwhaley on 26 Nov 2019

🎉1

Thanks @grahamwhaley, I appreciate that.

get this working on the 4.19 kernel (should be easy),
find and document the minimum set of kernel features strictly necessary to run nested docker (I didn't want to rebuild a million times so I just turned on basically all of the nf_tables / nat / netfilter features),
determine if we actually want to include btrfs support (I found docker build to be completely broken with the 9p / overlay approach but that doesn't mean we must use btrfs),
write some real docs (my stream of thought notes are likely not quite good enough :nerd_face:),
write some tests.

I am happy to do any and all of that, I would just want to confirm that there is interest first :smile_cat:.

Again, thanks to everyone for all their hard work on kata-containers. I have really enjoyed playing with this project :smiley:

daniel-noland on 26 Nov 2019

👍4

I was trying to run Docker in Kata. I tried two approaches in order to prevent getting the error failed to start daemon: error initializing graphdriver: driver not supported.

Using the vfs storage driver.
Using the overlay2 storage driver and mounting a tmpfs on /var/lib/docker/overlay2.

With both approaches I don't get above error and the docker daemon starts successfully.

/ # docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
Digest: sha256:8e10956422503824ebb599f37c26a90fe70541942687f70bbdb744530fc9eba4
Status: Image is up to date for nginx:latest
docker.io/library/nginx:latest

However, I can't run containers. Equally with both approaches, I get the following error:

/ # docker run nginx
docker: Error response from daemon: cgroups: cannot find cgroup mount destination: unknown.

Does anybody have an idea how to fix this?

Additional information

Image: docker.io/library/docker:19-dind
Image ID: docker.io/library/docker@sha256:3f102649d944f417085acce09362927f8a6371c919c044ff1176d5c4c007f351
Docker version inside Kata container:
/ # docker --version Docker version 19.03.15, build 99e3ed8

haslersn on 15 Feb 2021

After reading this, I found out that Docker in Kata works for me after executing the following commands in the container:

mkdir /sys/fs/cgroup/systemd
mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

Is there a way to start a Kata Container on Kubernetes with that mount already present?

haslersn on 15 Feb 2021

Was this page helpful?

0 / 5 - 0 ratings

Related issues

[RFC] Implement a runtime interface v2 for both containerd and cri-o

sboeuf · 13Comments

The number of percpu_usages is incorrect

ZeroMagic · 5Comments

OpenStack Zun via DevStack doesn't work with Kata-containers (Could not setup network routes)

eadamsintel · 11Comments

Use "virtiofs" instead of "virtio_fs" for the virtio-fs module

fidencio · 9Comments

virtcontainers: Make sandbox manage VM CPUs.

jcvenegas · 7Comments