Runtime: Docker-in-Docker broken (Kata 1.8.0 -> 1.8.1 regression)

[AkihiroSuda]

Description of problem

Docker-in-Docker broken in Kata 1.8.1.

Kata 1.8.0 doesn't have this issue.

Expected result

Docker-in-Docker should work

Actual result

$ docker run -it --rm --name dind --runtime=kata --privileged docker:19.03.2-dind --storage-driver overlay2
...
ERRO[2019-09-12T06:11:01.381080608Z] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.  storage-driver=overlay2
INFO[2019-09-12T06:11:01.384753732Z] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
INFO[2019-09-12T06:11:01.385497114Z] stopping healthcheck following graceful shutdown  module=libcontainerd
INFO[2019-09-12T06:11:01.397572996Z] pickfirstBalancer: HandleSubConnStateChange: 0xc00066b460, TRANSIENT_FAILURE  module=grpc
INFO[2019-09-12T06:11:01.397947975Z] pickfirstBalancer: HandleSubConnStateChange: 0xc00066b460, CONNECTING  module=grpc
failed to start daemon: error initializing graphdriver: driver not supported

The daemon can start up with --storage-driver=vfs, but containers cannot be executed

$ docker run -it --rm --name dind --runtime=kata --privileged docker:19.03.2-dind --storage-driver vfs
...
WARN[2019-09-12T06:12:56.949739356Z] Your kernel does not support cgroup rt period
WARN[2019-09-12T06:12:56.949991027Z] Your kernel does not support cgroup rt runtime
WARN[2019-09-12T06:12:56.950005314Z] Your kernel does not support cgroup blkio weight
WARN[2019-09-12T06:12:56.950011935Z] Your kernel does not support cgroup blkio weight_device
WARN[2019-09-12T06:12:56.950019083Z] Your kernel does not support cgroup blkio throttle.read_bps_device
WARN[2019-09-12T06:12:56.950087701Z] Your kernel does not support cgroup blkio throttle.write_bps_device
WARN[2019-09-12T06:12:56.950093777Z] Your kernel does not support cgroup blkio throttle.read_iops_device
WARN[2019-09-12T06:12:56.950099250Z] Your kernel does not support cgroup blkio throttle.write_iops_device
INFO[2019-09-12T06:12:56.964435792Z] Loading containers: start.
WARN[2019-09-12T06:12:57.053876034Z] Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'
lsmod: /proc/modules: No such file or directory
ip: can't find device 'br_netfilter'
lsmod: /proc/modules: No such file or directory
modprobe: can't change directory to '/lib/modules': No such file or directory
, error: exit status 1
WARN[2019-09-12T06:12:57.132440125Z] Running modprobe nf_nat failed with message: `ip: can't find device 'nf_nat'
lsmod: /proc/modules: No such file or directory
modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1
WARN[2019-09-12T06:12:57.245639773Z] Running modprobe xt_conntrack failed with message: `ip: can't find device 'xt_conntrack'
lsmod: /proc/modules: No such file or directory
modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1
INFO[2019-09-12T06:13:00.139691020Z] Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address
INFO[2019-09-12T06:13:01.378499738Z] Loading containers: done.
INFO[2019-09-12T06:13:01.664019277Z] Docker daemon                                 commit=6a30dfca03 graphdriver(s)=vfs version=19.03.2
INFO[2019-09-12T06:13:01.674672890Z] Daemon has completed initialization
INFO[2019-09-12T06:13:01.886590631Z] API listen on [::]:2376
INFO[2019-09-12T06:13:01.892881057Z] API listen on /var/run/docker.sock

$ docker exec -it dind docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:b8ba256769a0ac28dd126d584e0a2011cd2877f3f76e093a7ae560f2a5301c00
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: failed to create endpoint priceless_davinci on network bridge: failed to add the host (veth5eb5dda) <=> sandbox (veth8f8aeaa) pair interfaces: operation not supported.

[AkihiroSuda]

@bergwolf PTAL?

[AkihiroSuda]

1.9.0-alpha1 is also affected.
1.9.0-alpha0 is not.

[egernst]

/cc @GabyCT - we may need a new unit test.

Thanks for opening @AkihiroSuda

[egernst]

@AkihiroSuda -- we switched how we manage config files for the guest kernel, and it looks like in our quest to minimize, we failed to include overlay. This is a quick fix, and I'll look to get a quick follow-on release to resolve the regression (1.8.2/1.9.0-alpha2).