Runtime: Bind mounts are not working

Created on 28 Feb 2019  ·  11Comments  ·  Source: kata-containers/runtime

Description of problem

# docker run -d -it -v /tmp/test:/tmp/test:rslave --runtime kata-runtime --name test ubuntu
# /tmp/
├── files
│   ├── aaa
│   └── bbb
├── test
# mount --bind /tmp/files/ /tmp/test/
# tree /tmp/
/tmp/
├── files
│   ├── aaa
│   └── bbb
├── test
│   ├── aaa
│   └── bbb
# docker exec -it test bash
root@6fd1a2448b1f:/# ls -lh /tmp/test/
total 0
root@6fd1a2448b1f:/#

New version of kata-time is the same:

# ./kata-runtime -v
kata-runtime  : 1.5.0
   commit   : e8a8e0db799564683fb891dd4566af366e270e90
   OCI specs: 1.0.1-dev

Expected result

(replace this text with an explanation of what you thought would happen)

Actual result

Meta details

Running kata-collect-data.sh version 1.5.0 (commit 5f7fcd7) at 2019-02-28.19:26:57.081206665+0800.

Runtime is /usr/bin/kata-runtime.

kata-env

Output of "/usr/bin/kata-runtime kata-env":

[Meta]
  Version = "1.0.20"

[Runtime]
  Debug = false
  Trace = false
  DisableGuestSeccomp = true
  DisableNewNetNs = false
  Path = "/usr/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.5.0"
    Commit = "5f7fcd7"
    OCI = "1.0.1-dev"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = false
  UseVSock = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.5.0_agent_a581aebf473.img"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.14.67.22-18.container"
  Parameters = "init=/usr/lib/systemd/systemd systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket"

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.5.0-9e77a0b"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = false

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.5.0-efbf3bb"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = false

[Agent]
  Type = "kata"

[Host]
  Kernel = "4.9.0-6-amd64"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = false
  [Host.Distro]
    Name = "Debian GNU/Linux"
    Version = "9"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz"

[Netmon]
  Version = "kata-netmon version 1.5.0"
  Path = "/usr/libexec/kata-containers/kata-netmon"
  Debug = false
  Enable = false

Runtime default config files

/etc/kata-containers/configuration.toml
/usr/share/defaults/kata-containers/configuration.toml

Runtime config file contents

Config file /etc/kata-containers/configuration.toml not found
Output of "cat "/usr/share/defaults/kata-containers/configuration.toml"":
default

Runtime logs

Recent runtime problems found in system journal:

time="2019-02-28T19:09:20.175852299+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=db5b6c01e0338dbd1f9d22286a211c4a2006c633b321c5f4e7382c2a38ab29fb error="open /run/vc/sbs/db5b6c01e0338dbd1f9d22286a211c4a2006c633b321c5f4e7382c2a38ab29fb/devices.json: no such file or directory" name=kata-runtime pid=4097 sandbox=db5b6c01e0338dbd1f9d22286a211c4a2006c633b321c5f4e7382c2a38ab29fb sandboxid=db5b6c01e0338dbd1f9d22286a211c4a2006c633b321c5f4e7382c2a38ab29fb source=virtcontainers subsystem=sandbox
time="2019-02-28T19:11:32.875029009+08:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=6fd1a2448b1f6b61e79fb6d68df0c91e60db8f16ab5dc2cf298c54b83da09614 error="open /run/vc/sbs/6fd1a2448b1f6b61e79fb6d68df0c91e60db8f16ab5dc2cf298c54b83da09614/devices.json: no such file or directory" name=kata-runtime pid=4572 sandbox=6fd1a2448b1f6b61e79fb6d68df0c91e60db8f16ab5dc2cf298c54b83da09614 sandboxid=6fd1a2448b1f6b61e79fb6d68df0c91e60db8f16ab5dc2cf298c54b83da09614 source=virtcontainers subsystem=sandbox

Proxy logs

Recent proxy problems found in system journal:

time="2019-02-28T19:11:17.976413901+08:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/db5b6c01e0338dbd1f9d22286a211c4a2006c633b321c5f4e7382c2a38ab29fb/kata.sock: use of closed network connection" name=kata-proxy pid=4159 sandbox=db5b6c01e0338dbd1f9d22286a211c4a2006c633b321c5f4e7382c2a38ab29fb source=proxy

Shim logs

No recent shim problems found in system journal.

Throttler logs

No recent throttler problems found in system journal.

Container manager details

Have docker

Docker

Output of "docker version":

Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:59 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:02:44 2019
  OS/Arch:          linux/amd64
  Experimental:     true

Output of "docker info":

Containers: 2
 Running: 2
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 18.09.1
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: kata-runtime runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 96ec2177ae841256168fcf76954f7177af9446eb
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-6-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 48
Total Memory: 188.8GiB
ID: 66SV:5RIZ:FLL4:CLLI:JB2Z:2AI2:4ZEO:GTZQ:ZNMF:ZUTO:46OO:REJF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
Registry Mirrors:
 https://registry.docker-cn.com/
Live Restore Enabled: false
Product License: Community Engine

Packages

Have dpkg
Output of "dpkg -l|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"":

ii  kata-containers-image               1.5.0-15                       amd64        Kata containers image
ii  kata-ksm-throttler                  1.5.0.git+3dd4c9f-18           amd64
ii  kata-linux-container                4.14.67.22-18                  amd64        linux kernel optimised for container-like workloads.
ii  kata-proxy                          1.5.0+git.9e77a0b-16           amd64
ii  kata-runtime                        1.5.0+git.5f7fcd7-24           amd64
ii  kata-shim                           1.5.0+git.efbf3bb-14           amd64
ii  qemu-lite                           2.11.0+git.87517afd72-19       amd64        linux kernel optimised for container-like workloads.
ii  qemu-vanilla                        2.11.2+git.0982a56a55-19       amd64        linux kernel optimised for container-like workloads.

No rpm

bug
Source
picture running99
👍1

Most helpful comment

On a separate notes, we should gather all the issues we have encountered with 9p storage (in a single github issue perhaps) to keep track and test that these are solved with virtio-fs.

picture amshinde on 2 Apr 2019
👍3

All 11 comments

Hi @running99 - just to clarify, you are testing that bind mounts on the host appear inside an already active container mount point, yes?
/cc @devimc @sboeuf for input.

grahamwhaley picture grahamwhaley on 28 Feb 2019

Yes, @grahamwhaley

running99 picture running99 on 28 Feb 2019

9p issue, bind mounts are not propagated

devimc picture devimc on 28 Feb 2019

@sboeuf This is a limitation of 9p - symlinks, bind-mounts not getting propogated.
Would virtio-fs solve this?

amshinde picture amshinde on 28 Feb 2019

@amshinde

Would virtio-fs solve this?

Yes! I just tried it out to double check and I can confirm this works.

sboeuf picture sboeuf on 1 Mar 2019
👍2

@running99 This is a 9p bug unfortunately. This should be solved once we move to vitio-fs.
cc @egernst

amshinde picture amshinde on 5 Mar 2019

Was just taking another look at this.
As far as I can see this is primarily caused due to the fact that we set private mount propogation for the rootfs, introduced by this change to mitigate the docker cp issue.
https://github.com/kata-containers/runtime/pull/980/commits/b029e442b2eeb4807feaa60f5d8ea7fe30c03c5a

This is not really a 9p inherent issue. So the questions is when we move to virtio-fs, do we still need private propogation (most likely?) and will we still face this issue.
cc @sboeuf @devimc @egernst

amshinde picture amshinde on 2 Apr 2019

On a separate notes, we should gather all the issues we have encountered with 9p storage (in a single github issue perhaps) to keep track and test that these are solved with virtio-fs.

amshinde picture amshinde on 2 Apr 2019
👍3

@amshinde this is very likely that we will run into the same issue if we set private mount propagation, even in the virtio-fs case.

sboeuf picture sboeuf on 2 Apr 2019

@amshinde I don't think so since the fuse daemon can monitor the merge directory directly

devimc picture devimc on 2 Apr 2019

@amshinde i can't find such type issue based on kata1.5 & kata1.6 + centos 7.3 (3.10) & Ubuntu 18.04(4.15). It seems related with host OS.

zhiminghufighting picture zhiminghufighting on 11 Apr 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

maartenvdezz picture maartenvdezz  ·  4Comments
teawater picture teawater  ·  5Comments
eadamsintel picture eadamsintel  ·  11Comments
sboeuf picture sboeuf  ·  4Comments
AkihiroSuda picture AkihiroSuda  ·  4Comments