Runtime: Name resolution failure when using Docker custom networks (swarm/compose)

Created on 3 Apr 2018  路  42Comments  路  Source: kata-containers/runtime

_From @gvancuts on March 1, 2018 20:55_

Description of problem

Applications built using Docker Compose will fail if the services that compose it depend on name resolution to talk to each other.

Expected result

Docker Compose applications using name resolution would continue to work correctly.

Actual result

The services are not able to talk to each other. This can easily be reproduced by following stesp 1 to 4 of this Docker Compose Getting Started guide: https://docs.docker.com/compose/gettingstarted/. It works correctly if you use Docker with its default runtime (runc) but fails as soon as you switch to Clear Containers 3's runtime (cc-runtime).

_Copied from original issue: clearcontainers/runtime#1042_

limitation needs-help
Source
picture egernst

Most helpful comment

Hello all, I was wondering how the users manage with this issue, I mean usually users are working only without internal docker network or there is a workaround ?

picture fredbcode on 13 May 2019
👍2

All 42 comments

_From @jodh-intel on March 2, 2018 9:6_

Hi @gvancuts - thanks for reporting.

As explained in the issue template, it would be helpful if you could paste in the output of sudo cc-collect-data.sh to give us further details on the problem.

egernst picture egernst on 3 Apr 2018

_From @gvancuts on March 2, 2018 10:52_

Hi @jodh-intel , here is the output from my Fedora 27 machine. Let me know if you also would like the output from my Clear Linux machine. If you have specific, additional debugging you'd like me to turn on, just shoot! Thanks!

$ sudo cc-collect-data.sh

Meta details

Running cc-collect-data.sh version 3.0.19 (commit 900d623) at 2018-03-02.11:50:07.327785583+0100.

Runtime is /bin/cc-runtime.

cc-env

Output of "/bin/cc-runtime cc-env":

[Meta]
  Version = "1.0.8"

[Runtime]
  Debug = false
  [Runtime.Version]
    Semver = "3.0.19"
    Commit = "900d623"
    OCI = "1.0.1"
  [Runtime.Config]
    Path = "/usr/share/defaults/clear-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.7.1(2.7.1+git.d4a337fe91-11.1.cc), Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/qemu-lite-system-x86_64"
  Debug = false

[Image]
  Path = "/usr/share/clear-containers/cc-20640-agent-4186b0.img"

[Kernel]
  Path = "/usr/share/clear-containers/vmlinuz-4.9.60-84.1.container"
  Parameters = ""

[Proxy]
  Type = "ccProxy"
  Version = "Version: 3.0.19+git.318ac7e"
  Path = "/usr/libexec/clear-containers/cc-proxy"
  Debug = false

[Shim]
  Type = "ccShim"
  Version = "shim version: 3.0.19 (commit: 557fe9b)"
  Path = "/usr/libexec/clear-containers/cc-shim"
  Debug = false

[Agent]
  Type = "hyperstart"
  Version = "<<unknown>>"

[Host]
  Kernel = "4.15.3-300.fc27.x86_64"
  Architecture = "amd64"
  VMContainerCapable = true
  [Host.Distro]
    Name = "Fedora"
    Version = "27"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Core(TM) i7-6770HQ CPU @ 2.60GHz"

Runtime config files

Runtime default config files

/usr/share/defaults/clear-containers/configuration.toml
/usr/share/defaults/clear-containers/configuration.toml

Runtime config file contents

Config file /etc/clear-containers/configuration.toml not found
Output of "cat "/usr/share/defaults/clear-containers/configuration.toml"":

# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "config/configuration.toml.in"
# XXX: Project:
# XXX:   Name: Intel庐 Clear Containers
# XXX:   Type: cc

[hypervisor.qemu]
path = "/usr/bin/qemu-lite-system-x86_64"
kernel = "/usr/share/clear-containers/vmlinuz.container"
image = "/usr/share/clear-containers/clear-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per POD/VM:
# unspecified or 0 --> will be set to 1
# < 0              --> will be set to the actual number of physical cores
# > 0 <= 255       --> will be set to the specified number
# > 255            --> will be set to 255
default_vcpus = -1


# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per POD/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for POD/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048
disable_block_device_use = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically 
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
# 
# Default false
#enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
# 
#disable_nesting_checks = true

[proxy.cc]
path = "/usr/libexec/clear-containers/cc-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[shim.cc]
path = "/usr/libexec/clear-containers/cc-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[agent.cc]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
#   - bridged
#     Uses a linux bridge to interconnect the container interface to
#     the VM. Works for most cases except macvlan and ipvlan.
#
#   - macvtap
#     Used when the Container network interface can be bridged using
#     macvtap.
internetworking_model="macvtap"

Logfiles

Runtime logs

Recent runtime problems found in system journal:

time="2018-02-28T21:18:47.663516121+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:18:55.292102004+01:00" level=error msg="Container not ready or running, impossible to signal the container" source=runtime
time="2018-02-28T21:18:55.32461801+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:19:03.744498522+01:00" level=error msg="Container not ready or running, impossible to signal the container" source=runtime
time="2018-02-28T21:19:03.776688495+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:19:13.501156989+01:00" level=error msg="Container not ready or running, impossible to signal the container" source=runtime
time="2018-02-28T21:19:13.532568091+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:19:24.342049176+01:00" level=error msg="Container not ready or running, impossible to signal the container" source=runtime
time="2018-02-28T21:19:24.373630996+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:19:38.781108422+01:00" level=error msg="Container not ready or running, impossible to signal the container" source=runtime
time="2018-02-28T21:19:38.811340469+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:19:59.193989682+01:00" level=error msg="Container not ready or running, impossible to signal the container" source=runtime
time="2018-02-28T21:19:59.223581366+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:20:32.876844229+01:00" level=error msg="Container not ready or running, impossible to signal the container" source=runtime
time="2018-02-28T21:20:32.9068921+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:21:25.720982921+01:00" level=error msg="Container 5cb1ee24e1886b158b7b23a761ce477eb2394bbeead993b81dc57e2492c5b7f3 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:21:25.727206702+01:00" level=error msg="Container d4a9eb856b314c2cdc9975c8974e7883679d8f8acec1e321ee5388cc5d8000b3 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:21:25.751230189+01:00" level=error msg="Container 5cb1ee24e1886b158b7b23a761ce477eb2394bbeead993b81dc57e2492c5b7f3 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:21:25.757586563+01:00" level=error msg="Container d4a9eb856b314c2cdc9975c8974e7883679d8f8acec1e321ee5388cc5d8000b3 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:21:36.583984914+01:00" level=error msg="Container not ready or running, impossible to signal the container" source=runtime
time="2018-02-28T21:21:36.611343096+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:21:48.140403921+01:00" level=error msg="Container 5cb1ee24e1886b158b7b23a761ce477eb2394bbeead993b81dc57e2492c5b7f3 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:21:48.148178284+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:21:48.169489898+01:00" level=error msg="Container 5cb1ee24e1886b158b7b23a761ce477eb2394bbeead993b81dc57e2492c5b7f3 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:21:48.175397363+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:21:57.11218988+01:00" level=error msg="Container not ready or running, impossible to signal the container" source=runtime
time="2018-02-28T21:21:57.147906574+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:22:04.556458145+01:00" level=error msg="Container not ready or running, impossible to signal the container" source=runtime
time="2018-02-28T21:22:04.585055303+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:22:22.11785248+01:00" level=error msg="Container 5cb1ee24e1886b158b7b23a761ce477eb2394bbeead993b81dc57e2492c5b7f3 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:22:22.124523089+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:22:22.150037983+01:00" level=error msg="Container 5cb1ee24e1886b158b7b23a761ce477eb2394bbeead993b81dc57e2492c5b7f3 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:22:22.154721531+01:00" level=error msg="Container 916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:22:22.752811675+01:00" level=error msg="Container d4a9eb856b314c2cdc9975c8974e7883679d8f8acec1e321ee5388cc5d8000b3 not ready or running, cannot send a signal" source=runtime
time="2018-02-28T21:22:22.791242847+01:00" level=error msg="Container d4a9eb856b314c2cdc9975c8974e7883679d8f8acec1e321ee5388cc5d8000b3 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:15:51.513860568+01:00" level=error msg="Container 94f0af192b110aca2cf691649682bdb86b055275e6c2d05fb652b5a5a74c3f03 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:15:51.520435032+01:00" level=error msg="Container 7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:15:51.545352546+01:00" level=error msg="Container 94f0af192b110aca2cf691649682bdb86b055275e6c2d05fb652b5a5a74c3f03 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:15:51.549786457+01:00" level=error msg="Container 7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:18:48.133687907+01:00" level=error msg="ERROR received from VM agent, control msg received : Process could not be started: container_linux.go:296: starting container process caused \"exec: \\\"/bin/bash\\\": stat /bin/bash: no such file or directory\"" source=runtime
time="2018-03-01T22:18:58.32919645+01:00" level=error msg="ERROR received from VM agent, control msg received : Process could not be started: container_linux.go:296: starting container process caused \"exec: \\\"/sbin/bash\\\": stat /sbin/bash: no such file or directory\"" source=runtime
time="2018-03-01T22:19:04.497648266+01:00" level=error msg="ERROR received from VM agent, control msg received : Process could not be started: container_linux.go:296: starting container process caused \"exec: \\\"bash\\\": executable file not found in $PATH\"" source=runtime
time="2018-03-01T22:25:08.12909365+01:00" level=error msg="Container 94f0af192b110aca2cf691649682bdb86b055275e6c2d05fb652b5a5a74c3f03 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:25:08.140169208+01:00" level=error msg="Container 7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:25:08.173568807+01:00" level=error msg="Container 94f0af192b110aca2cf691649682bdb86b055275e6c2d05fb652b5a5a74c3f03 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:25:08.181373326+01:00" level=error msg="Container 7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:27:25.299973936+01:00" level=error msg="Container df86b51c42d56324714619929f139994ef67c555e54a5b68f58bd75f2aa2c2c2 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:27:25.315432984+01:00" level=error msg="Container b89d9b82528cf561fa658962c42a24cbda1a79fe51936f25305560482369e9a3 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:27:25.400627119+01:00" level=error msg="Container df86b51c42d56324714619929f139994ef67c555e54a5b68f58bd75f2aa2c2c2 not ready or running, cannot send a signal" source=runtime
time="2018-03-01T22:27:25.406428328+01:00" level=error msg="Container b89d9b82528cf561fa658962c42a24cbda1a79fe51936f25305560482369e9a3 not ready or running, cannot send a signal" source=runtime

Proxy logs

Recent proxy problems found in system journal:

time="2018-02-28T21:13:19.721426857+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=11970 source=proxy
time="2018-02-28T21:13:34.108554836+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=12148 source=proxy
time="2018-02-28T21:13:54.72420581+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=12323 source=proxy
time="2018-02-28T21:14:28.344572069+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=12593 source=proxy
time="2018-02-28T21:15:27.268590967+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=12858 source=proxy
time="2018-02-28T21:16:34.867902939+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=13089 source=proxy
time="2018-02-28T21:17:00.09513485+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=13292 source=proxy
time="2018-02-28T21:17:08.620269946+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=13471 source=proxy
time="2018-02-28T21:17:17.101334895+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=13640 source=proxy
time="2018-02-28T21:17:25.676465366+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=13810 source=proxy
time="2018-02-28T21:17:33.861087168+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=13987 source=proxy
time="2018-02-28T21:17:43.532682966+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=14159 source=proxy
time="2018-02-28T21:17:54.379966602+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=14318 source=proxy
time="2018-02-28T21:18:08.864153736+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=14509 source=proxy
time="2018-02-28T21:18:12.570087667+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/d4a9eb856b314c2cdc9975c8974e7883679d8f8acec1e321ee5388cc5d8000b3/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=10860 source=proxy
time="2018-02-28T21:18:22.614523923+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/5cb1ee24e1886b158b7b23a761ce477eb2394bbeead993b81dc57e2492c5b7f3/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=10924 source=proxy
time="2018-02-28T21:18:30.563386924+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=14863 source=proxy
time="2018-02-28T21:18:39.094516915+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=15231 source=proxy
time="2018-02-28T21:18:47.531899729+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=15421 source=proxy
time="2018-02-28T21:18:55.180182482+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=15587 source=proxy
time="2018-02-28T21:19:03.631922862+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=15762 source=proxy
time="2018-02-28T21:19:13.379834503+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=15922 source=proxy
time="2018-02-28T21:19:24.225490913+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=16154 source=proxy
time="2018-02-28T21:19:38.670207607+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=16381 source=proxy
time="2018-02-28T21:19:59.071222018+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=16626 source=proxy
time="2018-02-28T21:20:32.775707131+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=16838 source=proxy
time="2018-02-28T21:21:15.622029717+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/d4a9eb856b314c2cdc9975c8974e7883679d8f8acec1e321ee5388cc5d8000b3/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=14996 source=proxy
time="2018-02-28T21:21:25.592148699+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/5cb1ee24e1886b158b7b23a761ce477eb2394bbeead993b81dc57e2492c5b7f3/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=15062 source=proxy
time="2018-02-28T21:21:36.483956352+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=17493 source=proxy
time="2018-02-28T21:21:44.852858203+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=17731 source=proxy
time="2018-02-28T21:21:48.012557028+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/5cb1ee24e1886b158b7b23a761ce477eb2394bbeead993b81dc57e2492c5b7f3/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=17557 source=proxy
time="2018-02-28T21:21:56.980477023+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=18136 source=proxy
time="2018-02-28T21:22:04.44142081+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=18355 source=proxy
time="2018-02-28T21:22:12.876275834+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/916c8e65accb3d1cd1fc9218ec9666891e29d899201332d52ca59e6d72f9add7/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=18520 source=proxy
time="2018-02-28T21:22:21.99189055+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/5cb1ee24e1886b158b7b23a761ce477eb2394bbeead993b81dc57e2492c5b7f3/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=18169 source=proxy
time="2018-02-28T21:22:22.549024385+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/d4a9eb856b314c2cdc9975c8974e7883679d8f8acec1e321ee5388cc5d8000b3/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=17957 source=proxy
time="2018-03-01T22:15:51.345407846+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/94f0af192b110aca2cf691649682bdb86b055275e6c2d05fb652b5a5a74c3f03/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=9433 source=proxy
time="2018-03-01T22:15:51.392308659+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=9368 source=proxy
time="2018-03-01T22:19:18.135742135+01:00" level=error msg="timeout waiting for process with token LJHPFHenzBoHhMUcupZTfWkzLRnfy2wlhDNwHlhwtDo=" name=cc-proxy pid=19926 section=io source=proxy vm=7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812
time="2018-03-01T22:19:18.137147659+01:00" level=error msg="error serving client: timeout waiting for process with token LJHPFHenzBoHhMUcupZTfWkzLRnfy2wlhDNwHlhwtDo=" client=8 name=cc-proxy pid=19926 source=proxy
time="2018-03-01T22:19:28.330454921+01:00" level=error msg="timeout waiting for process with token e-UGjniQM-R5UY-POWoP5lpJTSHxIuafYgSOg5XCeyU=" name=cc-proxy pid=19926 section=io source=proxy vm=7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812
time="2018-03-01T22:19:28.330659399+01:00" level=error msg="error serving client: timeout waiting for process with token e-UGjniQM-R5UY-POWoP5lpJTSHxIuafYgSOg5XCeyU=" client=11 name=cc-proxy pid=19926 source=proxy
time="2018-03-01T22:19:34.499001895+01:00" level=error msg="timeout waiting for process with token EEgcuxiK7ax7HalFuVzoB8gTiy_2yR05MDPjVuynFuA=" name=cc-proxy pid=19926 section=io source=proxy vm=7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812
time="2018-03-01T22:19:34.499236554+01:00" level=error msg="error serving client: timeout waiting for process with token EEgcuxiK7ax7HalFuVzoB8gTiy_2yR05MDPjVuynFuA=" client=14 name=cc-proxy pid=19926 source=proxy
time="2018-03-01T22:25:04.382996394+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock->@: use of closed network connection" client=17 name=cc-proxy pid=19926 source=proxy
time="2018-03-01T22:25:07.899000303+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/94f0af192b110aca2cf691649682bdb86b055275e6c2d05fb652b5a5a74c3f03/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=19990 source=proxy
time="2018-03-01T22:25:07.964708121+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=19926 source=proxy
time="2018-03-01T22:27:21.153887724+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/df86b51c42d56324714619929f139994ef67c555e54a5b68f58bd75f2aa2c2c2/proxy.sock->@: use of closed network connection" client=8 name=cc-proxy pid=21060 source=proxy
time="2018-03-01T22:27:24.950612868+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/df86b51c42d56324714619929f139994ef67c555e54a5b68f58bd75f2aa2c2c2/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=21060 source=proxy
time="2018-03-01T22:27:25.148172809+01:00" level=error msg="error serving client: write unix /run/virtcontainers/pods/b89d9b82528cf561fa658962c42a24cbda1a79fe51936f25305560482369e9a3/proxy.sock->@: use of closed network connection" client=4 name=cc-proxy pid=21129 source=proxy

Shim logs

Recent shim problems found in system journal:

time="2018-03-01T22:25:16.806350944+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:16.806364009+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:16.906602665+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:16.906619661+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:16.906622867+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.007160388+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.007160593+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.007160466+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.107419963+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.107448422+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.107446516+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.207739528+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.207738460+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.207738478+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.308072782+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.308072964+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.308096097+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.408417546+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.408415968+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.408415355+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.508681915+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.508682457+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.508860036+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.609024456+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.609020378+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.609128937+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.709392399+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.709392332+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.709406710+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.809710268+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.809711580+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.809711756+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.910002968+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.910000406+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:17.910022480+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.010326587+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.010329096+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.010329405+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.110524759+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.110524858+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.110524936+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.210725790+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.210726095+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.210726461+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.310954435+0100" level="warning" pid=20292 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.310956614+0100" level="warning" pid=20338 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.310956619+0100" level="warning" pid=20376 function="establish_connection_to_proxy" line=905 source="shim" name="cc-shim" msg="Error while connecting to proxy with address /run/virtcontainers/pods/7fc2f614a04123732a615f6a046373aa954f1fc33ec664ce64c55c6aedc2f812/proxy.sock: No such file or directory"
time="2018-03-01T22:25:18.411170650+0100" level="error" pid=20292 function="reconnect_to_proxy" line=1027 source="shim" name="cc-shim" msg="Failed to reconnect to cc-proxy (timeout 10 s)"
time="2018-03-01T22:25:18.411172545+0100" level="error" pid=20376 function="reconnect_to_proxy" line=1027 source="shim" name="cc-shim" msg="Failed to reconnect to cc-proxy (timeout 10 s)"
time="2018-03-01T22:25:18.411172624+0100" level="error" pid=20338 function="reconnect_to_proxy" line=1027 source="shim" name="cc-shim" msg="Failed to reconnect to cc-proxy (timeout 10 s)"

Container manager details

Have docker

Docker

Output of "docker version":

Client:
 Version:   18.02.0-ce
 API version:   1.36
 Go version:    go1.9.3
 Git commit:    fc4de44
 Built: Wed Feb  7 21:15:49 2018
 OS/Arch:   linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:  18.02.0-ce
  API version:  1.36 (minimum version 1.12)
  Go version:   go1.9.3
  Git commit:   fc4de44
  Built:    Wed Feb  7 21:19:22 2018
  OS/Arch:  linux/amd64
  Experimental: false

Output of "docker info":

Containers: 4
 Running: 0
 Paused: 0
 Stopped: 4
Images: 47
Server Version: 18.02.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: cc-runtime runc
Default Runtime: cc-runtime
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 900d623 (expected: 9f9c96235cc97674e935002fc3d78361b696a69e)
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.15.3-300.fc27.x86_64
Operating System: Fedora 27 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.31GiB
Name: geoffroy-nuc-fedora
ID: RBT4:PUIM:7MLH:HPNW:DZOW:EUYU:6BWH:GGXU:HRN5:E6WZ:M6NT:PMO3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 22
 Goroutines: 35
 System Time: 2018-03-02T11:50:07.572411172+01:00
 EventsListeners: 0
Username: gvancuts
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Output of "systemctl show docker":

Type=notify
Restart=on-failure
NotifyAccess=main
RestartUSec=100ms
TimeoutStartUSec=infinity
TimeoutStopUSec=1min 30s
RuntimeMaxUSec=infinity
WatchdogUSec=0
WatchdogTimestamp=Fri 2018-03-02 11:47:53 CET
WatchdogTimestampMonotonic=2239835526
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=4845
ControlPID=0
FileDescriptorStoreMax=0
NFileDescriptorStore=0
StatusErrno=0
Result=success
UID=4294967295
GID=4294967295
ExecMainStartTimestamp=Fri 2018-03-02 11:47:51 CET
ExecMainStartTimestampMonotonic=2238597567
ExecMainExitTimestampMonotonic=0
ExecMainPID=4845
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -D --add-runtime cc-runtime=/usr/bin/cc-runtime --default-runtime=cc-runtime ; ignore_errors=no ; start_time=[Fri 2018-03-02 11:47:51 CET] ; stop_time=[n/a] ; pid=4845 ; code=(null) ; status=0/0 }
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/docker.service
MemoryCurrent=30797824
CPUUsageNSec=916217376
TasksCurrent=35
Delegate=yes
CPUAccounting=no
CPUWeight=18446744073709551615
StartupCPUWeight=18446744073709551615
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
IOAccounting=no
IOWeight=18446744073709551615
StartupIOWeight=18446744073709551615
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLow=0
MemoryHigh=18446744073709551615
MemoryMax=18446744073709551615
MemorySwapMax=18446744073709551615
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=yes
TasksMax=18446744073709551615
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615
LimitFSIZE=18446744073709551615
LimitFSIZESoft=18446744073709551615
LimitDATA=18446744073709551615
LimitDATASoft=18446744073709551615
LimitSTACK=18446744073709551615
LimitSTACKSoft=8388608
LimitCORE=18446744073709551615
LimitCORESoft=18446744073709551615
LimitRSS=18446744073709551615
LimitRSSSoft=18446744073709551615
LimitNOFILE=18446744073709551615
LimitNOFILESoft=18446744073709551615
LimitAS=18446744073709551615
LimitASSoft=18446744073709551615
LimitNPROC=18446744073709551615
LimitNPROCSoft=18446744073709551615
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=18446744073709551615
LimitLOCKSSoft=18446744073709551615
LimitSIGPENDING=127462
LimitSIGPENDINGSoft=127462
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=18446744073709551615
LimitRTTIMESoft=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOSchedulingClass=0
IOSchedulingPriority=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
DynamicUser=no
RemoveIPC=no
MountFlags=0
PrivateTmp=no
PrivateDevices=no
ProtectKernelTunables=no
ProtectKernelModules=no
ProtectControlGroups=no
PrivateNetwork=no
PrivateUsers=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
MemoryDenyWriteExecute=no
RestrictRealtime=no
RestrictNamespaces=no
MountAPIVFS=no
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=docker.service
Names=docker.service
Requires=sysinit.target system.slice
Wants=network-online.target
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=basic.target system.slice systemd-journald.socket network-online.target firewalld.service sysinit.target
Documentation=https://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/usr/lib/systemd/system/docker.service
DropInPaths=/etc/systemd/system/docker.service.d/clear-containers.conf
UnitFileState=enabled
UnitFilePreset=disabled
StateChangeTimestamp=Fri 2018-03-02 11:47:53 CET
StateChangeTimestampMonotonic=2239835527
InactiveExitTimestamp=Fri 2018-03-02 11:47:51 CET
InactiveExitTimestampMonotonic=2238597607
ActiveEnterTimestamp=Fri 2018-03-02 11:47:53 CET
ActiveEnterTimestampMonotonic=2239835527
ActiveExitTimestamp=Fri 2018-03-02 11:47:50 CET
ActiveExitTimestampMonotonic=2237575034
InactiveEnterTimestamp=Fri 2018-03-02 11:47:51 CET
InactiveEnterTimestampMonotonic=2238595224
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobRunningTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Fri 2018-03-02 11:47:51 CET
ConditionTimestampMonotonic=2238596666
AssertTimestamp=Fri 2018-03-02 11:47:51 CET
AssertTimestampMonotonic=2238596666
Transient=no
Perpetual=no
StartLimitIntervalSec=60000000
StartLimitBurst=3
StartLimitAction=none
InvocationID=7e3afb824c694da8b8c8182139566c2b

No kubectl

Packages

No dpkg
Have rpm
Output of "rpm -qa|egrep "(cc-oci-runtime|cc-proxy|cc-runtime|cc-shim|kata-proxy|kata-runtime|kata-shim|clear-containers-image|linux-container|qemu-lite|qemu-system-x86)"":

cc-runtime-config-3.0.19+git.900d623-24.1.x86_64
linux-container-4.9.60-84.1.x86_64
cc-shim-3.0.19+git.557fe9b-24.1.x86_64
cc-proxy-bin-3.0.19+git.318ac7e-24.1.x86_64
qemu-lite-2.7.1+git.d4a337fe91-11.1.x86_64
cc-runtime-3.0.19+git.900d623-24.1.x86_64
cc-shim-bin-3.0.19+git.557fe9b-24.1.x86_64
qemu-system-x86-2.10.1-2.fc27.x86_64
cc-runtime-bin-3.0.19+git.900d623-24.1.x86_64
clear-containers-image-20640-47.1.x86_64
qemu-lite-bin-2.7.1+git.d4a337fe91-11.1.x86_64
qemu-lite-data-2.7.1+git.d4a337fe91-11.1.x86_64
cc-proxy-3.0.19+git.318ac7e-24.1.x86_64
qemu-system-x86-core-2.10.1-2.fc27.x86_64
egernst picture egernst on 3 Apr 2018

_From @jodh-intel on March 2, 2018 15:29_

ping @egernst, @mcastelino.

egernst picture egernst on 3 Apr 2018

ack, @jodh-intel -- taking a look at this today.

egernst picture egernst on 3 Apr 2018

_From @mcastelino on March 2, 2018 17:47_

@gvancuts we have an issue with DNS w.rt. swarm as described here.
https://github.com/clearcontainers/runtime/issues/121

I suspect this is also a similar issue. We will root cause this failure.

/cc @egernst @jodh-intel

egernst picture egernst on 3 Apr 2018

_From @mcastelino on March 2, 2018 22:12_

@gvancuts just confirming that this the same issue we have seen with docker swarm. Docker is running a resolver in the network namespace of the container and running all DNS traffic through it. And only if dockerd cannot handle the same, it is then send to the host.

egernst picture egernst on 3 Apr 2018

_From @amshinde on March 2, 2018 22:25_

@mcastelino I ran a bunch of tests and see the same behaviour, containers are on a separate network with an separate dns resolver(127.0.0.11) handling the name resolution for the network.

egernst picture egernst on 3 Apr 2018

_From @mcastelino on March 2, 2018 23:2_

@amshinde is this true for all custom networks with docker. That means that we have a broader problem with all docker advanced networks. If so we should priortize tc mirroring support in Kata. As we have a potential path to addressing this once we have tc support.

/cc @egernst

egernst picture egernst on 3 Apr 2018

_From @amshinde on March 2, 2018 23:7_

@mcastelino This seems to be the case for all custom networks other than the default bridge network : https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/

egernst picture egernst on 3 Apr 2018

_From @sboeuf on March 2, 2018 23:20_

@mcastelino I am curious to understand how the tc mirroring case can solve this issue. Is there an explanation somewhere ?

egernst picture egernst on 3 Apr 2018

_From @mcastelino on March 2, 2018 23:33_

@sboeuf

Here are the details

If we implement support in the agent to proxy the DNS requests coming in from the VM by listening to 127.0.0.11:53 and sending it to the shim running in the host name space, which can forward the request on to dockerd, internal DNS resolution will work. This is a bit of work, but technically can be done, and will need to built into our gRPC protocol.

However the external DNS resolution will not work, details are below

Internal DNS Resolution

Internal DNS resolution is handled completely by dockerd. So dockerd directly responds to the DNS request from the container process for any cluster local resource.

External DNS Resolution

External DNS resolution is not handled by dockerd. When dockerd is unable to resolve the name to a cluster local resource it will then perform a DNS resolution using the host's resolv.conf.

Hence the DNS resolution process for external name is

Request path: process (shim) -> dockerd:43411  (on resolv failure) -> host (via the namespace) -> external DNS -> host 
Response path: host-> dockerd -> process (shim)

Here you will notice, dockerd sends packets out from within the namespace to the host via the interface bound to the docker_gwbridge.

In the case of clear containers using macvtap/bridge as there is network connectivity between the container network namespace and the host, this request can never be fulfilled as there is no path out of the network namespace on the host to the host network.

tc based external DNS resolution

In the case of the tc based approach, the network interface exists and is fully active in the network namespace on the host side. So this external DNS resolution traffic can be sent down the interface on the host side. The response received will have to be send to dockerd (i.e. not mirrored back into VM).

So with tc, we have a path to a potential solution even though it is complex.

That said given that we have described the problem, I am also open to simpler solutions.

egernst picture egernst on 3 Apr 2018

_From @sboeuf on March 2, 2018 23:47_

@mcastelino I don't know if you know, but we now spawn our shims in the pod network namespace. This means we have a process that could be able to actually redirect external DNS requests, right ?

egernst picture egernst on 3 Apr 2018

_From @mcastelino on March 2, 2018 23:51_

@sboeuf yes I am aware of that and that is good, and makes this solution possible in the first place. That in itself will not solve the external or internal DNS issue. For the internal dns we need to add more logic to the shim. Once we add that the external DNS should just work, provided we setup the tc mirroring rules properly to allow external DNS responses to make their way back to dockerd.

So in short you still need a path out/in to the host network via the docker setup veth interface. Which is where tc comes in.

egernst picture egernst on 3 Apr 2018

Since this is also observed in kata containers at this point, I think we should move this over there and open a new issue. I'd recommend marking swarm/compose as a "not fix" for Clear.

This way it'll be a bit easier/more straightforward for asking for input from other Kata contributors...

Agreed?

egernst picture egernst on 3 Apr 2018

_From @sboeuf on March 3, 2018 0:5_

Yes makes sense to me @egernst

egernst picture egernst on 3 Apr 2018

/cc @bekars

egernst picture egernst on 19 Jun 2018

Same issue, without docker-compose :

docker network create net1
docker run -it -d --rm --network net1 --network-alias bb1 busybox sleep 30
docker run -it  --rm --network net1 --network-alias bb2 busybox ping bb1

expected output :

PING bb1 (172.18.0.2): 56 data bytes
64 bytes from 172.18.0.2: seq=0 ttl=64 time=0.116 ms

actual output :

ping: bad address 'bb1'
Gabasjob picture Gabasjob on 11 Oct 2018

@mcastelino I don't know if you know, but we now spawn our shims in the pod network namespace. This means we have a process that could be able to actually redirect external DNS requests, right ?

@sboeuf that does not help as we do not have network connectivity back to the host from the network namespace. So only if we add support of tc will we have any chance of fixing this issue

/cc @amshinde

mcastelino picture mcastelino on 11 Oct 2018

Here is a dirty workaround :
Using deprecated features of dockerd (link) and deprecated docker-compose file.
The principle is to user /etc/hosts file instead of build-in dns :
docker-compose.yml

bb1:
  image: busybox
  command: "ping bb2"
  links:
  - bb2
bb2:
  image: busybox
  command: "sleep 5"

output of docker-compose up :

Starting bugkata_bb2_1 ... 
Starting bugkata_bb2_1 ... done
Recreating bugkata_bb1_1 ... 
Recreating bugkata_bb1_1 ... done
Attaching to bugkata_bb2_1, bugkata_bb1_1
bb1_1  | PING bb2 (172.17.0.2): 56 data bytes
bb1_1  | 64 bytes from 172.17.0.2: seq=0 ttl=64 time=0.391 ms

please note that this workaround doesn't cover every case since many features of docker-compose are not present in version 1

Gabasjob picture Gabasjob on 12 Oct 2018

I don't want to rush you, but I like to know if you have any clue to fix this?
With #785 fixed, kata is almost production ready for me now :)

fredbcode picture fredbcode on 19 Oct 2018

@fredbcode I am planning to try out a solution this week based on proposal outlined in this comment to see if this works out:
https://github.com/kata-containers/runtime/issues/175#issuecomment-378333209

We already have some work done to implement tc mirroring in Kata.

amshinde picture amshinde on 22 Oct 2018

@amshinde I also hope to have some good news soon

alebourdoulous picture alebourdoulous on 28 Oct 2018
👍1

Hello there,
Any news on this issue ?

Gabasjob picture Gabasjob on 5 Nov 2018
👍1

@fredbcode @Gabasjob @alebourdoulous we prototyped a solution for this last week which works with tc based networking. @amshinde will be creating a design document and presenting the proposed design in the next kata architecture forum for feedback. We will also raise a new issue to obtain feedback on this approach shortly.

mcastelino picture mcastelino on 5 Nov 2018

Just to note (as discussed in arch call today) - that might be presented in 3 weeks time at a guess, as next week many folks are out travelling for conferences etc., and the week after that is thanksgiving week - so, best check/confirm etc.

grahamwhaley picture grahamwhaley on 5 Nov 2018

@mcastelino @grahamwhaley Thanks !
I would be pleased to make some tests, if needed

fredbcode picture fredbcode on 6 Nov 2018

Here is the link to the slides with the proposed solution I presented in the Kata architecture forum:
https://docs.google.com/presentation/d/1ivBToHgNY0fWEhGBaaE6tdUz7p-mn1M0iFKT-Xw6bRw/edit?usp=sharing

@bergwolf @gnawux @WeiZhang555 @jon PTAL and add in comments if you have any feedback/questions.

cc @egernst

amshinde picture amshinde on 19 Nov 2018

@fredbcode @Gabasjob @alebourdoulous We have got the ball rolling on this one. This does involve substantial changes that need to be implemented. Will let you know once we have this ready for testing.

amshinde picture amshinde on 19 Nov 2018

Thanks, please let me know if you need help for testing

fredbcode picture fredbcode on 30 Nov 2018

@amshinde Hello, there will be a testing version soon ?

fredbcode picture fredbcode on 20 Dec 2018

Hello, I am really sorry to insist on this, but right now Kata is unusable in most cases, using internal network is a very basic situation. It's very frustrating to have a such great project disabled on our systems :)
Like I said before, I'm very interested to help, in any ways I can

fredbcode picture fredbcode on 2 Feb 2019

Maybe the team focus on integration with k8s more than docker swarm now. PR is welcome

caoruidong picture caoruidong on 3 Feb 2019

Unfortunately this issue is more deeper than only a specific problem with swarm or compose
Without any orchestrator: https://github.com/kata-containers/runtime/issues/785#issuecomment-428631242

fredbcode picture fredbcode on 3 Feb 2019

@amshinde @mcastelino - we should probably have an update on any progress or plans here - or if we don't have resource to implement, have we documented how it could be done so somebody else might be able to pick it up?

grahamwhaley picture grahamwhaley on 4 Feb 2019

@grahamwhaley I probably would not be able to get to this for atleast couple of weeks.

Here is the link to the slides with the proposed solution I presented in the Kata architecture forum:
https://docs.google.com/presentation/d/1ivBToHgNY0fWEhGBaaE6tdUz7p-mn1M0iFKT-Xw6bRw/edit?usp=sharing

I have included the proposal I came up in the link above. Anyone interested in picking this up can refer to the design proposal above

amshinde picture amshinde on 5 Feb 2019

Hello,
Any news about this issue ?

alebourdoulous picture alebourdoulous on 4 Apr 2019
👀1

Hello all, I was wondering how the users manage with this issue, I mean usually users are working only without internal docker network or there is a workaround ?

fredbcode picture fredbcode on 13 May 2019
👍2

/cc @amshinde @mcastelino

grahamwhaley picture grahamwhaley on 13 May 2019

+1

itsgk92 picture itsgk92 on 8 Nov 2019

Has anyone found a different workaround than these 3 :

  • Configuring fully static custom networks + bind volume on /etc/hosts and /etc/resolv.conf: Very error prone and much more maintenance
  • Not using custom networks + the docker link feature (legacy): Less secure + link feature might be removed
  • Not using kata: Much less secure

?

nab-os picture nab-os on 19 Aug 2020

have you tried with the netmon ?

devimc picture devimc on 19 Aug 2020

Yes, but still not able to reach Docker's embedded DNS

nab-os picture nab-os on 19 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

gnawux picture gnawux  路  11Comments
jcvenegas picture jcvenegas  路  7Comments
cespo picture cespo  路  11Comments
mcastelino picture mcastelino  路  8Comments
egernst picture egernst  路  11Comments