K3d: [BUG] k3d not working with "Docker Toolbox on Windows"

[kkimdev]

What did you do?
Tried installing k3d on Windows 10. With Docker Toolbox on Windows https://docs.docker.com/toolbox/toolbox_install_windows/

• How was the cluster created?

  • k3d create

• What did you do afterwards?

  • Inspect if the kube-system pods(e.g., coredns) are running successfully

What did you expect to happen?
No errors.

Screenshots or terminal output

• Full log: https://gist.github.com/kkimdev/fcec5e938d216677f26007116575f33f
• Additional information: https://gist.github.com/kkimdev/557a8c7b0227cd5ce0d9bc397852ad23

Which OS & Architecture?

• Windows, amd64

Which version of k3d?

• k3d version v1.2.1

Which version of docker?

$ docker version
Client:
 Version:           18.09.5
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        e8ff056
 Built:             Thu Apr 11 04:43:57 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:41:57 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[iwilltry42]

Hey there, thanks for creating this issue... Easier to follow-up than the slack chat :+1:
Unfortunately, I couldn't check this on windows yet and didn't have the time to properly read through the error messages.
But maybe in the meantime, someone who's using k3d on Windows/WSL can write something about their setup (@nunix maybe? :) )

[nunix]

hi Team, indeed I’m writing a blog post how to get started with k3d and wsl.

Still, the difference will be that I use Docker Desktop and not the toolbox, but the setup might give you ideas.

And, @iwilltry42, if really you fell like adding a nice feature soon then could look for handling the ssh:// docker_host capability?
This would solve/ease quite some setups 😁

I will update this thread once I have the blog post out

[kkimdev]

I tried again with k3d 1.2.2 today but the problem still persists.

$ k3d --version
k3d version v1.2.2

$ kubectl logs coredns-695688789-rf6qn --namespace kube-system
.:53
2019-06-07T20:08:54.721Z [INFO] CoreDNS-1.3.0
2019-06-07T20:08:54.721Z [INFO] linux/amd64, go1.11.4, c8f0e94
CoreDNS-1.3.0
linux/amd64, go1.11.4, c8f0e94
2019-06-07T20:08:54.721Z [INFO] plugin/reload: Running configuration MD5 = ef347efee19aa82f09972f89f92da1cf
E0607 20:09:19.722915       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:322: Failed to list *v1.Namespace: Get https://10.43.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: i/o timeout
E0607 20:09:19.726107       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:317: Failed to list *v1.Endpoints: Get https://10.43.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: i/o timeout
E0607 20:09:19.726740       1 reflector.go:205] github.com/coredns/coredns/plugin/kubernetes/controller.go:315: Failed to list *v1.Service: Get https://10.43.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.43.0.1:443: i/o timeout

$  docker logs k3d-k3s-default-server
time="2019-06-07T20:13:22.950796265Z" level=info msg="Starting k3s v0.5.0 (8c0116dd)"
time="2019-06-07T20:13:24.981478532Z" level=info msg="Running kube-apiserver --bind-address=127.0.0.1 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/localhost.key --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-issuer=k3s --api-audiences=unknown --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --watch-cache=false --authorization-mode=Node,RBAC --requestheader-extra-headers-prefix=X-Remote-Extra- --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/token-node-1.crt --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --secure-port=6444 --insecure-port=0 --tls-cert-file=/var/lib/rancher/k3s/server/tls/localhost.crt --basic-auth-file=/var/lib/rancher/k3s/server/cred/passwd --kubelet-client-key=/var/lib/rancher/k3s/server/tls/token-node.key --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-group-headers=X-Remote-Group --allow-privileged=true --advertise-port=6445 --advertise-address=127.0.0.1 --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-allowed-names=kubernetes-proxy --requestheader-username-headers=X-Remote-User --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --service-cluster-ip-range=10.43.0.0/16"
E0607 20:13:26.286312       1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0607 20:13:26.286721       1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0607 20:13:26.286826       1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0607 20:13:26.286907       1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0607 20:13:26.287009       1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0607 20:13:26.287117       1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
W0607 20:13:26.362989       1 genericapiserver.go:315] Skipping API batch/v2alpha1 because it has no resources.
W0607 20:13:26.369368       1 genericapiserver.go:315] Skipping API node.k8s.io/v1alpha1 because it has no resources.
E0607 20:13:26.405625       1 prometheus.go:138] failed to register depth metric admission_quota_controller: duplicate metrics collector registration attempted
E0607 20:13:26.405759       1 prometheus.go:150] failed to register adds metric admission_quota_controller: duplicate metrics collector registration attempted
E0607 20:13:26.405835       1 prometheus.go:162] failed to register latency metric admission_quota_controller: duplicate metrics collector registration attempted
E0607 20:13:26.405917       1 prometheus.go:174] failed to register work_duration metric admission_quota_controller: duplicate metrics collector registration attempted
E0607 20:13:26.406035       1 prometheus.go:189] failed to register unfinished_work_seconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0607 20:13:26.406161       1 prometheus.go:202] failed to register longest_running_processor_microseconds metric admission_quota_controller: duplicate metrics collector registration attempted
E0607 20:13:26.417035       1 controller.go:148] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/127.0.0.1, ResourceVersion: 0, AdditionalErrorMsg:
time="2019-06-07T20:13:26.418457350Z" level=info msg="Running kube-scheduler --port=10251 --bind-address=127.0.0.1 --secure-port=0 --kubeconfig=/var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --leader-elect=false"
time="2019-06-07T20:13:26.420140177Z" level=info msg="Running kube-controller-manager --root-ca-file=/var/lib/rancher/k3s/server/tls/token-ca.crt --allocate-node-cidrs=true --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --cluster-cidr=10.42.0.0/16 --leader-elect=false --port=10252 --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key"
W0607 20:13:26.427442       1 authorization.go:47] Authorization is disabled
W0607 20:13:26.427560       1 authentication.go:55] Authentication is disabled
time="2019-06-07T20:13:26.600939505Z" level=info msg="Creating CRD listenerconfigs.k3s.cattle.io"
time="2019-06-07T20:13:26.631666904Z" level=info msg="Creating CRD addons.k3s.cattle.io"
time="2019-06-07T20:13:26.639995483Z" level=info msg="Creating CRD helmcharts.k3s.cattle.io"
time="2019-06-07T20:13:26.660327563Z" level=info msg="Waiting for CRD addons.k3s.cattle.io to become available"
E0607 20:13:26.689685       1 autoregister_controller.go:193] v1.k3s.cattle.io failed with : apiservices.apiregistration.k8s.io "v1.k3s.cattle.io" already exists
time="2019-06-07T20:13:27.164125768Z" level=info msg="Done waiting for CRD addons.k3s.cattle.io to become available"
time="2019-06-07T20:13:27.164232821Z" level=info msg="Waiting for CRD helmcharts.k3s.cattle.io to become available"
time="2019-06-07T20:13:27.665183681Z" level=info msg="Done waiting for CRD helmcharts.k3s.cattle.io to become available"time="2019-06-07T20:13:27.667772269Z" level=info msg="Listening on :6443"
time="2019-06-07T20:13:28.388514109Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/node-token"time="2019-06-07T20:13:28.388647322Z" level=info msg="To join node to cluster: k3s agent -s https://172.20.0.2:6443 -t ${NODE_TOKEN}"
time="2019-06-07T20:13:28.390770208Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-1.64.0.tgz"
time="2019-06-07T20:13:28.391086379Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
time="2019-06-07T20:13:28.391278538Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml"
time="2019-06-07T20:13:28.755609659Z" level=info msg="Wrote kubeconfig /output/kubeconfig.yaml"
time="2019-06-07T20:13:28.755864521Z" level=info msg="Run: k3s kubectl"
time="2019-06-07T20:13:28.756029303Z" level=info msg="k3s is up and running"
time="2019-06-07T20:13:29.090786444Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
time="2019-06-07T20:13:29.091144730Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
time="2019-06-07T20:13:29.093699530Z" level=info msg="Waiting for containerd startup: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or directory\""
W0607 20:13:29.842733       1 lease.go:222] Resetting endpoints for master service "kubernetes" to [127.0.0.1]
time="2019-06-07T20:13:30.096615342Z" level=warning msg="failed to start br_netfilter module"
time="2019-06-07T20:13:30.097721676Z" level=info msg="Connecting to wss://localhost:6443/v1-k3s/connect"
time="2019-06-07T20:13:30.097865253Z" level=info msg="Connecting to proxy" url="wss://localhost:6443/v1-k3s/connect"
time="2019-06-07T20:13:30.135367642Z" level=info msg="Handling backend connection request [k3d-k3s-default-server]"
time="2019-06-07T20:13:30.136626973Z" level=warning msg="Disabling CPU quotas due to missing cpu.cfs_period_us"
time="2019-06-07T20:13:30.136946207Z" level=info msg="Running kubelet --read-only-port=0 --root-dir=/var/lib/rancher/k3s/agent/kubelet --tls-private-key-file=/var/lib/rancher/k3s/agent/token-node.key --cpu-cfs-quota=false --cni-bin-dir=/bin --cluster-dns=10.43.0.10 --serialize-image-pulls=false --anonymous-auth=false --hostname-override=k3d-k3s-default-server --fail-swap-on=false --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --allow-privileged=true --cgroup-driver=cgroupfs --cert-dir=/var/lib/rancher/k3s/agent/kubelet/pki --healthz-bind-address=127.0.0.1 --resolv-conf=/tmp/k3s-resolv.conf --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --eviction-hard=imagefs.available<5%,nodefs.available<5% --authorization-mode=Webhook --address=0.0.0.0 --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.pem --tls-cert-file=/var/lib/rancher/k3s/agent/token-node.crt --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --authentication-token-webhook=true --cluster-domain=cluster.local --kubeconfig=/var/lib/rancher/k3s/agent/kubeconfig.yaml --seccomp-profile-root=/var/lib/rancher/k3s/agent/kubelet/seccomp --container-runtime=remote"
Flag --allow-privileged has been deprecated, will be removed in a future version
W0607 20:13:30.152766       1 info.go:52] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
W0607 20:13:30.153900       1 server.go:214] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
W0607 20:13:30.155448       1 proxier.go:480] Failed to read file /lib/modules/4.14.92-boot2docker/modules.builtin with error open /lib/modules/4.14.92-boot2docker/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0607 20:13:30.166009       1 proxier.go:493] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0607 20:13:30.166398       1 proxier.go:493] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0607 20:13:30.166675       1 proxier.go:493] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0607 20:13:30.166937       1 proxier.go:493] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0607 20:13:30.167247       1 proxier.go:493] Failed to load kernel module nf_conntrack_ipv4 with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0607 20:13:30.175682       1 probe.go:268] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
E0607 20:13:30.180964       1 cri_stats_provider.go:373] Failed to get the info of the filesystem with mountpoint "/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs": unable to find data in memory cache.
E0607 20:13:30.181277       1 kubelet.go:1250] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
E0607 20:13:30.220396       1 kubelet_network_linux.go:68] Failed to ensure rule to drop packet marked by KUBE-MARK-DROP in filter chain KUBE-FIREWALL: error appending rule: exit status 1: iptables: No chain/target/match by that name.
time="2019-06-07T20:13:30.227474523Z" level=info msg="waiting for node k3d-k3s-default-server: nodes \"k3d-k3s-default-server\" not found"
W0607 20:13:30.231456       1 node.go:113] Failed to retrieve node info: nodes "k3d-k3s-default-server" not found
W0607 20:13:30.231599       1 proxier.go:314] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
E0607 20:13:30.263932       1 conntrack.go:85] failed to set conntrack hashsize to 32768: write /sys/module/nf_conntrack/parameters/hashsize: operation not supported
W0607 20:13:30.277928       1 manager.go:537] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
E0607 20:13:30.304303       1 kubelet.go:2207] node "k3d-k3s-default-server" not found
E0607 20:13:30.312426       1 eviction_manager.go:247] eviction manager: failed to get summary stats: failed to get node info: node "k3d-k3s-default-server" not found
E0607 20:13:30.375850       1 proxier.go:696] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error appending rule: exit status 1: iptables: No chain/target/match by that name.
time="2019-06-07T20:13:30.381726431Z" level=info msg="Updated coredns node hosts entry [172.20.0.2 k3d-k3s-default-server]"
E0607 20:13:30.410034       1 kubelet.go:2207] node "k3d-k3s-default-server" not found
E0607 20:13:30.510831       1 kubelet.go:2207] node "k3d-k3s-default-server" not found
time="2019-06-07T20:13:32.238436249Z" level=info msg="waiting for node k3d-k3s-default-server CIDR not assigned yet"
time="2019-06-07T20:13:34.242393917Z" level=info msg="waiting for node k3d-k3s-default-server CIDR not assigned yet"
time="2019-06-07T20:13:36.244361095Z" level=info msg="waiting for node k3d-k3s-default-server CIDR not assigned yet"
time="2019-06-07T20:13:38.265227923Z" level=info msg="waiting for node k3d-k3s-default-server CIDR not assigned yet"
time="2019-06-07T20:13:40.267084842Z" level=info msg="waiting for node k3d-k3s-default-server CIDR not assigned yet"
E0607 20:13:40.315604       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:13:40.917358       1 resource_quota_controller.go:171] initial monitor sync has error: [couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=addons": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=addons", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=helmcharts": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=helmcharts", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=listenerconfigs": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=listenerconfigs"]
W0607 20:13:40.924184       1 controllermanager.go:445] Skipping "csrsigning"
W0607 20:13:41.324620       1 controllermanager.go:445] Skipping "root-ca-cert-publisher"
W0607 20:13:41.699480       1 actual_state_of_world.go:503] Failed to update statusUpdateNeeded field in actual state of world: Failed to set statusUpdateNeeded to needed true, because nodeName="k3d-k3s-default-server" does not exist
E0607 20:13:41.902939       1 proxier.go:696] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error appending rule: exit status 1: iptables: No chain/target/match by that name.
W0607 20:13:41.929625       1 node_lifecycle_controller.go:833] Missing timestamp for Node k3d-k3s-default-server. Assuming now as a timestamp.
E0607 20:13:42.560074       1 resource_quota_controller.go:437] failed to sync resource monitors: [couldn't start monitor for resource "k3s.cattle.io/v1, Resource=listenerconfigs": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=listenerconfigs", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=helmcharts": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=helmcharts", couldn't start monitor for resource "extensions/v1beta1, Resource=networkpolicies": unable to monitor quota for resource "extensions/v1beta1, Resource=networkpolicies", couldn't start monitor for resource "k3s.cattle.io/v1, Resource=addons": unable to monitor quota for resource "k3s.cattle.io/v1, Resource=addons"]
E0607 20:13:50.321145       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:13:56.852078       1 proxier.go:696] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error appending rule: exit status 1: iptables: No chain/target/match by that name.
E0607 20:14:00.332924       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:14:10.346264       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:14:20.358508       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:14:26.857162       1 proxier.go:696] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error appending rule: exit status 1: iptables: No chain/target/match by that name.
E0607 20:14:30.227840       1 kubelet_network_linux.go:68] Failed to ensure rule to drop packet marked by KUBE-MARK-DROP in filter chain KUBE-FIREWALL: error appending rule: exit status 1: iptables: No chain/target/match by that name.
E0607 20:14:30.428978       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:14:40.447210       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:14:50.474718       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:14:56.878386       1 proxier.go:696] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error appending rule: exit status 1: iptables: No chain/target/match by that name.
E0607 20:15:00.530070       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:15:10.546225       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:15:20.565363       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:15:26.884090       1 proxier.go:696] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error appending rule: exit status 1: iptables: No chain/target/match by that name.
E0607 20:15:30.235345       1 kubelet_network_linux.go:68] Failed to ensure rule to drop packet marked by KUBE-MARK-DROP in filter chain KUBE-FIREWALL: error appending rule: exit status 1: iptables: No chain/target/match by that name.
E0607 20:15:30.579147       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:15:40.596127       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:15:50.604860       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:15:56.889318       1 proxier.go:696] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error appending rule: exit status 1: iptables: No chain/target/match by that name.
E0607 20:16:00.615836       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:16:10.653171       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:16:20.678713       1 summary_sys_containers.go:47] Failed to get system container stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get cgroup stats for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": failed to get container info for "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy": unknown container "/docker/8f9b573153acbd3effe58b08e4511328c1e8a8ac96efd9ffcbb04cdb7c46111f/kube-proxy"
E0607 20:16:26.896603       1 proxier.go:696] Failed to ensure that filter chain INPUT jumps to KUBE-EXTERNAL-SERVICES: error appending rule: exit status 1: iptables: No chain/target/match by that name.