Flannel: Unable to install flannel on kubernetes 1.14.0 with error: error: SchemaError(io.k8s.api.autoscaling.v2beta2.MetricValueStatus): invalid object doesn't have additional properties

Created on 26 Mar 2019  路  7Comments  路  Source: coreos/flannel

Unable to install flannel on kubernetes 1.14.0

Expected Behavior

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel.yaml

Should setup flannel

Current Behavior

Failes with the following output

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel.yaml
error: SchemaError(io.k8s.api.autoscaling.v2beta2.MetricValueStatus): invalid object doesn't have additional properties

Steps to Reproduce (for bugs)

  1. Install k8s
yum install -y kubectl kubelet kubeadm
  1. Init control plane:
    kubeadm init --pod-network-cidr=10.244.0.0/16
    2. 3. 
kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:32Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
  1. Attempt to install flannel
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel.yaml

Context

I am setting up a new k8s cluster but my pods are lonely.

Your Environment

  • Flannel version:
    latest master
  • Kubernetes version (if used):
kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:32Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
  • Operating System and version:
CentOS Linux release 7.6.1810 (Core)
Derived from Red Hat Enterprise Linux 7.6 (Source)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

cat: /etc/prelink.conf.d: Is a directory
CentOS Linux release 7.6.1810 (Core)
CentOS Linux release 7.6.1810 (Core)
cpe:/o:centos:centos:7
picture jspenc72

Most helpful comment

It may be that client kubectl (1.10.11) differs from server kubectl (1.14.0) thus producing validation errors.
--validate=false flag might be helpful while applying kube-flannel.yaml
Just try this way kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel.yaml --validate=false

picture stumanov on 28 Mar 2019
👍12

All 7 comments

It may be that client kubectl (1.10.11) differs from server kubectl (1.14.0) thus producing validation errors.
--validate=false flag might be helpful while applying kube-flannel.yaml
Just try this way kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel.yaml --validate=false

stumanov picture stumanov on 28 Mar 2019
👍12

@stumanov I tried the same thing from the kubernetes master so the server and client versions matched and got the same issue. Ill retry with --validate=false

jspenc72 picture jspenc72 on 28 Mar 2019
👍2

@jspenc72 your init command should be 

kubeadm init --pod-network-cidr=10.244.0.0/16
discostur picture discostur on 2 Apr 2019
👍2

@stumanov I tried the same thing from the kubernetes master so the server and client versions matched and got the same issue. Ill retry with --validate=false

This omits the error but the overall status of the node is still NotReady with the output of sudo kubectl get pods --all-namespaces:

NAMESPACE     NAME                                 READY     STATUS    RESTARTS   AGE
kube-system   coredns-fb8b8dccf-5c4l7              0/1       Pending   0          1h
kube-system   coredns-fb8b8dccf-tvr49              0/1       Pending   0          1h
kube-system   etcd-k8s-master                      1/1       Running   0          59m
kube-system   kube-apiserver-k8s-master            1/1       Running   0          59m
kube-system   kube-controller-manager-k8s-master   1/1       Running   0          59m
kube-system   kube-flannel-ds-amd64-4bwmk          0/1       Evicted   0          3m
kube-system   kube-proxy-vnd8q                     0/1       Evicted   0          14m
kube-system   kube-scheduler-k8s-master            1/1       Running   0          59m
seenitall picture seenitall on 8 Apr 2019
👍3

@seenitall I run into the same issue, kube-flannel and kube-proxy being evicted. I fix it by setting systemd as Cgroup driver of docker and Kubernetes

and executing the following as root:

modprobe overlay
modprobe br_netfilter

Setup required sysctl params, these persist across reboots.

cat > /etc/sysctl.d/99-kubernetes-cri.conf < net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system

you can check this link for it:
https://kubernetes.io/docs/setup/cri/
https://blog.heptio.com/properly-resetting-your-kubeadm-bootstrapped-cluster-nodes-heptioprotip-473bd0b824aa

I then perform,

1.) kubeadm reset (as root)
2.) copy the config file to ~/.kube
3.) kubeadm init --pod-network-cidr=10.244.0.0/16 (as root)
4.) kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Host OS : Ubuntu 18.04
Kubeadmn and Kubectl client/server version: 1.14.1
Docker client/server version: 18.09.5

nightcoder824 picture nightcoder824 on 2 May 2019
👍2

@stumanov I tried the same thing from the kubernetes master so the server and client versions matched and got the same issue. Ill retry with --validate=false

This omits the error but the overall status of the node is still NotReady with the output of sudo kubectl get pods --all-namespaces:

NAMESPACE     NAME                                 READY     STATUS    RESTARTS   AGE
kube-system   coredns-fb8b8dccf-5c4l7              0/1       Pending   0          1h
kube-system   coredns-fb8b8dccf-tvr49              0/1       Pending   0          1h
kube-system   etcd-k8s-master                      1/1       Running   0          59m
kube-system   kube-apiserver-k8s-master            1/1       Running   0          59m
kube-system   kube-controller-manager-k8s-master   1/1       Running   0          59m
kube-system   kube-flannel-ds-amd64-4bwmk          0/1       Evicted   0          3m
kube-system   kube-proxy-vnd8q                     0/1       Evicted   0          14m
kube-system   kube-scheduler-k8s-master            1/1       Running   0          59m

I have the same error on Ubuntu 16.04 with the following output from kubectl describe node

Name:               skyuuka
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=skyuuka
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/master=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 17 Sep 2019 00:06:18 -0700
Taints:             node-role.kubernetes.io/master:NoSchedule
                    node.kubernetes.io/disk-pressure:NoSchedule
                    node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 17 Sep 2019 00:13:19 -0700   Tue, 17 Sep 2019 00:06:14 -0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     True    Tue, 17 Sep 2019 00:13:19 -0700   Tue, 17 Sep 2019 00:06:29 -0700   KubeletHasDiskPressure       kubelet has disk pressure
  PIDPressure      False   Tue, 17 Sep 2019 00:13:19 -0700   Tue, 17 Sep 2019 00:06:14 -0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Tue, 17 Sep 2019 00:13:19 -0700   Tue, 17 Sep 2019 00:06:14 -0700   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
  InternalIP:  10.145.83.31
  Hostname:    skyuuka
Capacity:
 cpu:                44
 ephemeral-storage:  3026930480Ki
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             1585242464Ki
 pods:               110
Allocatable:
 cpu:                44
 ephemeral-storage:  2789619125750
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             1585140064Ki
 pods:               110
System Info:
 Machine ID:                 3087eb2f22d568d6800ce4c957a12a75
 System UUID:                4C4C4544-004C-3310-804D-B1C04F4B4232
 Boot ID:                    c090d61b-0ab4-4e20-a15d-897cfd086401
 Kernel Version:             4.4.0-47-generic
 OS Image:                   Ubuntu 16.04.6 LTS
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://18.9.7
 Kubelet Version:            v1.15.3
 Kube-Proxy Version:         v1.15.3
Non-terminated Pods:         (4 in total)
  Namespace                  Name                                   CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                                   ------------  ----------  ---------------  -------------  ---
  kube-system                etcd-skyuuka                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         6m36s
  kube-system                kube-apiserver-skyuuka             250m (0%)     0 (0%)      0 (0%)           0 (0%)         6m34s
  kube-system                kube-controller-manager-skyuuka    200m (0%)     0 (0%)      0 (0%)           0 (0%)         6m52s
  kube-system                kube-scheduler-skyuuka             100m (0%)     0 (0%)      0 (0%)           0 (0%)         6m40s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                550m (1%)  0 (0%)
  memory             0 (0%)     0 (0%)
  ephemeral-storage  0 (0%)     0 (0%)
Events:
  Type     Reason                   Age                    From                     Message
  ----     ------                   ----                   ----                     -------
  Normal   NodeHasSufficientMemory  7m59s (x8 over 7m59s)  kubelet, skyuuka     Node skyuuka status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    7m59s (x8 over 7m59s)  kubelet, skyuuka     Node skyuuka status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     7m59s (x7 over 7m59s)  kubelet, skyuuka     Node skyuuka status is now: NodeHasSufficientPID
  Normal   Starting                 7m39s                  kube-proxy, skyuuka  Starting kube-proxy.
  Normal   Starting                 7m37s                  kube-proxy, skyuuka  Starting kube-proxy.
  Normal   Starting                 7m6s                   kube-proxy, skyuuka  Starting kube-proxy.
  Normal   Starting                 6m36s                  kube-proxy, skyuuka  Starting kube-proxy.
  Normal   Starting                 6m5s                   kube-proxy, skyuuka  Starting kube-proxy.
  Normal   Starting                 5m35s                  kube-proxy, skyuuka  Starting kube-proxy.
  Normal   Starting                 5m5s                   kube-proxy, skyuuka  Starting kube-proxy.
  Normal   Starting                 4m32s                  kube-proxy, skyuuka  Starting kube-proxy.
  Normal   Starting                 3m5s                   kube-proxy, skyuuka  Starting kube-proxy.
  Warning  ImageGCFailed            2m59s                  kubelet, skyuuka     failed to garbage collect required amount of images. Wanted to free 269566173184 bytes, but freed 0 bytes
  Normal   Starting                 17s                    kube-proxy, skyuuka  Starting kube-proxy.
skyuuka picture skyuuka on 17 Sep 2019

@seenitall I run into the same issue, kube-flannel and kube-proxy being evicted. I fix it by setting systemd as Cgroup driver of docker and Kubernetes

and executing the following as root:

modprobe overlay
modprobe br_netfilter

Setup required sysctl params, these persist across reboots.

cat > /etc/sysctl.d/99-kubernetes-cri.conf < net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system

you can check this link for it:
https://kubernetes.io/docs/setup/cri/
https://blog.heptio.com/properly-resetting-your-kubeadm-bootstrapped-cluster-nodes-heptioprotip-473bd0b824aa

I then perform,

1.) kubeadm reset (as root)
2.) copy the config file to ~/.kube
3.) kubeadm init --pod-network-cidr=10.244.0.0/16 (as root)
4.) kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Host OS : Ubuntu 18.04
Kubeadmn and Kubectl client/server version: 1.14.1
Docker client/server version: 18.09.5
@nightcoder824
what is mean"copy the config file to ~/.kube"?
I run into the same issue,

Chen-YiDA picture Chen-YiDA on 10 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Dieken picture Dieken  路  6Comments
ozbillwang picture ozbillwang  路  3Comments
pquerna picture pquerna  路  3Comments
lukaszpy picture lukaszpy  路  5Comments
smarkm picture smarkm  路  4Comments