Minikube: Cannot pass extra config parameters to kube-apiserver

[cvila84]

Using --extra-config does not work anymore for kube-apiserver

Minikube version: 1.7.1
Kubernetes version: 1.17.2
The operating system version: Win7

The exact command to reproduce the issue:
minikube start --extra-config=apiserver.runtime-config=apps/v1beta1=true,apps/v1beta2=true,extensions/v1beta1/daemonsets=true,extensions/v1beta1/deployments=true,extensions/v1beta1/replicasets=true,extensions/v1beta1/networkpolicies=true,extensions/v1beta1/podsecuritypolicies=true

The full output of the command that failed:

any helm installation with deprecated API (that should be activated thanks to extra-config) gives :
no kind statefulset is registered for version apps/v1beta1

The output of the minikube logs command:

No logs to give as the error is during helm installation (after minikube start which was successful)

The kube-apiserver extra configuration is not passed anymore as we can see below (there is no --runtime-config flag)

$ kubectl -n kube-system describe pod kube-apiserver-minikube
Name:                 kube-apiserver-minikube
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 minikube/192.168.99.100
Start Time:           Fri, 07 Feb 2020 09:06:17 +0100
Labels:               component=kube-apiserver
                      tier=control-plane
Annotations:          kubernetes.io/config.hash: 35413dc13b2b07959960ac3335a08c70
                      kubernetes.io/config.mirror: 35413dc13b2b07959960ac3335a08c70
                      kubernetes.io/config.seen: 2020-02-06T17:31:36.081791655Z
                      kubernetes.io/config.source: file
Status:               Running
IP:                   192.168.99.100
IPs:
  IP:           192.168.99.100
Controlled By:  Node/minikube
Containers:
  kube-apiserver:
    Container ID:  docker://08b0b0e453d676d6bc449cdb4deac91fe45eb289e82e993771c84f57eb622baf
    Image:         k8s.gcr.io/kube-apiserver:v1.17.2
    Image ID:      docker-pullable://k8s.gcr.io/kube-apiserver@sha256:b22f7be5165a0022d282815067bda22f0282922f5ee65151e64cf3b54be09543
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-apiserver
      --advertise-address=192.168.99.100
      --allow-privileged=true
      --authorization-mode=Node,RBAC
      --client-ca-file=/var/lib/minikube/certs/ca.crt
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/var/lib/minikube/certs/etcd/ca.crt
      --etcd-certfile=/var/lib/minikube/certs/apiserver-etcd-client.crt
      --etcd-keyfile=/var/lib/minikube/certs/apiserver-etcd-client.key
      --etcd-servers=https://127.0.0.1:2379
      --insecure-port=0
      --kubelet-client-certificate=/var/lib/minikube/certs/apiserver-kubelet-client.crt
      --kubelet-client-key=/var/lib/minikube/certs/apiserver-kubelet-client.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --proxy-client-cert-file=/var/lib/minikube/certs/front-proxy-client.crt
      --proxy-client-key-file=/var/lib/minikube/certs/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --secure-port=8443
      --service-account-key-file=/var/lib/minikube/certs/sa.pub
      --service-cluster-ip-range=10.96.0.0/12
      --tls-cert-file=/var/lib/minikube/certs/apiserver.crt
      --tls-private-key-file=/var/lib/minikube/certs/apiserver.key
    State:          Running
      Started:      Fri, 07 Feb 2020 09:06:18 +0100
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 06 Feb 2020 18:31:28 +0100
      Finished:     Thu, 06 Feb 2020 18:35:04 +0100
    Ready:          True
    Restart Count:  1
    Requests:
      cpu:        250m
    Liveness:     http-get https://192.168.99.100:8443/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:  <none>
    Mounts:
      /etc/ssl/certs from ca-certs (ro)
      /usr/share/ca-certificates from usr-share-ca-certificates (ro)
      /var/lib/minikube/certs from k8s-certs (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  ca-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:  DirectoryOrCreate
  k8s-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/minikube/certs
    HostPathType:  DirectoryOrCreate
  usr-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/share/ca-certificates
    HostPathType:  DirectoryOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Normal   SandboxChanged    7m32s                 kubelet, minikube  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled            7m32s                 kubelet, minikube  Container image "k8s.gcr.io/kube-apiserver:v1.17.2" already present on machine
  Normal   Created           7m32s                 kubelet, minikube  Created container kube-apiserver
  Normal   Started           7m32s                 kubelet, minikube  Started container kube-apiserver
  Warning  DNSConfigForming  49s (x10 over 7m32s)  kubelet, minikube  Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 10.42.178.160 10.105.152.15 10.150.22.18

Possible cause is the template used to generate the kubeadm config which when --extra-config for kube-apiserver is used will generate the apiServer key twice

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
  extraArgs:
    enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
    runtime-config: "apps/v1beta1=true,apps/v1beta2=true,extensions/v1beta1/daemonsets=true,extensions/v1beta1/deployments=true,extensions/v1beta1/replicasets=true,extensions/v1beta1/networkpolicies=true,extensions/v1beta1/podsecuritypolicies=true"
certificatesDir: /var/lib/minikube/certs
clusterName: kubernetes
apiServer:
  certSANs: ["127.0.0.1", "localhost", "192.168.99.100"]
controlPlaneEndpoint: localhost:8443
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/minikube/etcd
kubernetesVersion: v1.17.2
networking:
  dnsDomain: cluster.local
  podSubnet: ""
  serviceSubnet: 10.96.0.0/12

It was working with minikube 1.16 as the generated key for certSANs was not under apiServer key before (and we had only one apiServer key with the extra config)

[mbarzilovich]

extra-config for apiserver works if you set kubernetes-version=v1.13.10
I.e
minikube start --kubernetes-version=v1.13.10 --extra-config=apiserver.service-node-port-range=1-65535
kubernetes v1.14 and higher dont respect extra-config

[cvila84]

It makes sense as starting from 1.14.0, the kubeadm v1beta1 template is used, containing a default setting for apiServer

apiServer:
  certSANs: ["127.0.0.1", "localhost", "{{.AdvertiseAddress}}"]

which is not merged with the ones coming from extra-config

Before 1.14.0, the template has a different key

apiServerCertSANs: ["127.0.0.1", "localhost", "{{.AdvertiseAddress}}"]

which is this time not conflicting with extra-config values generated under the apiServer key

[tstromberg]

Good catch! Thank you for the detailed bug report. It seems there is definitely missing coverage in our integration tests (and unit tests!) for this feature.

This seems entirely fixable. Help wanted!

[tstromberg]

This looks like an easy fix thanks to the detailed feedback. I'll try to send a PR out for it today.

[tstromberg]

Does someone mind testing whether or not #6547 fixes their use case? Here are pre-built binaries:

https://storage.googleapis.com/minikube-builds/6547/minikube-darwin-amd64
https://storage.googleapis.com/minikube-builds/6547/minikube-linux-amd64
https://storage.googleapis.com/minikube-builds/6547/minikube-windows-amd64.exe

[ggreer]

The linux binary works for me. Running minikube start --extra-config=apiserver.service-node-port-range=80-30000 now sets the right port range.

If I apply a manifest that adds cassandra nodeport service, it succeeds with the binary from build 6547. If I do the same thing with minikube v1.7.1, it always fails with The Service "cassandra" is invalid: spec.ports[0].nodePort: Invalid value: 9042: provided port is not in the valid range. The range of valid ports is 30000-32767.

[tstromberg]

@ggreer - Thank you for the confirmation. We'll ship v1.7.2 today to fix this regression.