Kind: Documentation: how to enable alpha features

[mitar]

I am trying to use PodPreset from settings.k8s.io/v1alpha1 but it seems that kube through kind does not support that. This is probably just a question of enabling those experimental features, but it is unclear how does one do that?

[BenTheElder]

this is not currently possible _unless_ you override the kubeadm config template, which is probably a bad idea and will be broken soon, still discussing with @munnerz strategies for better exposing access to the kubeadm config.

We can add specific knobs for things like this, which may limit what can be configured but also explicitly allows us to support them including across kubeadm versions (NOTE: the kubeadm config itself is superficially unstable / alpha versioned so we do have trivial breakages between versions, but with this route we can handle them...)

OR we can do something like allow the user to supply an overlay kustomize style to the kubeadm config, but then we're exposing what is in turn another alpha API...

Currently leaning towards adding our own knobs on top so we can allow this to work with the same kind config for different kubernetes versions.

[mitar]

I ended up simply doing this:

docker exec -i kind-1-control-plane patch --directory=/etc/kubernetes/manifests <<EOF
--- kube-apiserver.yaml.orig    2018-10-10 12:58:56.739231422 -0700
+++ kube-apiserver.yaml 2018-10-10 13:12:41.832606463 -0700
@@ -18,7 +18,7 @@
     - --allow-privileged=true
     - --client-ca-file=/etc/kubernetes/pki/ca.crt
     - --disable-admission-plugins=PersistentVolumeLabel
-    - --enable-admission-plugins=NodeRestriction
+    - --enable-admission-plugins=NodeRestriction,PodPreset,NamespaceLifecycle,NamespaceExists,ResourceQuota
     - --enable-bootstrap-token-auth=true
     - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
     - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
@@ -40,6 +40,7 @@
     - --service-cluster-ip-range=10.96.0.0/12
     - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
     - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
+    - --runtime-config=settings.k8s.io/v1alpha1=true
     image: k8s.gcr.io/kube-apiserver-amd64:v1.11.3
     imagePullPolicy: IfNotPresent
     livenessProbe:
EOF

So I just patch the config and it detects the change and auto-reloads.

[neolit123]

all kubeadm wrappers are facing similar issues.
one way of allowing some flexibility from kind would be to instead of a template also allow the user to pass the kubeadm config yaml directly.

from there it would be up to the user to match the kubeadm version and the contents of the config.

[BenTheElder]

We can do that, but then kind won't work unless they're careful with their yaml. I have yet another fix in the works that requires specifying things in the kubeadm yaml, user provided config will be missing necessary fields.

[neolit123]

you can also consider exposing the direct config, while adding a command to dump the suggested kubeadm config for a certain k8s version from kind.
we have kubeadm config print-default for that, which is ironically also changing in 1.13:
https://github.com/kubernetes/kubernetes/pull/69617

[BenTheElder]

So we're adding support for patching the config (#77), but we should probably add some kubeadm-api-version independent knobs for a few highly desirable config features like this one.

[BenTheElder]

/assign
/priority important-longterm
/kind feature

[BenTheElder]

this should also go in #70

[mitar]

Has there been any change to this since I last looked at this? (I am doing one update on our codebase using kind and I wonder if I can remove changes in my fork.)

[BenTheElder]

a kubeadm config patch in the kind config + https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1#ClusterConfiguration featureGates should cover this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[mitar]

/remove-lifecycle stale

[aojea]

@mitar does this solve this issue https://github.com/kubernetes-sigs/kind/pull/572 ?

[BenTheElder]

I meant to close this when https://github.com/kubernetes-sigs/kind/pull/572 merged.
/close

[k8s-ci-robot]

@BenTheElder: Closing this issue.

In response to this:

I meant to close this when https://github.com/kubernetes-sigs/kind/pull/572 merged.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mitar]

Awesome!