Minikube: Kubectl 1.8 broken for minikube - OpenAPI swagger file doesn't include `x-kubernetes-group-version-kind`

[apelisse]

Environment:

Minikube version (use minikube version): v0.22.2

• OS (e.g. from /etc/os-release): Linux
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): VirtualBox
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): v0.23.4
• Install tools:
• Others:

What happened:
Swagger doesn't have x-kubernetes-group-version-kind while it should.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

minikube start
kubectl proxy &
curl 127.0.0.1:8001/swagger.json | less
# Look for v1.Service definition, it doesn't have `x-kubernetes-group-version-kind` field

Output of minikube logs (if applicable):

Anything else do we need to know:

@pwittrock @mbohlool

[r2d4]

localkube-swagger.txt
kubeadm-swagger.txt

@apelisse Can you upload the full swagger.json file or verify that the above outputs look similar to yours?

Here are the outputs from minikube start --bootstrapper localkube and minikube start --bootstrapper kubeadm. I did receive an error when I tried to use kubectl version 1.8.0-rc.1 on the 1.7.5 localkube cluster, but not on the kubeadm based cluster.

The localkube swagger.json does seem to include the x-kubernetes-group-version-kind headers, but there are some differences which I'm working through identifying.

[mbohlool]

@r2d4 Before we go into debuging this (and I would like to help debugging it), can you confirm if we found a fix here, we can patch minikube 1.7 images with it?

[r2d4]

@mbohlool Well, minikube ships with a default version of a localkube, which we won't be able to patch. We would be able to patch non-default versions, since those are fetched from GCS.

The latest version includes a --bootstrapper kubeadm flag which would provide a workaround without relying on localkube, but it isn't the default option.

[apelisse]

localkube-swagger.txt

This one is not good

kubeadm-swagger.txt

This one is fine, as far as I can tell

[r2d4]

@apelisse I've verified that, but what exactly is missing? The headers are there, but I'm not sure which ones are missing - they are present for things like service

     "x-kubernetes-action": "list",
     "x-kubernetes-group-version-kind": {
      "group": "",
      "version": "v1",
      "kind": "Service"
     }

[apelisse]

Yeah, you're looking at the "paths", not the "definitions" section

[apelisse]

IOW,

With kubeadm:

$ jq '.definitions."io.k8s.kubernetes.pkg.api.v1.Pod"."x-kubernetes-group-version-kind"' kubeadm-swagger.txt
[
 {
   "group": "",
   "version": "v1",
   "kind": "Pod"
 }
]

With localkube:

$ jq '.definitions."io.k8s.kubernetes.pkg.api.v1.Pod"."x-kubernetes-group-version-kind"' localkube-swagger.txt
null

[apelisse]

We had a similar type of issue in apiserver-builder, where x-kubernetes-group-version-kind was missing: https://github.com/kubernetes-incubator/apiserver-builder/pull/149

[mbohlool]

Any update on this? Let me know if you guys need any help.

[mbohlool]

Found the problem after a long debugging :) Short Answer: This is the fix: https://github.com/kubernetes/kubernetes/pull/53152 and you need to update your vendor folder after this gets into 1.7
You should not have any problem with 1.8 as this fix is already there.

Longer Answer: We do use full package name as the name of the OpenAPI definitions. The OpenAPI generator take the vendoring folder into account but (in 1.7 code) the extension in staging/src/k8s.io/apiserver/pkg/endpoints/openapi/openapi.go did not. We were OK in kubernetes at the time, but when we moved packages out and vendor some part of APIs in, that became a problem and we fixed it in 1.8. All you need to do is to backport this fix (that seems safe and should not affect kubernetes itself at all) to 1.7 branch and update your vendor folder.

[apelisse]

It didn't take time for someone to notice: https://github.com/kubernetes/kubectl/issues/67.
is there a new release of minikube? What are people supposed to do if they hit that bug? Thanks

[r2d4]

We're working on cherry-picking the fix and backporting it to our localkube releases. We'll make sure that this is fixed in the next version, but for now the only solution is to not upgrade your kubectl to version 1.8.

[apelisse]

for now the only solution is to not upgrade your kubectl to version 1.8.

• Or use --validate=false,
• Or use --openapi-validation=false.

Thank you Matt

[r2d4]

I've backported the fix to previous versions of localkube, and it should be fixed for the default version in the next release

[sauravomar]

Is it resolved in the latest release?

[r2d4]

Yes @sauravomar