Gardener: Protobuf generation is missing

[rfranzke]

What happened:
Using Gardener on a 1.16 Kubernetes cluster does not work properly because the kube-controller-manager only uses protobuf in 1.16, and the Gardener does not yet support this content type.

What you expected to happen:
Gardener should provide protobuf definitons.

How to reproduce it (as minimally and precisely as possible):
Deploy Gardener on 1.16 cluster and try to delete a namespace. See error logs in the kube-controller-manager like:

E1111 09:24:51.733918       1 namespace_controller.go:148] deletion of namespace extension-provider-openstack-7ld84 failed: [object *v1beta1.ShootList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message, object *v1alpha1.ShootList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message, object *v1beta1.SecretBindingList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message, object *v1alpha1.OpenIDConnectPresetList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message, object *v1beta1.QuotaList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message, object *v1alpha1.SecretBindingList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message, object *v1alpha1.PlantList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message, object *v1alpha1.QuotaList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message, object *v1alpha1.BackupEntryList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message]

Environment:

• Gardener version: 0.31.3
• Kubernetes version (use kubectl version): 1.16.2
• Cloud provider or hardware configuration: AWS

[vlerenc]

@mvladev Didn't you want to build that right away? Can someone else help?

[ialidzhikov]

@vlerenc, I will check today.

[ialidzhikov]

Ok, we reproduced this with @vpnachev . In my local setup (minikube v1.16)

k create ns foo
k delete ns foo

The ns status

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2019-12-10T14:02:34Z"
  deletionTimestamp: "2019-12-10T14:02:55Z"
  name: bar
  resourceVersion: "5538800"
  selfLink: /api/v1/namespaces/bar
  uid: 235ebe73-a788-41bb-8e57-f6904ef8f4c6
spec:
  finalizers:
  - kubernetes
status:
  conditions:
  - lastTransitionTime: "2019-12-10T14:03:00Z"
    message: All resources successfully discovered
    reason: ResourcesDiscovered
    status: "False"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2019-12-10T14:03:00Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2019-12-10T14:03:00Z"
    message: 'Failed to delete all resource types, 10 remaining: object *v1alpha1.OpenIDConnectPresetList
      does not implement the protobuf marshalling interface and cannot be encoded
      to a protobuf message, object *v1alpha1.ShootStateList does not implement the
      protobuf marshalling interface and cannot be encoded to a protobuf message,
      object *v1beta1.BackupEntryList does not implement the protobuf marshalling
      interface and cannot be encoded to a protobuf message, object *v1beta1.PlantList
      does not implement the protobuf marshalling interface and cannot be encoded
      to a protobuf message, object *v1beta1.QuotaList does not implement the protobuf
      marshalling interface and cannot be encoded to a protobuf message, object *v1beta1.QuotaList
      does not implement the protobuf marshalling interface and cannot be encoded
      to a protobuf message, object *v1beta1.SecretBindingList does not implement
      the protobuf marshalling interface and cannot be encoded to a protobuf message,
      object *v1beta1.SecretBindingList does not implement the protobuf marshalling
      interface and cannot be encoded to a protobuf message, object *v1beta1.ShootList
      does not implement the protobuf marshalling interface and cannot be encoded
      to a protobuf message, object *v1beta1.ShootList does not implement the protobuf
      marshalling interface and cannot be encoded to a protobuf message'
    reason: ContentDeletionFailed
    status: "True"
    type: NamespaceDeletionContentFailure
  phase: Terminating

garden-apiserver logs:

E1210 16:09:11.045248   79338 writers.go:172] apiserver was unable to write a JSON response: object *v1beta1.PlantList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1210 16:09:11.065287   79338 writers.go:172] apiserver was unable to write a JSON response: object *v1alpha1.OpenIDConnectPresetList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1210 16:09:11.116720   79338 writers.go:172] apiserver was unable to write a JSON response: object *v1beta1.SecretBindingList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1210 16:09:11.140543   79338 writers.go:172] apiserver was unable to write a JSON response: object *v1beta1.SecretBindingList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1210 16:09:11.173161   79338 writers.go:172] apiserver was unable to write a JSON response: object *v1beta1.ShootList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1210 16:09:11.178005   79338 writers.go:172] apiserver was unable to write a JSON response: object *v1beta1.QuotaList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1210 16:09:11.218308   79338 writers.go:172] apiserver was unable to write a JSON response: object *v1beta1.BackupEntryList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1210 16:09:11.253629   79338 writers.go:172] apiserver was unable to write a JSON response: object *v1beta1.QuotaList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1210 16:09:11.266425   79338 writers.go:172] apiserver was unable to write a JSON response: object *v1beta1.ShootList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E1210 16:09:11.271274   79338 writers.go:172] apiserver was unable to write a JSON response: object *v1alpha1.ShootStateList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message

[rfranzke]

cc @danielfoehrKn

[ialidzhikov]

Ok, I managed to reproduce the same with github.com/kubernetes/sample-apiserver on kubernetes-1.14 branch on minikube v 1.16.3.

$ k get ns foo -o yaml
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2019-12-10T16:56:07Z"
  deletionTimestamp: "2019-12-10T16:56:12Z"
  name: foo
  resourceVersion: "5464"
  selfLink: /api/v1/namespaces/foo
  uid: ea9042a8-33c8-430d-bacf-3f194cd46814
spec:
  finalizers:
  - kubernetes
status:
  conditions:
  - lastTransitionTime: "2019-12-10T16:56:17Z"
    message: All resources successfully discovered
    reason: ResourcesDiscovered
    status: "False"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2019-12-10T16:56:17Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2019-12-10T16:56:17Z"
    message: 'Failed to delete all resource types, 1 remaining: object *v1alpha1.FlunderList
      does not implement the protobuf marshalling interface and cannot be encoded
      to a protobuf message'
    reason: ContentDeletionFailed
    status: "True"
    type: NamespaceDeletionContentFailure
  phase: Terminating

With github.com/kubernetes/sample-apiserver on kubernetes-1.16 branch (which also vendors kubernetes-1.16) the namespaces can be successfully deleted.
We already started adopting kubernetes-1.16, so I think this issue will be resolved once we adopt kubernetes-1.16 for g/g.

Depends on (will be fixed with) #1206 .

[ialidzhikov]

/assign @ialidzhikov

[ialidzhikov]

Ahh, sorry, I was wrong about:

With github.com/kubernetes/sample-apiserver on kubernetes-1.16 branch (which also vendors kubernetes-1.16) the namespaces can be successfully deleted.

I now reproduce the namespace deletion issue with [email protected] on kubernetes v1.16 and [email protected] on kubernetes v1.17. Probably I missed to create the apiservice when I did the initial test. :|

I will continue the investigation.

[ialidzhikov]

Ref kubernetes/kubernetes#86666.

[rfranzke]

@ialidzhikov what's the status? Will you add the generation to g/g?

[ialidzhikov]

Using Gardener on a 1.16 Kubernetes cluster does not work properly because the kube-controller-manager only uses protobuf in 1.16, and the Gardener does not yet support this content type.

This statement is not true IMO. I think there is no (and should not be) hard requirement for protobuf from kubernetes side. I think it is a bug in k/k and I described my findings in kubernetes/kubernetes#86666. The apiserver currently cannot fallback to the second ContentType in the Accept header. I also think that not all kubernetes built-in types have protobuf generation.
So the status is that I am waiting for kubernetes/kubernetes#86666 to see what will be the reply.

[rfranzke]

OK 👍 thank you for the update. Do you think it makes still sense to generate the protobuf on our side anyways?

[ialidzhikov]

OK 👍 thank you for the update. Do you think it makes still sense to generate the protobuf on our side anyways?

Looks like it makes sense.

However, clients may request the more efficient Protobuf representation of these objects for better performance at scale

https://kubernetes.io/docs/reference/using-api/api-concepts/#alternate-representations-of-resources

Will check.

[ialidzhikov]

Currently the code-generation related to protobufs generates a Size() func to each type. We also have types such as Volume, MachineTypeStorage which have also field named Size. And the compilation fails in a ugly way with

pkg/apis/core/v1alpha1/generated.pb.go:11243:6: type MachineTypeStorage has both field and method named Size
pkg/apis/core/v1alpha1/generated.pb.go:12189:6: type Volume has both field and method named Size

I described my findings in kubernetes/kubernetes#87336.

[ialidzhikov]

To workaround the issue described above, we decided to rename the Size fields to something else but preserving the json tag (I hope this would be backwards compatible from API side). Currently I am facing issues with int/*int fields handling as go-to-protobuf tries to assign int64 to int (I described my findings in https://github.com/kubernetes/kubernetes/issues/89492). I need to dig more about it.
Generally api-conventions guide has the following section:

All public integer fields MUST use the Go (u)int32 or Go (u)int64 types, not (u)int (which is ambiguous depending on target platform). Internal types may use (u)int.

Ref https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md

I am wondering whether the int -> int32 change of public field is a compatible?

[rfranzke]

Thanks Ismail. I think it’s safe to go from int->int32