Cloud-on-k8s: OpenAPIV3 structural schema violations

[sebgl]

Some violations appear when describing the CRD with kubectl describe crd elasticsearch:

Status:  Accepted Names:
    Categories:      elastic
    Kind:       Elasticsearch    List Kind:  ElasticsearchList
    Plural:     elasticsearches    Short Names:
      es    Singular:  elasticsearch
  Conditions:    Last Transition Time:  2020-01-06T10:47:54Z
    Message:               no conflicts found    Reason:                NoConflicts
    Status:                True    Type:                  NamesAccepted
    Last Transition Time:  <nil>    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted    Status:                True
    Type:                  Established    Last Transition Time:  2020-01-06T10:47:54Z
    Message:               [spec.validation.openAPIV3Schema.properties[spec].properties[http].properties[service].properties[spec].properties[ports].items.properties[targetPort].anyOf[0].type: Forbidden: must be empty to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[http].properties[s
ervice].properties[spec].properties[ports].items.properties[targetPort].anyOf[1].type: Forbidden: must be empty to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[http].properties[service].properties[spec].properties[ports].items.properties[targetPort].type: Required value: must not be 
empty for specified object fields, spec.validation.openAPIV3Schema.properties[spec].properties[podDisruptionBudget].properties[spec].properties[maxUnavailable].anyOf[0].type: Forbidden: must be empty to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[podDisruptionBudget].properties[spec
].properties[maxUnavailable].anyOf[1].type: Forbidden: must be empty to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[podDisruptionBudget].properties[spec].properties[maxUnavailable].type: Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.p
roperties[spec].properties[podDisruptionBudget].properties[spec].properties[minAvailable].anyOf[0].type: Forbidden: must be empty to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[podDisruptionBudget].properties[spec].properties[minAvailable].anyOf[1].type: Forbidden: must be empty to 
be structural, spec.validation.openAPIV3Schema.properties[spec].properties[podDisruptionBudget].properties[spec].properties[minAvailable].type: Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root]
    Reason:                Violations    Status:                True
    Type:                  NonStructuralSchema  Stored Versions:
    v1

[barkbay]

I don't have these violations, could you give more details about the steps you took and the K8S version (client and server) you are using ?

With K8S 1.12 / Client 1.17 and all-in-one at version 1.0.0-rc4 applied:

Status:
  Accepted Names:
    Categories:
      elastic
    Kind:       Elasticsearch
    List Kind:  ElasticsearchList
    Plural:     elasticsearches
    Short Names:
      es
    Singular:  elasticsearch
  Conditions:
    Last Transition Time:  2020-01-06T11:09:43Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  <nil>
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1

[sebgl]

I'm running Server 1.15.4-gke2 and Client 1.14 with 1.0.0-rc4.
1.15 brings additional support for structural schemas.

[anyasabo]

I can repro it in 1.15 and opened https://github.com/elastic/cloud-on-k8s/pull/2365 to make spinning up 1.15 a little easier. Same in Kibana:

[spec.validation.openAPIV3Schema.properties[spec].properties[http].properties[service].properties[spec].properties[ports].items.properties[targetPort].anyOf[0].type: Forbidden: must be empty to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[http].properties[service].properties[spec].properties[ports].items.properties[targetPort].anyOf[1].type: Forbidden: must be empty to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[http].properties[service].properties[spec].properties[ports].items.properties[targetPort].type: Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root]

and apm

[spec.validation.openAPIV3Schema.properties[spec].properties[http].properties[service].properties[spec].properties[ports].items.properties[targetPort].anyOf[0].type: Forbidden: must be empty to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[http].properties[service].properties[spec].properties[ports].items.properties[targetPort].anyOf[1].type: Forbidden: must be empty to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[http].properties[service].properties[spec].properties[ports].items.properties[targetPort].type: Required value: must not be empty for specified object fields, spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root]

[anyasabo]

So this comes from the us embedding the corev1.ServiceSpec which has TargetPort intstr.IntOrString
https://github.com/kubernetes/kubernetes/blob/8548a253de80c787dbcd2b022faffc75c7c23025/pkg/apis/core/types.go#L3545

and we would also need to specify x-kubernetes-int-or-string: true for that to work with structural schema.

See: https://github.com/kubernetes/website/blob/dev-1.15/content/en/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions.md#specifying-a-structural-schema

Looks like this already landed in controller-tools, but just missed the last release:
https://github.com/kubernetes-sigs/controller-tools/pull/360

So we should be able to update to the next version of controller-tools and regenerate CRDs to resolve these.

[anyasabo]

I went ahead and verified tried to verify that would fix it. Looks like it adds that marker at least. Here's the CRD generated against the master branch of controller-tools
https://github.com/anyasabo/cloud-on-k8s/commit/6c40e06d67a158d5aaf2f29c272cc508b7152b4d

But it causes more issues due to adding some other ones related to server side apply that don't seem to be recognized in 1.15:
https://github.com/kubernetes-sigs/structured-merge-diff/issues/115

19:23 $ kubectl apply -f config/crds/all-crds.yaml
error: error validating "config/crds/all-crds.yaml": error validating data: [ValidationError(CustomResourceDefinition.spec.validation.openAPIV3Schema.properties.spec.properties.http.properties.service.properties.spec.properties.ports): unknown field "x-kubernetes-list-map-keys" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaProps, ValidationError(CustomResourceDefinition.spec.validation.openAPIV3Schema.properties.spec.properties.http.properties.service.properties.spec.properties.ports): unknown field "x-kubernetes-list-type" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaProps]; if you choose to ignore these errors, turn validation off with --validate=false

tl;dr I think we want to wait until the next release of controller-tools and revisit

[CannibalVox]

Status: Accepted Names: Categories: elastic Kind: Elasticsearch List Kind: ElasticsearchList Plural: elasticsearches Short Names: es Singular: elasticsearch Conditions: Last Transition Time: 2021-04-06T03:34:19Z Message: [spec.preserveUnknownFields: Invalid value: true: must be false, spec.versions[0].schema.openAPIV3Schema.properties[spec].properties[http].properties[service].properties[spec].properties[ports].items.properties[targetPort].type: Required value: must not be empty for specified object fields, spec.versions[0].schema.openAPIV3Schema.properties[spec].properties[nodeSets].items.properties[volumeClaimTemplates].items.properties[spec].properties[resources].properties[limits].additionalProperties.type: Required value: must not be empty for specified object fields, spec.versions[0].schema.openAPIV3Schema.properties[spec].properties[nodeSets].items.properties[volumeClaimTemplates].items.properties[spec].properties[resources].properties[requests].additionalProperties.type: Required value: must not be empty for specified object fields, spec.versions[0].schema.openAPIV3Schema.properties[spec].properties[nodeSets].items.properties[volumeClaimTemplates].items.properties[status].properties[capacity].additionalProperties.type: Required value: must not be empty for specified object fields, spec.versions[0].schema.openAPIV3Schema.properties[spec].properties[podDisruptionBudget].properties[spec].properties[maxUnavailable].type: Required value: must not be empty for specified object fields, spec.versions[0].schema.openAPIV3Schema.properties[spec].properties[podDisruptionBudget].properties[spec].properties[minAvailable].type: Required value: must not be empty for specified object fields, spec.versions[0].schema.openAPIV3Schema.properties[spec].properties[transport].properties[service].properties[spec].properties[ports].items.properties[targetPort].type: Required value: must not be empty for specified object fields, spec.versions[0].schema.openAPIV3Schema.type: Required value: must not be empty at the root, spec.versions[1].schema.openAPIV3Schema.properties[spec].properties[http].properties[service].properties[spec].properties[ports].items.properties[targetPort].type: Required value: must not be empty for specified object fields, spec.versions[1].schema.openAPIV3Schema.properties[spec].properties[nodeSets].items.properties[volumeClaimTemplates].items.properties[spec].properties[resources].properties[limits].additionalProperties.type: Required value: must not be empty for specified object fields, spec.versions[1].schema.openAPIV3Schema.properties[spec].properties[nodeSets].items.properties[volumeClaimTemplates].items.properties[spec].properties[resources].properties[requests].additionalProperties.type: Required value: must not be empty for specified object fields, spec.versions[1].schema.openAPIV3Schema.properties[spec].properties[nodeSets].items.properties[volumeClaimTemplates].items.properties[status].properties[capacity].additionalProperties.type: Required value: must not be empty for specified object fields, spec.versions[1].schema.openAPIV3Schema.properties[spec].properties[podDisruptionBudget].properties[spec].properties[maxUnavailable].type: Required value: must not be empty for specified object fields, spec.versions[1].schema.openAPIV3Schema.properties[spec].properties[podDisruptionBudget].properties[spec].properties[minAvailable].type: Required value: must not be empty for specified object fields, spec.versions[1].schema.openAPIV3Schema.properties[spec].properties[transport].properties[service].properties[spec].properties[ports].items.properties[targetPort].type: Required value: must not be empty for specified object fields, spec.versions[1].schema.openAPIV3Schema.type: Required value: must not be empty at the root, spec.versions[2].schema.openAPIV3Schema.properties[spec].properties[http].properties[service].properties[spec].properties[ports].items.properties[targetPort].type: Required value: must not be empty for specified object fields, spec.versions[2].schema.openAPIV3Schema.properties[spec].properties[nodeSets].items.properties[volumeClaimTemplates].items.properties[spec].properties[resources].properties[limits].additionalProperties.type: Required value: must not be empty for specified object fields, spec.versions[2].schema.openAPIV3Schema.properties[spec].properties[nodeSets].items.properties[volumeClaimTemplates].items.properties[spec].properties[resources].properties[requests].additionalProperties.type: Required value: must not be empty for specified object fields, spec.versions[2].schema.openAPIV3Schema.properties[spec].properties[nodeSets].items.properties[volumeClaimTemplates].items.properties[status].properties[capacity].additionalProperties.type: Required value: must not be empty for specified object fields, spec.versions[2].schema.openAPIV3Schema.properties[spec].properties[podDisruptionBudget].properties[spec].properties[maxUnavailable].type: Required value: must not be empty for specified object fields, spec.versions[2].schema.openAPIV3Schema.properties[spec].properties[podDisruptionBudget].properties[spec].properties[minAvailable].type: Required value: must not be empty for specified object fields, spec.versions[2].schema.openAPIV3Schema.properties[spec].properties[transport].properties[service].properties[spec].properties[ports].items.properties[targetPort].type: Required value: must not be empty for specified object fields, spec.versions[2].schema.openAPIV3Schema.type: Required value: must not be empty at the root] Reason: Violations Status: True Type: NonStructuralSchema Last Transition Time: 2021-04-06T03:34:20Z Message: no conflicts found Reason: NoConflicts Status: True Type: NamesAccepted Last Transition Time: 2021-04-06T03:34:20Z Message: the initial names have been accepted Reason: InitialNamesAccepted Status: True Type: Established Stored Versions: v1 Events: <none>

Similar issues, installed from helm chart eck-operator-1.5.0, server version v1.20.2, kubectl client v1.19.7

[CannibalVox]

I just saw in a related issue an elastic engineer say that you could still spin up resources despite this issue, and I want to clarify that now that schema is out and gaining adoption, tools are starting to depend on it. The new experimental terraform kubernetes provider can spin up CRD resources, but it can't spin up ECK ones, because the schema is busted.