Kubebuilder: Parse byte array for yaml value - kubebuilder

[JennyMet]

HI,

My team are using kubebuilder (version 3 beta ) to build our custom controller, the problem is we are not able to parse raw data as it comes empty when you apply the file.

I’ve created some minimal example which describe the issue.
apiVersion: mygroup.test.com/v1alpha1

kind: Rawtest
metadata:
  name: rawtest-sample
spec:
  system:
    type: test
    provider:
      type: aws
      infrastructureConfig:
        kind: InfrastructureConfig
        apiVersion: v1alpha1
        networks:
          vpc:
            cidr: aaa

In runtime I was able to get the the spec.system.type value=test and spec.system.provider.type value="aws", however I wasn’t able to get all the data under the infrastructureConfig: (line 10) any idea how can I overcome this ?

I’ve created this very simple project to demonstrate the issue ,
See the api/type folder, after getting the reconcile object (after apply the config/sample/ file ,you see that the infrastructureconfig and all related data are

Here is the code which is trying to read the raw value
https://github.com/JennyMet/kuberaw/blob/master/controllers/rawtest_controller.go#L57

&rawtest should contain all the data

please see the type
https://github.com/JennyMet/kuberaw/blob/master/api/v1alpha1/rawtest_types.go#L32

raw type
https://github.com/gardener/gardener/blob/bf32324d9d1a366d8a0a7514956dc39c2f22f7b7/pkg/apis/core/v1beta1/types_shoot.go#L945

https://github.com/gardener/gardener/blob/bf32324d9d1a366d8a0a7514956dc39c2f22f7b7/pkg/apis/core/types_shoot.go#L774

https://github.com/gardener/gardener/blob/bf32324d9d1a366d8a0a7514956dc39c2f22f7b7/vendor/k8s.io/apimachinery/pkg/runtime/types.go#L94:6

I need a way to make it work in the kubebuilder, as while I apply the file I dont get the values in debug ...

debug pic

/kind bug

[camilamacedo86]

It is not a bug in the kubebuilder tool. It seems like you are looking for help to know how to implement your API spec in this case. So, I am changing the label here for support. Also, appears that it also is answered in https://stackoverflow.com/questions/66637260/parse-byte-array-for-yaml-value-kubebuilder.

In this way, could you please let us know if you still looking for help or if we can close this one?

[JennyMet]

@camilamacedo86 - The answer in SO is not solving the issue therefore I put some bounty (which doesnt help either). if you think that this is not an issue with the kubebuilder, where it is ? could you give any hint or something as we are stuck a lot of time with it... thanks.

[Adirio]

@JennyMet I don't see your issue. runtime.RawExtension is working as expected:

package main

import (
    "fmt"

    "k8s.io/apimachinery/pkg/runtime"
    "sigs.k8s.io/yaml"
)

type A struct {
    RawField *runtime.RawExtension `json:"rawField,omitempty"`
}

const aYaml = `rawField: {}`

func main() {
    var a A

    if err := yaml.Unmarshal([]byte(aYaml), &a); err != nil {
        fmt.Println(err)
        return
    }
    fmt.Println(a.RawField.Raw)

    if output, err := yaml.Marshal(a); err != nil {
        fmt.Println(err)
        return
    } else {
        fmt.Println(string(output))
    }
}

outputs

[123 125]
rawField: {}

as expected.

Could you print rawtest in your example controller (where the // your logic here is placed) and provide us the input and the output you are getting?

[JennyMet]

HI @Adirio ,

the output is {nil}{}

what I've tried is

type A struct {
    RawField *runtime.RawExtension `json:"rawField,omitempty"`
}
func (r *RawtestReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    _ = r.Log.WithValues("rawtest", req.NamespacedName)
    var a A
    var rawtest mygroupv1alpha1.Rawtest
    // get edge object
    if err := r.Get(ctx, req.NamespacedName, &rawtest); err != nil {
        if !apierrors.IsNotFound(err) {
            fmt.Println(err)
        }
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    if err := yaml.Unmarshal([]byte(rawtest.Spec.System.Provider.InfrastructureConfig.Raw), &a); err != nil {
        fmt.Println(err)
        return ctrl.Result{}, nil
    }

    fmt.Print(a)


    if output, err := yaml.Marshal(a); err != nil {
        fmt.Println(err)
        return ctrl.Result{}, nil
    } else {
        fmt.Println(string(output))
    }

    // your logic here

    return ctrl.Result{}, nil
}

do I miss something ?
https://github.com/JennyMet/kuberaw

Thanks,
Jenny

[Adirio]

The A type was just an example, you use Rawtest directly:

func (r *RawtestReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    _ = r.Log.WithValues("rawtest", req.NamespacedName)
    var rawtest mygroupv1alpha1.Rawtest
    // get edge object
    if err := r.Get(ctx, req.NamespacedName, &rawtest); err != nil {
        if !apierrors.IsNotFound(err) {
            fmt.Println(err)
        }
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    fmt.Println(rawtest.Spec.System.Provider.InfrastructureConfig)

    return ctrl.Result{}, nil
}

[JennyMet]

tried it now... did you able to get the data as I got {} see in debug please

Do I miss something ? does it works for you?

you can just clone the project run make install and debug it, you will see it quickly :(

[Adirio]

And which is the YAML you are applying? I mean [123 125] which is {} is a valid object. I don't know which value you are expecting if you don't provide the input that is being reconciled.

[JennyMet]

this is the yaml https://github.com/JennyMet/kuberaw/blob/master/config/samples/mygroup_v1alpha1_rawtest.yaml and from here the data is omitted, please see https://github.com/JennyMet/kuberaw/blob/master/config/samples/mygroup_v1alpha1_rawtest.yaml#L11

[JennyMet]

@Adirio - Hey, did you able to figure out this tricky issue?
if there is something is missing please let me know. thanks!

[Adirio]

I couldn't. I will move this question to the #kubebuilder slack channel to see if anyone knows about RawExtansion usage with controller-runtime library.

[alvaroaleman]

Well, the client doesn't return it because its not in the server, probably an issue with your CRD definition:

$ k apply -f  config/crd/bases/mygroup.test.com_rawtests.yaml 
customresourcedefinition.apiextensions.k8s.io/rawtests.mygroup.test.com created
$ k apply -f config/samples/mygroup_v1alpha1_rawtest.yaml 
rawtest.mygroup.test.com/rawtest-sample created
$ cat  config/samples/mygroup_v1alpha1_rawtest.yaml |yaml2json |jq .spec
{
  "system": {
    "provider": {
      "infrastructureConfig": {
        "apiVersion": "v1alpha1",
        "kind": "InfrastructureConfig",
        "networks": {
          "vpc": {
            "cidr": "aaa"
          },
          "zones": [
            {
              "internal": 123,
              "name": "abc",
              "public": 456,
              "workers": 789
            }
          ]
        }
      },
      "type": "aws",
      "workers": [
        {
          "machine": {
            "image": {
              "name": "name1",
              "version": "2"
            },
            "type": "mt"
          },
          "maxUnavailable": 0,
          "maximum": 2,
          "minimum": 1,
          "name": "myworker",
          "volume": {
            "size": "20Gi",
            "type": "a1"
          },
          "zones": [
            "zone1"
          ]
        }
      ]
    },
    "type": "test"
  }
}
$ k get rawtest rawtest-sample -o json|jq .spec
{
  "system": {
    "provider": {
      "infrastructureConfig": {},
      "type": "aws",
      "workers": [
        {
          "machine": {
            "image": {
              "name": "name1",
              "version": "2"
            },
            "type": "mt"
          },
          "maxUnavailable": 0,
          "maximum": 2,
          "minimum": 1,
          "name": "myworker",
          "volume": {
            "size": "20Gi",
            "type": "a1"
          },
          "zones": [
            "zone1"
          ]
        }
      ]
    },
    "type": "test"
  }
}

[JennyMet]

@Adirio - thanks a lot for your help!
@alvaroaleman - thanks, not sure I got it as I use the kubebuilder to generate it, (using make manifests which generate the CRD)should I do something in addition?

[alvaroaleman]

I don't know offhand why this doesn't work, the CRD looks correct to me.
@rfranzke @timebertt Do you know offhand about any issues with the generated openapi schema for the gardener apis/core/v1beta1.Provider type? Ref: https://github.com/JennyMet/kuberaw/blob/dfdf7bace15b6b46b707f41e7ed9424e616cdd5b/api/v1alpha1/rawtest_types.go#L32

[coderanger]

I think your version of controller-tools is too old. Looking at your generated CRD

https://github.com/JennyMet/kuberaw/blob/master/config/crd/bases/mygroup.test.com_rawtests.yaml#L52-L56

It doesn't have x-kubernetes-preserve-unknown-fields. Compare to a similar setup with newer controller-tools:

https://github.com/coderanger/rabbitmq-operator/blob/82f2d4a5e943ea318fe3e95fabe9b5b7961ef878/config/crd/bases/rabbitmq.coderanger.net_rabbitvhosts.yaml#L68-L72

[rfranzke]

I think @coderanger is right, I remember something similar with other CRDs using runtime.RawExtension (e.g., https://github.com/gardener/gardener/blob/9df08ae3ae865f96db4161eb4ad01ced73107f3e/charts/seed-bootstrap/charts/extensions/templates/crd-infrastructure.yaml#L59).

[JennyMet]

@coderanger @rfranzke as this is generated code and we are using kubebuilder https://github.com/kubernetes-sigs/kubebuilder/releases/tag/v3.0.0-beta.0 , should we use different version? or diff lib version ?

[coderanger]

Check the version of controller-gen you are using locally.

[JennyMet]

@coderanger - Thanks a lot for your help! not sure how to do it as I dont have the controller gen installed in my machine ...I tried to change the version of controller gen in my Makefile to 0.5.0 (before we use 0.4.1)

And run

1. make controller-gen
2. make uninstall
3. make install
4. make run
5. apply the file

Still the propertyx-kubernetes-preserve-unknown-fields: true doesnt added, should I do something else ?