Client-go: Support for parsing K8s yaml spec into client-go data structures

[newtonkishore]

Hi,

I am trying to create K8s cluster with spec coming from yaml files. I had to write Yaml equivalent of the same datastructures to convert the spec from yaml to go data structures to start the cluster.

The Kubernetes/client-go data structures are tuned to Json Parsing, can we just add "yaml:"" " support to client-go datastructures?

Or is there a better way to do this?

Thanks.

[ahmetb]

@newtonkishore consider converting from YAML to JSON and then unmarshaling into structs as the json:"" field tags are there. The API works with JSON primarily and kubectl converts YAML to JSON before sending the request.

[ahmetb]

Or actually I'm not sure, I saw something in the issue description of #194 which looks like there's a universal decoder you can use.

[rmohr]

@newtonkishore you can use yaml.NewYAMLOrJSONDecoder(reader, buffSize).Decode(k8sStruct) from k8s.io/client-go/pkg/util/yaml. You can pass in JSON or YAML, and it will use the json:"" annotations for both formats.

[kenontech]

Follow up to this question, what is the right decoder to use if the type of the k8sStruct is not known before decoding?

[christianhuening]

yaml.NewYAMLOrJSONDecoder() etc. has been moved to k8s.io/apimachinery/pkg/util/yaml

[lavalamp]

Yes, there is already code laying around that does this and possibly a few
other necessary things. I'd be a bit surprised if it's not included in the
client library; if not, we can move it.

On Tue, Nov 7, 2017 at 8:57 AM, Christian Hüning [e-mail]
wrote:

yaml.NewYAMLOrJSONDecoder() etc. has been moved to
k8s.io/apimachinery/pkg/util/yaml

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/client-go/issues/193#issuecomment-342548856,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAnglqdo5YO9PkNlFPTGrO8GfqKaxsgkks5s0IwGgaJpZM4NahN8
.

[christianhuening]

@kenontech @ahmetb @newtonkishore @rmohr Here's my solution (thanks to Stefan Schimanski for valuable hints!):

import (
    "k8s.io/client-go/pkg/api"
    _ "k8s.io/client-go/pkg/api/install"
    _ "k8s.io/client-go/pkg/apis/extensions/install"
    _ "k8s.io/client-go/pkg/apis/rbac/install"
    //...
    "k8s.io/client-go/pkg/api/v1"
    "k8s.io/client-go/pkg/apis/rbac/v1beta1"
    // make sure to import all client-go/pkg/api(s) you need to cover with your code
)

func someFunc() {
    // yamlFiles is an []string
    for _, f := range yamlFiles {

        decode := api.Codecs.UniversalDeserializer().Decode
        obj, groupVersionKind, err := decode([]byte(f), nil, nil)

        if err != nil {
            log.Fatal(fmt.Sprintf("Error while decoding YAML object. Err was: %s", err))
        }

        // now use switch over the type of the object
        // and match each type-case
        switch o := obj.(type) {
        case *v1.Pod:
            // o is a pod
        case *v1beta1.Role:
            // o is the actual role Object with all fields etc
        case *v1beta1.RoleBinding:
        case *v1beta1.ClusterRole:
        case *v1beta1.ClusterRoleBinding:
        case *v1.ServiceAccount:
        default:
            //o is unknown for us
        }
    }
}

[mofelee]

@christianhuening

Why my client-go doesn't have api inside pkg folder?

I use go get k8s.io/client-go/... to get client-go

[lavalamp]

You are at a different version of the client, in which that code has moved
to the k8s.io/api/... package.

On Thu, Nov 9, 2017 at 3:53 PM, MofeLee [e-mail] wrote:

@christianhuening https://github.com/christianhuening

[image: image]
https://user-images.githubusercontent.com/5069410/32635653-cd7eaeb2-c576-11e7-9bac-b86cc724c79c.png

[image: image]
https://user-images.githubusercontent.com/5069410/32635545-5b54cf92-c576-11e7-91b2-9650571b333e.png

Why my client-go doesn't have api inside pkg folder?

I use go get k8s.io/client-go/... to get client-go

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/client-go/issues/193#issuecomment-343330519,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAnglnhOQxMGBNsQqGM0_2flv4CUM23Gks5s05B3gaJpZM4NahN8
.

[mofelee]

@lavalamp Thanks!

[Iazel]

For future reference, if you don't have have k8s.io/client-go/pkg/api, then:

import "k8s.io/client-go/kubernetes/scheme"
scheme.Codecs.UniversalDeserializer()

[mhindery]

I am trying a similar thing and am completely lost. What packages are moved to where? I've read the scheme.Codecs.* are not supposed to be used anymore, and I have no clue what to import.

My goal is to read a yaml file with a list of rolebindings into a client-go structure, and there seem to be no possibility to do so, and no documentation at all. I've asked on the Slack channel and nobody could help there. There are some SO questions, but none have an answer to how to work with multiple items in a list in a yaml file. e.g.:

https://stackoverflow.com/questions/44306554/how-to-deserialize-kubernetes-yaml-file.
https://stackoverflow.com/questions/47116811/client-go-parse-kubernetes-json-files-to-k8s-structures?noredirect=1&lq=1

My yaml file looks like this

apiVersion: v1
kind: List
items:
# --- # Full cluster admin
- kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: ***
    labels:
      source: ***
  roleRef:
    kind: ClusterRole
    name: cluster-admin
    apiGroup: rbac.authorization.k8s.io
  subjects:
  - kind: User
    name: ***
    apiGroup: rbac.authorization.k8s.io
...

This can be used perfectly with kubectl apply -f ... however it seems impossible to parse using go ...

[christianhuening]

@mhindery https://github.com/kubernetes/client-go/issues/193#issuecomment-343138889 doesn't work anymore?

[sttts]

The packages have changes. The codec to use is https://github.com/kubernetes/client-go/blob/master/kubernetes/scheme/register.go#L55. But otherwise, not much has changed.

[mhindery]

While I can read the file using the method above, I never get into the case in the switch with the correct type. Using this file

package main

import (
    "fmt"

    rbacv1 "k8s.io/api/rbac/v1"
    "k8s.io/client-go/kubernetes/scheme"
)

var filecontent = `
apiVersion: v1
kind: List
items:
- kind: ClusterRoleBinding
  apiVersion: rbac.authorization.k8s.io/v1
  metadata:
    name: myName
    labels:
      source: myLabel
  roleRef:
    kind: ClusterRole
    name: cluster-admin
    apiGroup: rbac.authorization.k8s.io
  subjects:
  - kind: User
    name: [email protected]
    apiGroup: rbac.authorization.k8s.io
`

func main() {
    decode := scheme.Codecs.UniversalDeserializer().Decode
    obj, _, _ := decode([]byte(filecontent), nil, nil)

    fmt.Printf("%++v\n\n", obj.GetObjectKind())
    fmt.Printf("%++v\n\n", obj)

    cbl := obj.(*rbacv1.ClusterRoleBindingList) // This fails
    fmt.Printf("%++v\n", cbl)

    switch o := obj.(type) {
    case *rbacv1.ClusterRoleBindingList:
        fmt.Println("correct found") // Never happens
    default:
        fmt.Println("default case")
        _ = o
    }
}

I get this output:

&TypeMeta{Kind:List,APIVersion:v1,}

&List{ListMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ListMeta{SelfLink:,ResourceVersion:,Continue:,},Items:[{[123 34 97 112 105 86 101 114 115 105 111 110 34 58 34 114 98 97 99 46 97 117 116 104 111 114 105 122 97 116 105 111 110 46 107 56 115 46 105 111 47 118 49 34 44 34 107 105 110 100 34 58 34 67 108 117 115 116 101 114 82 111 108 101 66 105 110 100 105 110 103 34 44 34 109 101 116 97 100 97 116 97 3458 123 34 108 97 98 101 108 115 34 58 123 34 115 111 117 114 99 101 34 58 34 109 121 76 97 98 101 108 34 125 44 34 110 97 109 101 34 58 34 109 121 78 97 109 101 34 125 44 34 114 111 108 101 82 101 102 3458 123 34 97 112 105 71 114 111 117 112 34 58 34 114 98 97 99 46 97 117 116 104 111 114 105 122 97 116 105 111 110 46 107 56 115 46 105 111 34 44 34 107 105 110 100 34 58 34 67 108 117 115 116 101 114 82111 108 101 34 44 34 110 97 109 101 34 58 34 99 108 117 115 116 101 114 45 97 100 109 105 110 34 125 44 34 115 117 98 106 101 99 116 115 34 58 91 123 34 97 112 105 71 114 111 117 112 34 58 34 114 98 97 99 46 97 117 116 104 111 114 105 122 97 116 105 111 110 46 107 56 115 46 105 111 34 44 34 107 105 110 100 34 58 34 85 115 101 114 34 44 34 110 97 109 101 34 58 34 109 121 64 117 115 101 114 46 99 111 109 34 125 93 125] <nil>}],}

panic: interface conversion: runtime.Object is *v1.List, not *v1.ClusterRoleBindingList

The object I end up with is of the type runtime.Object and I don't see anywhere how to convert this to my desired type (the ClusterRoleBindingList). From the printed output, I can see the object has the fields items / typemeta / listmeta somewhere in it, which the ClusterRoleBindingList also defines, so the data seems to be in there. It's 'simply' the conversion between runtime.Object and my type that I'm missing.

In https://github.com/kubernetes/client-go/issues/193#issuecomment-343138889, you seem to immediately end up with the right types in the switch-case, while I have a generic one. I've looked at the linked scheme.Scheme.Conver() function, but that hasn't worked for me. What am I missing?

[christianhuening]

@mhindery As your error states your (...) runtime.Object is *v1.List, not *v1.ClusterRoleBindingList, so I guess you should first switch case for v1.List and then look into the items of that list.
I don't use that list type, but have a single file, where each entry is separated via ---.
Here's my parseK8sYaml(...):

func parseK8sYaml(fileR []byte) []runtime.Object {

    acceptedK8sTypes := regexp.MustCompile(`(Role|ClusterRole|RoleBinding|ClusterRoleBinding|ServiceAccount)`)
    fileAsString := string(fileR[:])
    sepYamlfiles := strings.Split(fileAsString, "---")
    retVal := make([]runtime.Object, 0, len(sepYamlfiles))
    for _, f := range sepYamlfiles {
        if f == "\n" || f == "" {
            // ignore empty cases
            continue
        }

        decode := scheme.Codecs.UniversalDeserializer().Decode
        obj, groupVersionKind, err := decode([]byte(f), nil, nil)

        if err != nil {
            log.Println(fmt.Sprintf("Error while decoding YAML object. Err was: %s", err))
            continue
        }

        if !acceptedK8sTypes.MatchString(groupVersionKind.Kind) {
            log.Printf("The custom-roles configMap contained K8s object types which are not supported! Skipping object with type: %s", groupVersionKind.Kind)
        } else {
            retVal = append(retVal, obj)
        }

    }
    return retVal
}

[huydinhle]

@christianhuening Thanks for your solution. I am having a hard time to convert the runtime.Object to let's say something like v1.ConfigMap. Any suggestion?

[harryleesan]

I have a similar issue to @huydinhle, however, I am trying to convert runtime.Object to a Deployment using the following:

package main

import (
  "fmt"

  "k8s.io/api/apps/v1beta1"
  "k8s.io/client-go/kubernetes/scheme"
)

var json = `
{
  "apiVersion": "extensions/v1beta1",
  "kind": "Deployment",
  "metadata": null,
  "name": "my-nginx",
  "replicas": 2,
  "spec": null,
  "template": {
    "metadata": {
      "labels": {
        "run": "my-nginx"
      }
    },
    "spec": {
      "containers": [
        {
          "image": "nginx",
          "name": "my-nginx",
          "ports": [
            {
              "containerPort": 80
            }
          ]
        }
      ]
    }
  }
}
`

func main() {
    decode := scheme.Codecs.UniversalDeserializer().Decode

    obj, _, err := decode([]byte(json), nil, nil)
    if err != nil {
        fmt.Printf("%#v", err)
    }

    deployment := obj.(*v1beta1.Deployment)

    fmt.Printf("%#v\n", deployment)
}

I get this error:

panic: interface conversion: runtime.Object is *v1beta1.Deployment, not *v1beta1.Deployment

My glide.yaml:

package: main
import:
- package: k8s.io/client-go
  version: v6.0.0

The error does not make sense to me since it seems like the type is the same. The conversion from runtime.Object to resources is not intuitive to me. Was anyone able to solve this?

[harryleesan]

I have managed to solve it... the deployment resource is of apiVersion extensions/v1beta1 and I imported k8s.io/api/apps/v1beta1, once I changed it to k8s.io/api/extensions/v1beta1 everything worked!

Here is the working code:

package main

import (
  "fmt"

  "k8s.io/api/extensions/v1beta1"
  "k8s.io/client-go/kubernetes/scheme"
)

var json = `
{
  "apiVersion": "extensions/v1beta1",
  "kind": "Deployment",
  "metadata": null,
  "name": "my-nginx",
  "replicas": 2,
  "spec": null,
  "template": {
    "metadata": {
      "labels": {
        "run": "my-nginx"
      }
    },
    "spec": {
      "containers": [
        {
          "image": "nginx",
          "name": "my-nginx",
          "ports": [
            {
              "containerPort": 80
            }
          ]
        }
      ]
    }
  }
}
`

func main() {
    decode := scheme.Codecs.UniversalDeserializer().Decode

    obj, _, err := decode([]byte(json), nil, nil)
    if err != nil {
        fmt.Printf("%#v", err)
    }

    deployment := obj.(*v1beta1.Deployment)

    fmt.Printf("%#v\n", deployment)
}

[trobert2]

why does scheme.Codecs.UniversalDeserializer().Decode while fetching a kubernetes object from file return the proper TypeMeta and when fetching the object from the cluster (a config map in this case) using

clientset, err := kubernetes.NewForConfig(config)
clientset.CoreV1().ConfigMaps(namespace).Get(name, metav1.GetOptions{})

the TypeMeta is empty.

  TypeMeta: {
-  Kind: "ConfigMap",
-  APIVersion: "v1",
+  Kind: "",
+  APIVersion: "",
  },

Tested on HEAD and version 6 of the client-go

If I add a print statement in the func (r *Request) Do() Result function in: k8s.io/client-go/rest/request.go here

like so: fmt.Printf("%++v\n\n", string(result.body[:len(result.body)]))

I can see the response from the server being:

{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"test-map","namespace":"default","selfLink":"/api/v1/namespaces/default/configmaps/test-map","uid":"27ccefdc-463b-11e8-bd71-080027205854","resourceVersion":"1584679","creationTimestamp":"2018-04-22T14:40:51Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"data\":{\"game.properties\":\"enemies=aliens\\nlives=3\\nenemies.cheat=true\\nenemies.cheat.level=noGoodRotten\\nsecret.code.passphrase=UUDDLRLRBABAS\\nsecret.code.allowed=true\\nsecret.code.lives=30\\n\",\"ui.properties\":\"color.good=purple\\ncolor.bad=yellow\\nallow.textmode=true\\nhow.nice.to.look=fairlyNice\\n\"},\"kind\":\"ConfigMap\",\"metadata\":{\"annotations\":{},\"name\":\"test-map\",\"namespace\":\"default\"}}\n"}},"data":{"game.properties":"enemies=aliens\nlives=3\nenemies.cheat=true\nenemies.cheat.level=noGoodRotten\nsecret.code.passphrase=UUDDLRLRBABAS\nsecret.code.allowed=true\nsecret.code.lives=30\n","ui.properties":"color.good=purple\ncolor.bad=yellow\nallow.textmode=true\nhow.nice.to.look=fairlyNice\n"}}

kind and apiVersion are there in the response. Is there something wrong with the Marshal/Unmarshal process? what's happening here?

[27149chen]

Hi, I'm trying to covert a json to a crd struct in my unittest, using the decoder "scheme.Codecs.UniversalDeserializer().Decode". but got an error no kind "*" is registered for version "*"
Do I need to "register" it, and how?

[dionysius]

@27149chen

This happened to me with my own CRD's, you'll need to add the scheme like:

//import
    myscheme "github.com/user/project/go/client/clientset/versioned/scheme"

//use AddToScheme before using the Decoder
    myscheme.AddToScheme(scheme.Scheme)

[integrii]

I used a nifty YAMLtoJSON package to get this working.

Check it out:

// yamlBytes contains a []byte of my yaml job spec
// convert the yaml to json
jsonBytes, err := yaml.YAMLToJSON(yamlBytes)
if err != nil {
    return v1.Job{}, err
}
// unmarshal the json into the kube struct
var job = v1.Job{}
err = json.Unmarshal(jsonBytes, &job)
if err != nil {
    return v1.Job{}, err
}
// job now contains the for the job that was in the YAML!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[weibeld]

Currently also solving it with github.com/ghodss/yaml. This package can deserialise from YAML by using the json struct tags that are defined in the K8s API object structs. So, the deserialisation should be correct, as if you would deserialise from JSON instead of YAML.

~~~go
import (
"io/ioutil"
"github.com/ghodss/yaml"
"k8s.io/api/apps/v1"
)

func main() {
bytes, err := ioutil.ReadFile("deployment.yml")
if err != nil {
panic(err.Error())
}

var spec v1.Deployment
err = yaml.Unmarshal(bytes, &spec)
if err != nil {
panic(err.Error())
}
}
~~~

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[geekodour]

@dionysius @27149chen
I am getting the error:

 no kind "CustomResourceDefinition" is registered for version "apiextensions.k8s.io/v1beta1" in scheme "pkg/runtime/scheme.go:101"
usage: prombench gke resource apply

here the Kind is CustomResourceDefinition itself, should not it have been the name of the CRD?

[g0194776]

For future reference, if you don't have have k8s.io/client-go/pkg/api, then:

import "k8s.io/client-go/kubernetes/scheme"
scheme.Codecs.UniversalDeserializer()

thank you very much.

[Colstuwjx]

just like @geekodour , I also met error:

&runtime.notRegisteredErr{schemeName:"k8s.io/client-go/kubernetes/scheme/register.go:61", gvk:schema.GroupVersionKind{Group:"apps", Version:"__internal", Kind:"Deployment"}, target:runtime.GroupVersioner(nil), t:reflect.Type(nil)}--- FAIL: TestCodecsManifests (0.00s)
panic: interface conversion: runtime.Object is nil, not *v1.Deployment [recovered]
    panic: interface conversion: runtime.Object is nil, not *v1.Deployment

deployment yaml definition is here:

var yml = `
apiVersion: apps/v1
kind: Deployment
metadata:
    name: my-complex-app-my-complex-app
    labels:
    app: my-complex-app
    chart: my-complex-app-0.2.0
    release: my-complex-app
    heritage: Tiller
spec:
    replicas: 1
    template:
    metadata:
        labels:
        app: my-complex-app
        release: my-complex-app
    spec:
        containers:
        - name: my-complex-app
        image: "nginx:stable"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        livenessProbe:
        httpGet:
            path: /
            port: 80
        readinessProbe:
        httpGet:
            path: /
            port: 80
        resources: {}
`

Updated:

it seems that I should use scheme.Codecs.UniversalDeserializer().Decode rather than scheme.Codecs.UniversalDecoder().Decode

[DiveInto]

@dionysius @27149chen
I am getting the error:

 no kind "CustomResourceDefinition" is registered for version "apiextensions.k8s.io/v1beta1" in scheme "pkg/runtime/scheme.go:101"
usage: prombench gke resource apply

here the Kind is CustomResourceDefinition itself, should not it have been the name of the CRD?

@geekodour have you solved the problem?

I come to the same situation here, trying to parse CRD:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: kongconsumers.configuration.konghq.com
spec:
  additionalPrinterColumns:
  ...

after addToScheme in init():

import (
    ...
    apiextv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1"
         ...
    clientgoscheme "k8s.io/client-go/kubernetes/scheme"
    ...
    // +kubebuilder:scaffold:imports
)

func init() {
    _ = clientgoscheme.AddToScheme(scheme)

    _ := apiextv1beta1.AddToScheme(scheme)
        ...
}

still see error like this:

no kind "CustomResourceDefinition" is registered for version "apiextensions.k8s.io/v1beta1" in scheme "pkg/runtime/scheme.go:101"

why is that?

---

update:

we need mannually add the scheme: apis/apiextensions/v1beta1 like this:

import(
    apiextv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1"
    ...
    clientgoscheme "k8s.io/client-go/kubernetes/scheme"
)

...

sch := runtime.NewScheme()
_ = clientgoscheme.AddToScheme(sch)
_ = apiextv1beta1.AddToScheme(sch)

decode := serializer.NewCodecFactory(sch).UniversalDeserializer().Decode

[ciiiii]

@DiveInto

import(
  apiextv1beta1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1"
  ...
  clientgoscheme "k8s.io/client-go/kubernetes/scheme"
)

...

sch := runtime.NewScheme()
_ = clientgoscheme.AddToScheme(sch)
_ = apiextv1beta1.AddToScheme(sch)

decode := serializer.NewCodecFactory(sch).UniversalDeserializer().Decode

Thank you, this method works!

[harsh-98]

It is really helpful for decoding CRDs or other resource from yaml.

[Abirdcfly]

but If my yaml is kubeconfig.(ns:kube-public cm:cluster-info).I have to use ("k8s.io/client-go/tools/clientcmd")clientcmd.Load.

yaml.NewYAMLOrJSONDecoder() etc will get me a error

"error unmarshaling JSON: while decoding JSON: json: cannot unmarshal array into Go struct field Config.clusters of type map[string]*api.Cluster"

why?  😭