Client-go: Improve the coverage of examples

[caesarxuchao]

Apart from helping users, increasing the coverage of examples can also help our CI to detect problems.

We can start with providing examples for all top-level packages, and packages in tools/:

• [ ] dynamic
• [ ] discovery
• [ ] rest
• [ ] testing
• [ ] transport
• [ ] tools/auth
• [ ] tools/cache (https://github.com/kubernetes/client-go/pull/30)
• [ ] tools/portforward
• [ ] tools/record
• [ ] util/workqueue (https://github.com/kubernetes/kubernetes/pull/44320) (and we should move workqueue to tools/)

User requested example

• [ ] simple list and watch https://github.com/kubernetes/client-go/issues/161

@lavalamp

related: https://github.com/kubernetes/client-go/issues/63

[rmohr]

@caesarxuchao I can add one for tools/record.

[caesarxuchao]

Thanks @rmohr. Could you share what you have in mind before writing down the example? Never mind if you think writing a PR is easier.

[caesarxuchao]

cc @mbohlool

[rmohr]

@caesarxuchao posted the inital PR, will clean and properly document it after the weekend (I had some issues compiling the latest client-go code). Let me know if you think that this example is too simplistic.

[ahmetb]

I have a different take on how the Client Library samples should be done. I will argue that the examples should be more scenario-oriented, and not package-oriented as @caesarxuchao suggested above.

I see the value of documenting what's in each package, but I would argue that it’s not necessarily something a beginner user to the library (or Kubernetes API) needs. Maybe this seems like I'm trying to target rookies more, but as authors of client libraries, we often make assumptions how much users know about the API or how it can be used.

My goal is to simplify getting started experience of this library and provide pointers from www.kubernetes.io to the examples here, and create (more beginner-level) samples that make it less intimidating for others to get their hands dirty with the Kubernetes API.

Here’s a quick rundown of what I have in mind for examples/ directory. I am looking for feedback to refine the list further.

• Beginner Tasks
  > Scenario: ”I am new to this, I just did go get, how can I write a simple program”
  
  
  
  • Auth: in-cluster
  
  
  • Auth: out-of-cluster
  
  
  • CRUD: e.g. {create, get, list, delete, update}-deployment
  
  
  • Using Namespaces
  
  
• Advanced Tasks
  > Scenario: “I can do basic things with the API, tell me what’s next.”
  
  
  
  • Using the Watch API
  
  
  • Using Informers
  
  
• Sample Applications
  > Scenario: “I want to write my own Kubernetes extension, get me there quickly.”
  >
  > These are programs that can be directly compiled. people should be able to just take these projects and modify for their needs to write extensions. (Perhaps each of these should be in separate repos and not in examples/.)
  
  
  
  • Sample Scheduler
  
  
  • Sample Operator
  
  
  • Sample Third Party Resource
  
  
  • Sample Controller
  
  

Please provide feedback on which route is more agreeable. It’s just my opinion that we should provide real-world scenarios in examples/ directory and keep package-reference docs somewhere else.

In fact, I am seeking for feedback from actual 3rd party users of the API, if you know anyone, please send them my way, I'd love to listen to their experience with client-go.

[fejta]

Can we start with figuring out how to eliminate permanent failures from the example suites and tests?

I have an issue opened to delete some of these example suites because they have not passed once during the past 60 days.

https://github.com/kubernetes/test-infra/issues/2528

[ahmetb]

@fejta it looks like ci-kubernetes-e2e-gci-gce-examples and ci-kubernetes-e2e-gce-examples are regarding the examples in the examples/ directory in the main repo; not the ones in client-go. So I think this is a bit off-topic. Regarding those permanent failures, it looks like some of the samples in the main repo are unmaintained. We are having discussions to move those to a separate repo (https://github.com/kubernetes/examples) and deprecate the ones that are no longer maintained.

[caesarxuchao]

@ahmetb the progressive layout of examples sounds great. They don't cover all the components though, like /tools/portforward or the /dynamic. I think we can put examples for these packages under "advanced tasks".

[ahmetb]

@caesarxuchao agreed. If we can maintain a set of common and simple tasks (such as CRUD Pod) and scaffolding (e.g. sample operator) across official client libraries, there's plenty of room to document advanced aspects and patterns of each client library.

[fejta]

Thanks for the clarification. Agree my comment is off topic. Please ignore.

[rothgar]

@ahmetb I love the idea of having code examples. It helps new people get started and lets you jump in at your own level rather than code training sites that take you from the beginning. A couple questions I have though.

1. Is this the place for sample applications? It's probably recommended to use the go client to write a scheduler but it by no means is the only way to do it. I don't have an answer but would hate to scare people away from extending kubernetes or writing their own controllers just because they don't know go or want to use this client.
2. How about tips for extending kubectl? That is (currently) most people's interface to kubernetes and I'm sure I'm not the only one who has aliases, functions, and wrappers to do various things that I would rather be plugins or custom cli applications.

[ymruan]

A sample implementation of operator(controller) with TPR may help the developer to understand the client-go.

[ahmetb]

@rothgar great questions:

1. I am hoping to have sample apps (scaffolding scheduler etc) for all officially supported languages. In this case I believe it’s Python and Go. It's also my goal to not to scare people in any way and lower the barrier to extend Kubernetes as much as possible. So we’re on the same page. 👍 That said, if the scaffolding apps turn out to be way more complex then we imagined, we can host them on separate repos to keep the client-{go,python} repos clean.

2. I'd love to learn more about how people can extend kubectl other than tricks like aliases and jsonpath queries. We already got some of this stuff documented. However, this seems to be a separate discussion than API client examples and maybe suited better for the docs repository.

[rmohr]

@ahmetb We are using client-go in KubeVirt. I think the layout looks great. It pretty much covers everything which we currently use and it would have helped us to get started.

I would also add an example for completely customized http clients, which integrate with the kubernetes security layers. It took me some time to figure out, how to write your own websocket client. The example I came up with is located at https://github.com/rmohr/kubernetes-custom-exec.

[nulltrope]

I think it would also be very helpful to have an example of initiating rollbacks on deployments from client-go. I have been trying to research ways to do this for a few days now, and it's a pretty typical use case of this client so I think it'd help a lot of beginners and more advanced users alike. I'm still trying to figure it out myself but if I do I'd be happy to contribute an example.

[ahmetb]

@jekohk please feel free to contribute the rollback operation to the crud-deployment example we have if you have time.

[nulltrope]

@ahmetb I've updated the CRUD example to include a rollback operation, however I'm having issues getting integration tests to pass locally.
Running make test is throws the following:

can't load package: package k8s.io/kubernetes/_tmp/federation/pkg/dnsprovider: code in directory /home/john/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/_tmp/federation/pkg/dnsprovider expects import "k8s.io/kubernetes/federation/pkg/dnsprovider"
_tmp/federation/pkg/dnsprovider/providers/google/clouddns/clouddns.go:32:2: use of internal package not allowed
_tmp/federation/pkg/dnsprovider/providers/google/clouddns/interface.go:21:2: use of internal package not allowed
_tmp/federation/pkg/dnsprovider/providers/google/clouddns/clouddns.go:33:2: use of internal package not allowed
can't load package: package k8s.io/kubernetes/_tmp/federation/pkg/federation-controller/cluster: code in directory /home/john/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/_tmp/federation/pkg/federation-controller/cluster expects import "k8s.io/kubernetes/federation/pkg/federation-controller/cluster"

Any ideas? I've just been following the Community development guide- not sure if there some extra steps required when working under /staging directory.

[ahmetb]

@jekohk can you open a separate issue for this? It looks like one of the dependencies broke the example.

[caesarxuchao]

@jekohk it looks wrong if a client-go example requires compiling k8s.io/kubernetes. As @ahmetb suggested, let's discuss the problem in a separate issue.

[nulltrope]

@ahmetb @caesarxuchao i've elaborated more in #303

[nulltrope]

@ahmetb opened PR #53474 with some updates to client-go CRUD example.

I'm interested in the TODO mentioning a sleep with exponential backoff- is this something you thought should be included in the example, or just a suggestion to those running an example like this in prod?

[ahmetb]

@jekohk we have a retry package. we should use it where appropriate, for sure.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[mhausenblas]

@ahmetb are you working on this? I'm iterating on http://using-client-go.cloudnative.sh and plan to PR when there's something substantial to share …

[ahmetb]

@mhausenblas not actively. Happy to review more examples in line with the current ones or the bespoke list above.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[Paxa]

Would be nice to have example for watch

[liggitt]

Would be nice to have example for watch

https://github.com/kubernetes/client-go/blob/master/examples/workqueue/main.go#L164-L195 is pretty exemplary