Kubevirt: Add everything that is needed to make virtctl a kubectl binary plugin

[fabiand]

virttl can be a binary plugin to kubectl.

In order to achieve this, we need provide additional meta-data and some tooling (i.e. a command in virtctl itself?) to install it.

The metadata description can be found in the official kubernetes docs:
https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/

Rough items:

• [ ] Define metadata
• [ ] virtctl kubectl plug/unplug to install
• [ ] User guide update

[rmohr]

+1 a virtctl install would be nice.

[slintes]

things to do for implementing this:

• read and understand how kubectl plugins work: https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/
• check which subcommands / arguments / flags are supported by virtctl, see https://github.com/kubevirt/kubevirt/blob/master/pkg/virtctl/root.go#L21
• create a plugin manually, by
  
  
  
  • creating the plugin.yaml file
  
  
  • put it alongside with the virtctl binary to ~/.kube/plugins/kubevirt
  
  
  • test it
  
  
  • add plugin.yaml to the kubevirt repository, I suggest in cmd/virtctl/kubectl-plugin, together with a readme which explains what the file is for and how to use it
  
  
• implement a virtctl install command, which installs itself as kubectl plugin, by putting plugin.yaml and itself into the location mentioned above
• last but not least it would be nice to generate the plugin.yaml file (maybe even "on the fly" during virtctl install), by getting all needed information from the code place mentioned above (I think this could also be a follow-up issue, if it's too complicated)

[SchSeba]

Hi I can work on this if no one take this.

@fabiand @rmohr @slintes just tell me what option we want to use.

Option 1:

kubectl -n namespace plugin console/vnc/start/stop/expose

Option 2:

kubectl -n namespace plugin kubevirt console/vnc/start/stop/expose

[slintes]

I'm for option 2: 1 kubevirt plugin with x subcommands. Otherwise it's not clear where the commands belong to imho.

[rmohr]

I'm for option 2: 1 kubevirt plugin with x subcommands. Otherwise it's not clear where the commands belong to imho.

Also option two, with kubevirt or virt as entrypoint for the same reason

[SchSeba]

@slintes @rmohr no problem start working on this

[fabiand]

2 with virt as the entrypoint.

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[slintes]

PR is in review, so:

/remove-lifecycle rotten

[fabiand]

By now several updates took place, see:
https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/

Optionally:
https://github.com/GoogleContainerTools/krew
https://github.com/GoogleContainerTools/krew-index

Take aways:

• Metadata now allows us ot define kubectl virt …
• Optionally: We should prep an archive to get into krew index

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubevirt-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[rmohr]

/remove-lifecycle rotten

[dhiller]

@fabiand IIUC making virtctl accessible as a kubectl plugin would be as easy as writing a thin wrapper script like so:

$ cat ~/bin/kubectl-virt
#!/bin/bash
virtctl "$@"

where one could use it like so

$ kubectl virt help     
virtctl controls virtual machine related operations on your kubernetes cluster.
...

Then virtctl install would boil down to sth like determining the OS and installing a wrapper script depending on the os somewhere beneath the virtctl binary, right?

OTOH if we would create a plugin package using krew, the scope would be delivering the virtctl binary including the wrapper script per OS, right?

[fabiand]

@fabiand IIUC making virtctl accessible as a kubectl plugin would be as easy as writing a thin wrapper script like so:

Yes, that can well be. Some additional metadata might be helpful to support i.e. tab-completion.

Then virtctl install would boil down to sth like determining the OS and installing a wrapper script depending on the os somewhere beneath the virtctl binary, right?

Hm. It depends - It can either be having the wrapper in the "right" path where kubect is looking or writing the correct metadata to let kubectl pick up the binary (not the wrapper) in the path where it resides.

Remark: We probably want to use virtctl plugin install and uninstall because at some point we might need virtctl plug/unplug for device hot-plug.

OTOH if we would create a plugin package using krew, the scope would be delivering the virtctl binary including the wrapper script per OS, right?

Yeah, along those lines (metadata for krew, building right binaries) but here I'd also eye on automationg to open new PRs on krew for new kubevirt releases.

[slintes]

Then virtctl install would boil down to sth like determining the OS and installing a wrapper script depending on the os somewhere beneath the virtctl binary, right?

Hm. It depends - It can either be having the wrapper in the "right" path where kubect is looking or writing the correct metadata to let kubectl pick up the binary (not the wrapper) in the path where it resides.

kubectl looks for everything starting with kubectl-* on the $PATH

OTOH if we would create a plugin package using krew, the scope would be delivering the virtctl binary including the wrapper script per OS, right?

Yeah, along those lines (metadata for krew, building right binaries) but here I'd also eye on automationg to open new PRs on krew for new kubevirt releases.

that would be nice :)

btw, this KubeCon session was exactly about this topic: https://www.youtube.com/watch?v=83ITOTsXsHU

[dhiller]

Yeah, along those lines (metadata for krew, building right binaries) but here I'd also eye on automationg to open new PRs on krew for new kubevirt releases.

I don't understand what you mean? Is there a mechanism in place for that?

[dhiller]

Remark: We probably want to use virtctl plugin install and uninstall because at some point we might need virtctl plug/unplug for device hot-plug.

I don't see any virtctl plugin install ... in the virtctl api. Can you point me into the right direction here, please? Or do you mean kubectl plugin install ... ?

[SchSeba]

It was https://github.com/kubevirt/kubevirt/pull/1361 but I think now the install should be much easier

[dhiller]

Heavy work in progress, so I just wanted to make sure if this goes into the right direction: https://github.com/dhiller/kubectl-virt-plugin

[dhiller]

So this is just a prototype. I found a way to use krew to install the virtctl binary as a plugin that is then usable with

kubectl virt ...

[slintes]

Looks nice on a very quick look :+1:
2 questions:

• is there are reason that you use a separate repository for it?
• and looking into the yaml file, can you use the released binaries from https://github.com/kubevirt/kubevirt/releases/tag/v0.17.0 instead of that new bundle file?

[dhiller]

• is there are reason that you use a separate repository for it?

Yes, because I can't create a release in kubevirt repo. The proposed way is to bundle the binaries with a GitHub release, so I created one in my own repository.

• and looking into the yaml file, can you use the released binaries from https://github.com/kubevirt/kubevirt/releases/tag/v0.17.0 instead of that new bundle file?

I've tried to use the binaries directly, unfortunately this didn't work. The documentation says the bundle must be a .tar.gz. :)

I'm going to create a PR in the kubevirt repository, so that we are ready for the next release. Yet I'm unsure where to put the mainfest.yaml though. Any ideas?

[slintes]

• and looking into the yaml file, can you use the released binaries from https://github.com/kubevirt/kubevirt/releases/tag/v0.17.0 instead of that new bundle file?

I've tried to use the binaries directly, unfortunately this didn't work. The documentation says the bundle must be a .tar.gz. :)

Too bad... found this issue, but it was closed, maybe we can revive it? With a PR? :wink: Not sure if it's worth it though....
https://github.com/kubernetes-sigs/krew/issues/45

I'm going to create a PR in the kubevirt repository, so that we are ready for the next release. Yet I'm unsure where to put the mainfest.yaml though. Any ideas?

Yes, put a virt.yaml.in template in /manifests/release (maybe a subdirectory). Templates are processed by /hack/build-manifests.sh (triggered by make manifests) and /tools/manifest-templator/manifest-templator.go, there you can use the DockerTag template variable for the current version.
The bundle file could be created in /hack/bazel-build.sh.

[slintes]

oh, we also need the sha sum of the bundle file in the manifest... mhhh. That makes it a bit more interesting... maybe you have an idea? :)

[dhiller]

I think that using .tar.gz is fine as we might need to put some other files into the package (i.e. like @fabiand said, auto completion for bash and the like)

I think that for now the way to go is to use a separate repository (i.e. mine) to prepare the package and such. This should go until we've ironed out the kinks, i.e. finished the package preparation and such and have a stable way of generating the release package. If that is done we can integrate it into the kubevirt main repo. To integrate the plugin into the krew directory we need to create a PR against the index plugin folder anyway

BTW I've got nothing against a separate repository within the kubevirt namespace, that would make it a little bit more official.

[dhiller]

PR created: kubernetes-sigs/krew-index#158, see kubectl-virt-plugin for more details. Posted to kubevirt-dev, now waiting for feedback.

[dhiller]

BTW: the feedback from krew-index contained a note[1] that we should address in one of the next builds:

[1] https://github.com/kubernetes-sigs/krew-index/pull/158#issuecomment-499613563

[slintes]

great work :+1:
can you create an issue (or PR :wink: ) for the comment you mentioned please?

[dhiller]

Done, see #2356

[dhiller]

Will have a look at that on ~Monday~Tuesday.

[dhiller]

great work
can you create an issue (or PR ) for the comment you mentioned please?

I started looking on Friday, and will continue working on this if noone else wants to do this. Can I just assign the issue to my self to make clear I am on this?

[dhiller]

I see that only members of the organization can be assigned issues :,(

[dhiller]

BTW the PR has been merged, so we may want to update the install guide with the option of installing via krew. Created #2366 for that.

[cynepco3hahue]

/assign @dhiller

[cynepco3hahue]

@dhiller You can you prow bot for this purpose

[dhiller]

/close

[kubevirt-bot]

@dhiller: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.