Kubebuilder: Windows release?

[mattfarina]

There are releases for both Mac and Linux (appears to be POSIX based). Would it be possible to cross-compile to Windows and have a Windows release as well?

[droot]

I think cross-compiling Kubebuilder binary itself should be easy because we use GoReleaser to generate the builds, packaging the dependencies for Windows is going to require some more work here. Happy to help if anyone wants to take this up.

[p-bakker]

If you want to run this on Windows, you could also enable the Windows Linux Subsystem on Windows
(10) and install for example Ubuntu from the Windows App store and just install and run the Linux version of Kubebuilder

https://github.com/michaeltreat/Windows-Subsystem-For-Linux-Setup-Guide

[mattfarina]

@p-bakker It's worth noting that ~50% of devs use Windows and most don't use the subsystem for Linux.

[p-bakker]

true, but WSL is fairly new/gaining traction and while Devs use windows, I'd say most deployments are on Linux, so imho devs would be wise to get familiar with WSL and start making it part of their vocabulary as it gives them the opportunity to test more realistically (testing signals on windows for example won't get you far). Heck, even in VS Code you can now run your debug targets by adding a single property to the configuration.

Not saying there's no point in having native support on Windows, just noting there are alternatives which are pretty straight forward imho

[neelesh3710]

WSL does not fully support all development workflows. For example:

Can't run delve debugger on WSL: https://github.com/Microsoft/WSL/issues/2977

There is also issue running kubebuilder tests on WSL because etcd shipped with kubebuilder does not run under WSL.

Would be nice if the native windows version was available.

[alexeldeib]

@droot Could you elaborate on packaging the dependencies for Windows is going to require some more work? Do you mean e.g. etcd/kube-apiserver/kubectl for testing? The only dependency I could find which did not have a windows build available was kube-apiserver.

With the two diffs below, I'm able to build and use Kubebuilder natively on windows. I'm happy to take this task up and do the work necessary to get it into mergeable state.

https://github.com/kubernetes-sigs/controller-tools/pull/179
https://github.com/kubernetes-sigs/kubebuilder/pull/661

[alexeldeib]

For testing on windows, I used a container and volume mounted my dev directory so that I could run kube-apiserver. This is one of the areas that would require more work for a smooth experience, e.g. providing a predefined dev docker container or possibly running kube-apiserver (and ideally other elements) from a docker image during tests.

[droot]

Could you elaborate on packaging the dependencies for Windows is going to require some more work? Do you mean e.g. etcd/kube-apiserver/kubectl for testing? The only dependency I could find which did not have a windows build available was kube-apiserver.

Yes, thats what I meant.

[alexeldeib]

/reopen

[k8s-ci-robot]

@alexeldeib: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alexeldeib]

@droot I marked this as fixed in the PR which was a mistake 😦 could you reopen?

[DirectXMan12]

sorry for the terrifically long delay here

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[DirectXMan12]

/lifecycle frozen

[DirectXMan12]

we still need help for this. we'll want CI too

[chx]

Re https://github.com/kubernetes-sigs/kubebuilder/issues/300#issuecomment-467586524 about WSL, Microsoft gave up on trying to translating syscalls and WSL2 is a very lightweight VM so now truly all Linux programs will work. Just sayin'.

[alexeldeib]

I think it’s reasonable to tell windows users to run e2e tests agains kind, and set envtest to use existing cluster?

I regularly work on windows and linux and use kubebuilder fine with this approach on both. Some random characters in help text but that’s my biggest complaint.

Rest of testing can probably be done in Go and be x-plat?

[Aarthisk]

I am starting off with kubebuilder on windows- is the easiest option to build by myself?

[alexeldeib]

That’s what I do

[DirectXMan12]

It'd be nice to at least have nightly builds using cross-compile, I think (if anyone wants to pitch in) and a test on prow/travis that tries to cross-compile.

[Adirio]

With both issues mentioned in https://github.com/kubernetes-sigs/kubebuilder/issues/300#issuecomment-480667169 merged, is it possible to compile for windows straight away or do I need to change anything?

The mentioned tests issues are with projects build with kubebuilder or with the kubebuilder cmd? The first is not an issue as they will be tested in my CI machines which are non-windows.

[alexeldeib]

You'll be fine. I did a sanity check right now:

git clone [email protected]:kubernetes-sigs/kubebuilder
cd kubebuilder
go build -o kubebuilder.exe ./cmd/
./kubebuilder.exe # works

[alexeldeib]

Elaborating on my original comment, the test issues are related to running e2e tests for controllers built using Kubebuilder which depend on a full Kubernetes cluster. On Linux, you can use the etcd/apiserver binaries and the envtest package to quickly spin up a real backend. On windows, you can't run the apiserver binary, so this doesn't work.

Using kind works well as a backing cluster for envtest on windows. This will let you run full e2e with controllers made using kubebuilder/controller-runtime on your windows machine.

The tests for kubebuilder itself are a separate story.

[Adirio]

I have a running kubernetes on windows (docker-for-windows). Well its actually not windows somehow as it is pulling the linux images from the manifests, but all the kubernetes cluster is accessible with kubectl from the windows command. Shouldn't that be enough for e2e tests?

[alexeldeib]

See https://godoc.org/sigs.k8s.io/controller-runtime/pkg/envtest for requirements, but that will probably work. I can't say without more certainty without trying 😄

[Adirio]

Which version of Golang are you using? I'm asking this to know if I must set go modules as I'm currently using 1.12.x which if I'm not mistaken is the last that doesn't have it turned on by default.

[alexeldeib]

Correct, the above requires go modules (I always enable them by default in my environment)

Barring windows specific issues on this thread, slack channel is probably a better forum for discussion + q/a 🙂

[Aarthisk]

@alexeldeib
you said "Using kind works well as a backing cluster for envtest on windows. This will let you run full e2e with controllers made using kubebuilder/controller-runtime on your windows machine."

So forgive the novice question, but with a backing "real" k8s cluster, this should just work then? The issues you are pointing out are unit testing without a cluster which I can honestly live without.

[Adirio]

I'm finding some issues with the paths inside the generated Makefile for the projects. To achieve compatibility with Windows I guess this generated Makefile (https://github.com/kubernetes-sigs/kubebuilder/blob/master/pkg/scaffold/v2/makefile.go) needs to be either generic or specific for the target OS.

[DirectXMan12]

happy to take fixes to make it windows-compatible (none of the core devs have windows machines, so it's a bit hard for us)