Che: Improve microk8s support

[benoitf]

From chectl created by l0rd: che-incubator/chectl#196

A first PR has been merged to support microk8s but there are some issues to address. In particular

• [ ] Use microk8s.kubectl instead of kubectl. Use kubectl if microk8s.kubectl is not found. Fail if none of them is found
• [ ] chectl should not have the responsibility to enable microk8s. we should simply fail if microk8s is not enabled and provide the command line to enable it
• [ ] when enabling the storage addon chectl should wait until it is enabled or timeout if it's not enabled after 60s

[TheRealChssAddct]

Starting with the refactor in commit b9f6ac7, _chectl server:start -p microk8s_ began failing for me during k8sTasks.testApiTasks with the following:

[13:33:41] Verify Kubernetes API [started]
[13:33:41] Verify Kubernetes API [failed]
[13:33:41] → Failed to connect to Kubernetes API. Error: connect ECONNREFUSED 127.0.0.1:8080

I found a couple of workarounds after poking around and seeing the loadFromDefault code in chectl/node_modules/@kubernetes/client-node/dist/config.js.

Workaround #1
I found I didn't have a $HOME/.kube/config file. This is because microk8s doesn't create one. (I don't have kubectl installed, because microk8s.kubectl.) The microk8s readme mentions how to create this file:

To use MicroK8s with your existing kubectl:
sudo microk8s.kubectl config view --raw > $HOME/.kube/config"

however this gives me this error: "error: write /dev/stdout: permission denied"

This alternative:

sudo microk8s.config > $HOME/.kube/config

gives me this error: "/snap/microk8s/1079/bin/sed: couldn't flush stdout: Permission denied"

What I found actually works is this:

microk8s.config | cat - > $HOME/.kube/config

Update: I found that the errors I mentioned above were somehow due to running the commands in the integrated terminal in vscode. When I run in a regular old external non managed terminal, I do not get the error.

Workaround #2
Guided by this nicja comment I looked in _/var/snap/microk8s/current/args/kubectl_ to find the pointer to the microk8s kubeconfig file (mine is here: _/var/snap/microk8s/current/credentials/client.config_).

So, theoretically another alternative would be to set the KUBECONFIG environment variable.
Note: I haven't actually tried it, I'm asserting this based on the aforementioned loadFromDefaults code.

Suggested Fix
These workarounds shouldn't be necessary. Perhaps the KubeHelper constructor could be modified to something like this:

if (!context) {
   if (flags.platform === 'microk8s') {
      const { stdout } = yield execa('microk8s.config', { timeout: 10000 });
      this.kc.loadFromString(stdout)

however, that can introduce an undesireable wait-time in a constructor, so maybe callers should pass in an already retrieved context to new KubeHelper.
Perhaps the change is as simple as not running the k8sTasks.testApiTasks task when platform = microk8s.

[mentos1386]

I can't makechectl work with microk8s. It's always failing at storage check.

[TheRealChssAddct]

@mentos1386

This is actually this bug (https://github.com/eclipse/che/issues/16408) I previously submitted and then closed, and which I will now reopen.

Re: chectl failing the storage check, it is because chectl uses the original microk8s.status format (some might say 'contract'). I previously submitted a bug here (https://github.com/ubuntu/microk8s/issues/1041) and the microk8s team (@ktsakalozos) very obligingly fixed it, quite quickly (PR here: https://github.com/ubuntu/microk8s/pull/1042), by having the default be the original format, which che uses.

However, on Aug 20, this commit (https://github.com/ubuntu/microk8s/commit/7bb1b251af8aa8ec55dca5582cc213b6c26bc130) and PR (https://github.com/ubuntu/microk8s/pull/1208) changed the default to 'pretty', probably unintentionally.

@balchua did notice it and commented here (https://github.com/ubuntu/microk8s/issues/1127#issuecomment-664309343). I missed it, but @ktsakalozos actually did give me a heads up here (https://github.com/ubuntu/microk8s/issues/1041#issuecomment-678251907)

I'm not happy with the breaking change, but the comment above makes it clear they intend the default to be 'pretty'.

I already changed my own scripts to accommodate this by explicitly setting the --format flag of the microk8s.status call.

This must be fixed on the che side as well.

[TheRealChssAddct]

@mentos1386

fyi, if you are using snaps, this is how you would go back to a version of microk8s that should work with chectl:

snap install microk8s --classic --channel=1.18/stable