jx install fails using default minikube config on macOS

[bitwiseman]

Description

Tried to install Jenkins X on a new minikube on my Mac. It failed in a way I couldn't make sense of.

I eventually tried deleting my minikube VM and started again. The shortcut jx create cluster minikube worked. It is unclear how the shortcut differs from the running the two separate commands minikube start && jx install, but it definitely does.

Both the README.md and http://jenkins-x.io/getting-started/create-cluster/ suggest using jx install.

Version

1.1.5

Steps

1. $ brew cask install minikube
2. $ minikube start
3. Install and init helm and draft (may require workaround of draft issue with git 2.16.1)
4. $ jx install
   5 Provide username and API key

Expected result

Successful Jenkins X install.

Actual result

Cloning the Jenkins X cloud environments repo to /Users/bitwiseman/.jx/cloud-environments
Counting objects: 377, done.
Compressing objects: 100% (6/6), done.
Total 377 (delta 2), reused 3 (delta 1), pack-reused 370
Generated helm values /Users/bitwiseman/.jx/extraValues.yaml
helm repo add jenkins-x https://chartmuseum.build.cd.jenkins-x.io
"jenkins-x" has been added to your repositories
rm -rf secrets.yaml.dec
helm repo add jenkins-x https://chartmuseum.build.cd.jenkins-x.io
"jenkins-x" has been added to your repositories
helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
...Successfully got an update from the "jenkins-x" chart repository
Update Complete. ⎈ Happy Helming!⎈ 
helm install jenkins-x/jenkins-x-platform --name jenkins-x -f ./myvalues.yaml -f ./secrets.yaml --version 0.0.348 --values=/Users/bitwiseman/.jx/gitSecrets.yaml --values=/Users/bitwiseman/.jx/adminSecrets.yaml --values=/Users/bitwiseman/.jx/extraValues.yaml --namespace=jx --timeout=6000
Error: release jenkins-x failed: roles.rbac.authorization.k8s.io "cleanup" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["get"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["list"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["watch"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["patch"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["create"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["update"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["delete"]} PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["patch"]} PolicyRule{Resources:["configmaps"], APIGroups:[""], Verbs:["update"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["patch"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["update"]} PolicyRule{Resources:["deployments"], APIGroups:["apps"], Verbs:["get"]} PolicyRule{Resources:["deployments"], APIGroups:["apps"], Verbs:["list"]} PolicyRule{Resources:["deployments"], APIGroups:["apps"], Verbs:["watch"]} PolicyRule{Resources:["deployments"], APIGroups:["apps"], Verbs:["patch"]} PolicyRule{Resources:["deployments"], APIGroups:["apps"], Verbs:["update"]}] user=&{system:serviceaccount:kube-system:default 8e1350f0-288b-11e8-a356-080027c37486 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[]
make: *** [install] Error 1
Error: Command failed  make ARGS=--values=/Users/bitwiseman/.jx/gitSecrets.yaml --values=/Users/bitwiseman/.jx/adminSecrets.yaml --values=/Users/bitwiseman/.jx/extraValues.yaml --namespace=jx --timeout=6000 install
error: exit status 2

[kellyjandrews]

You have to have a running instance with rbac enabled. Specifying that in the documentation would be useful as well.

[michaelneale]

minikube seems a very moving target still...

[kellyjandrews]

I think to a lot of kubernetes users, minikube is the first thing they would want to try. Otherwise, they will be required to have an account on gke with enough credit to run everything during testing.

It feels like a real barrier to early adopters, imho.

[rawlingsj]

GKE today is the best experience which is why we recommend it above others. If you are familiar with kubernetes and have a working minikube then jenkins x also works well there too (although webhooks can't work to trigger app and environment jobs).

We are not saying jenkins x will not run on minikube, we are just trying to nudge folks to the easiest way to see the glory of jenkins x :)

FWIW new users to GKE get a $300 credit which is ample to kick the tires.

[rawlingsj]

Updated docs to be consistent with the other create cluster commands. Thanks @bitwiseman

[bitwiseman]

@rawlingsj
Thanks for updating but, please reopen this issue. Updating the docs does not address the root issue that jx install fails minikube.

[rawlingsj]

If the cluster is created with jx then install will work, there is a workaround when starting minikube to work with RBAC so right now the recommended way is to not start minikube yourself and run jx install.

[rawlingsj]

i.e. if you follow the updated docs you wont fall into this situation, so I think this can remain closed as it is a getting started problem which we have resolved via docs.

[bitwiseman]

@rawlingsj @michaelneale

I'm sorry, I'm not trying to be a pain, but this is not just a getting started problem.
If I already have minikube started I get this:

$ jx create cluster minikube
an existing minikube cluster is already running, perhaps use jx install

So, jx itself says to try jx install. Which will then fail.

If the issue is that RBAC must be enabled when running jx install, then jx install should detect the core problem of RBAC not being enabled and report that.

Beyond that the dependency on RBAC needs to be added to the documentation.

[michaelneale]

no problems at all, - it is a pain.

so what is the ideal? It needs to tell you that it can't do it, but should it suggest creating it fresh?

A bit confused (minikube is a challenge, and hyperkit kills my machine so I have no willingness to try it, its just horrible)

[kellyjandrews]

I believe if you follow this page http://jenkins-x.io/getting-started/install-on-cluster/ you will run into the problem.

Can we add a note here to the effect of "if you are running a minikube cluster without rbac enabled, this will not work" since running jx install with a running minikube instance currently will fail?

I understand that $300 credits are a decent amount, but it also assumes that anyone testing us out will then

a) not have used the credits
b) have some familiarity with gcloud in general
c) be comfortable with spinning up a new cluster and then later need to start paying for it

Minikube exists for a reason, and trying things out is exactly that. Once I got through the initial hurdles, it runs great on virtualbox.

[michaelneale]

Yeah I think the desire is to keep things working with minkube, this is fair enough. Good point on credits and so on.

[michaelneale]

Happy to test it out fresh but I am not sure what instructions to follow.

[michaelneale]

@kellyjandrews any chance you can note your steps to get it going nicely on virtualbox? as that may end up being the way to go

[cendrier]

Thank you guys for pointing out the root cause, I was also trying to follow the instruction to setup jenkins-x based on minikube, but failed with the same reason. What I'm doing now is run minikube with RBAC activated and create clusterrolesbindings manually before install jenkins-x with jx command. Right now seems like that everything's working fine.

minikube start --vm-driver hyperkit --memory 4028 --extra-config=apiserver.Authorization.Mode=RBAC
kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

[rawlingsj]

Yeah that’s exactly the steps that jx create cluster minikube takes.

[bitwiseman]

@michaelneale
Right now, the failure results in a state where running the same command again results in a different error (helm complains). Ideally, jx install should detect missing RBAC as early as possible, and point the user to a part of the troubleshooting guide that discusses how to fix it.

Also, yes, I'm using virtual box.

[ryanwild]

I solved the issue by deleting my minikube cluster:

minikube delete;
jx create cluster minikube;
jx install;

Warning minikube delete will destroy any existing kubernetes cluster you have running

[ccojocar]

@bitwiseman I am going to close this issue. The minikube support has in the meantime improved. Please feel free to open another issue if you are still facing the problem. Thanks