Jx: Hook and Tide pods failing with credentials error

[tdcox]

Summary

Both Hook and Tide pods have started failing into CrashLoopBackOff with the same "Bad credentials" error against GitHub.

{"component":"tide","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://developer.github.com/v3\"}","level":"fatal","msg":"Error getting Git client.","time":"2019-03-21T10:15:31Z"}

Steps to reproduce the behavior

Unclear. This server has been running for about three days on preemptible nodes. The fault appears to have started a little while after upgrading the platform install and subsequent to a node being automatically flushed.

Expected behavior

Expected that Hook and Tide would continue to be able to connect to GitHub.

Actual behavior

Both pods seem to have lost the credentials needed to connect to GitHub. Are they unable to persist their config through a restart?

Jx version

The output of jx version is:

NAME               VERSION
jx                 1.3.998
jenkins x platform 0.0.3588
Kubernetes cluster v1.11.7-gke.4
kubectl            v1.13.4
helm client        Client: v2.13.0+g79d0794
git                git version 2.21.0
Operating System   Mac OS X 10.13.6 build 17G4015

Jenkins type

• [ ] Classic Jenkins
• [x] Serverless Jenkins

Kubernetes cluster

jx create cluster gke \
--cluster-name='d27' \
--default-admin-password='xxxxx' \
--environment-git-owner='tdcox' \
--enhanced-apis=true \
--enhanced-scopes=true \
--git-username='tdcox' \
--git-private=false \
--kaniko=true \
--labels='demo=true' \
--machine-type='n1-standard-4' \
--max-num-nodes='3' \
--min-num-nodes='2' \
--no-tiller=true \
--preemptible=true \
--project-id='jx-mar19' \
--prow=true \
--skip-login=true \
--tekton=true \
--vault=false \
--zone='europe-west1-d'

[ccojocar]

@tdcox Can you check the oauth-token secrets? Does still contains the proper token value? It might be also a permissions issue, the hook/tide is fetching the bot name from GitHub API. The token should have permissions for reading the user info.

[akshayks]

@ccojocar I am observing the same error in a serverless (tekton + prow) JenkinsX installation on EKS. I was unsure whether it made sense to create a new issue so I decided to add my observations / questions here. Apologies in advance if this turns to be not appropriate.

{"client":"github","component":"hook","level":"info","msg":"User()","time":"2019-04-23T01:15:50Z"}
{"component":"hook","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://developer.github.com/v3\"}","level":"fatal","msg":"Error getting Git client.","time":"2019-04-23T01:15:50Z"}

My installation consists of an on-premise BitBucket server so I find it strange that hook pods complain about GitHub credentials. Is it possible to configure a serverless Jenkins installation without a GitHub account?

[vfarcic]

I just upgraded the addons and tide is crashing with the same error in logs as what @akshayks described. It seems that the content of the oauth-token was removed during the upgrade.

kubectl describe secret oauth-token

Name:         oauth-token
Namespace:    cd
Labels:       app=jx-prow-prow
              chart=prow-0.0.647
              heritage=Tiller
              jenkins.io/chart-release=jx-prow
              jenkins.io/version=0.0.647
              release=jx-prow
Annotations:  jenkins.io/chart: prow

Type:  Opaque

Data
====
oauth:  0 bytes

I guess that a fast solution would be to recreate the secret, at least until the "real" issue is fixed.

Can someone let me know how I can do that (recreate the secret) if there is a jx command for it, or what should be in that secret if jx does not help with it.

[vfarcic]

I just confirmed that jx upgrade addons sets oauth-token to an empty value.

[vfarcic]

... to be more precise, jx upgrade addon jx-prow is the problem.

[vfarcic]

@ccojocar I believe that this should be changed to urgent. I tried it in GKE, EKS, and AKS clusters and the behavior is always the same. oauth-token becomes empty after executing jx upgrade addon. The only "special" thing I have in the cluster is that jx is not running in the default jx Namespace.

I tested it only in the serverless mode (tekton+prow) and cannot confirm whether the same thing happens with static jx.

[ccojocar]

@vfarcic Are you using vault or without? I will increase the priority.

[vfarcic]

No Vault...

I created one of the clusters I used to test this with the command that follows.

jx create cluster gke \
    --cluster-name jx-rocks \
    --project-id $PROJECT \
    --region us-east1 \
    --machine-type n1-standard-2 \
    --min-num-nodes 1 \
    --max-num-nodes 2 \
    --default-admin-password=admin \
    --default-environment-prefix tekton \
    --git-provider-kind github \
    --namespace cd \
    --prow \
    --tekton \
    --batch-mode

The other two (EKS and AKS) used the same settings.

[beebird]

Same issue, fresh install, no vault...

{"component":"hook","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://developer.github.com/v3\"}","level":"fatal","msg":"Error getting Git client.","time":"2019-06-11T09:32:22Z"}

jx version
NAME               VERSION
jx                 2.0.261
jenkins x platform 2.0.330
Kubernetes cluster v1.13.1
kubectl            v1.14.0
helm client        Client: v2.11.0+g2e55dbe
git                git version 2.21.0

[alexandru-georgescu-hs]

Hi guys, I have the same issue when trying to install with GitHub Enterprise:
https://github.com/jenkins-x/jx/issues/2285#issuecomment-512161530

Any updates on this? or a workaround? 🙏

thanks!

[vfarcic]

This easily breaks people's clusters. If the solution is not around the corner, I'd suggest disabling jx upgrade addons. Better not to be able to try to upgrade addons than to break the whole cluster.

[yuval-yaari]

@vfarcic it also prevents people from installing new clusters, so I'm hoping this issue is actually addressed...

[cabrinha]

Same issue here, EKS installation. Both tide and hook giving the same error:

{"component":"tide","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://developer.github.com/v3\"}","level":"fatal","msg":"Error getting Git client.","time":"2019-07-19T23:07:25Z"}

NAME               VERSION
jx                 2.0.493
jenkins x platform 2.0.861
Kubernetes cluster v1.13.7-eks-c57ff8
kubectl            v1.15.0
helm client        Client: v2.13.1+g618447c
git                git version 2.21.0
Operating System   Mac OS X 10.14.5 build 18F203

I was able to somewhat "solve" the issue by passing my API token and Username as flags with the install command. I had to create a new GitHub token in my browser, then use that token during the install.

[warrenbailey]

As @vfarcic suggested the problem is caused by upgrading prow via jx upgrade addons command. Basically it's not re-setting the oauth token so it ends up as null.

If you use jx upgrade addon prow command, which follows a different code path, then it correctly sets the oauth token correctly.

I'm going to look at a fix now

[warrenbailey]

It's a bit confusing as jx upgrade addon jx-prow is a different command to jx upgrade addon prow and follows the same path as jx upgrade addons

[warrenbailey]

For a workaround, running jx upgrade addon prow will repair the cluster and fix the tide and hook pods

[yuval-yaari]

@warrenbailey just making sure; I get this issue when trying to create a fresh new cluster, not when running any jx upgrade command, and it entirely breaks the installation process.
I'm not sure if these commands are part of jx create cluster...

[warrenbailey]

There seems to be multiple problems raised in this issue. The jx upgrade addon command is one cause of the empty oauth token which was problematic for @vfarcic and I've created a fix for that. The original post from @tdcox suggest jx upgrade platform is also causing the issue.

@yuval-yaari I'll investigate the new cluster issue today. It would be very helpful if you could provide the exact parameters you are passing to `jx create cluster?

Likewise @alexandru-georgescu-hs, @cabrinha, @akshayks and @beebird if you can also provide the commands you used to create a cluster/install jx and confirm if your oauth-token are empty that would be really useful.

[warrenbailey]

I could do with more info on the create cluster problem people are seeing. The exact cluster setup and Jenkins X install options would be great.

I've tried creating a cluster on the latest JX with both the commands provided by @tdcox and @vfarcic and both created a cluster and installed JX without issue.

[vfarcic]

@warrenbailey My problem is not with jx create cluster (that works fine for me) but with jx upgrade addons. The only reason I posted jx create cluster command was as the answer to the question whenther I'm using vault.

[andreabolandrina]

Hi, it looks like I got bitten by this too.

Setup:

• this is an EKS cluster setup with the eksctl utility, not with jenkins-x
• then I installed jenkins-x with the following command:
  jx install --verbose --provider=eks --no-default-environments --git-provider-url "https://bitbucket.org" --git-username "" --git-api-token ""

The hook and tide deployments failed and their current status is "CrashLoopBackOff".
Checking their logs I see errors related to github bad credentials, even though I'm trying to use bitbucket...

The various logs show:

$ kubectl logs tide-548574ff88-tcnhk
{"client":"github","component":"tide","controller":"sync","level":"info","msg":"Throttle(800, 39)","time":"2019-07-25T13:33:15Z"}
{"client":"github","component":"tide","controller":"status-update","level":"info","msg":"Throttle(400, 200)","time":"2019-07-25T13:33:15Z"}
{"client":"github","component":"tide","level":"info","msg":"User()","time":"2019-07-25T13:33:15Z"}
{"component":"tide","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3"}","level":"fatal","msg":"Error getting Git client.","time":"2019-07-25T13:33:16Z"}
$ kubectl logs hook-954dbdf4d-btbxg
{"client":"github","component":"hook","level":"info","msg":"User()","time":"2019-07-25T13:33:21Z"}
{"component":"hook","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3"}","level":"fatal","msg":"Error getting Git client.","time":"2019-07-25T13:33:22Z"}
$ kubectl logs hook-954dbdf4d-dvpjh
{"client":"github","component":"hook","level":"info","msg":"User()","time":"2019-07-25T13:33:22Z"}
{"component":"hook","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3"}","level":"fatal","msg":"Error getting Git client.","time":"2019-07-25T13:33:22Z"}

This is a test cluster so I'm happy to provide more details and/or try things which might potentially break it.
Just let me know if I can help further please.

Thanks

[yuval-yaari]

I'm experiencing a similar issue on GCP, spinning a new GKE cluster, also using BitBucket.
I'll provide more information later as I'm away from my computer...

[warrenbailey]

@warrenbailey My problem is not with jx create cluster (that works fine for me) but with jx upgrade addons. The only reason I posted jx create cluster command was as the answer to the question whenther I'm using vault.

Yes I appreciate that and I've fixed the upgrade add on problem here: https://github.com/jenkins-x/jx/pull/4832

[warrenbailey]

I see the problem here, serverless Jenkins X (i.e Prow/tekton based) only supports Github at the moment so if you try to use it with bitbucket as the provider it will failed.

Bitbucket support for serverless Jenkins X is imminent but unfortunately until that time you will need to use a static master Jenkins.

[vfarcic]

In my case, the issue is with a cluster using GitHub.

[warrenbailey]

Yes the jx upgrade addon issue affects github clusters. The create/install problem is result of using bitbucket with prow, both manifest with the same error.

The original post is to do with jx upgrade platform with preemptible nodes. However I'm not sure if that is now solved, needs more investigation

[cdbkr]

I have the same issues when trying to install JX Serverless on Github Enterprise. Installation goes through but Hook and Tide pods are failing to run due to bad credentials (using Open Github instead). Tried even to remove the GitHub git server but still using that one.
Then, it seems the serverless approach works only with Github (not GHE), so I see Static Jenkins as the only option :/ (I hope I am wrong).
Anyway, I am wondering if there is any workaround or if you plan to have a look at this issue.

[turgayozgur]

The same issue appears with installation on minikube by using bitbucket.

{"client":"github","component":"hook","level":"info","msg":"User()","time":"2019-08-16T08:36:25Z"}
{"component":"hook","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3"}","level":"fatal","msg":"Error getting Git client.","time":"2019-08-16T08:36:26Z"}

[withlin]

Same problem.

[root@localhost ~]# jx version
NAME               VERSION
jx                 2.0.842
jenkins x platform 2.0.1443
Kubernetes cluster v1.15.3
kubectl            v1.16.0
helm client        Client: v2.14.3+g0e7f3b6
git                1.8.3.1
Operating System   CentOS Linux release 7.6.1810 (Core) 

[root@localhost ~]# k logs hook-678fcc8698-lfxv8 
{"client":"github","component":"hook","level":"info","msg":"User()","time":"2019-10-15T07:29:45Z"}
{"component":"hook","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://developer.github.com/v3\"}","level":"fatal","msg":"Error getting Git client.","time":"2019-10-15T07:29:47Z"}



md5-d5e0b019c030ad686c058264316ef9cf



[root@localhost ~]# k logs tide-5957b56bcc-2xlrt 
{"client":"github","component":"tide","controller":"sync","level":"info","msg":"Throttle(800, 39)","time":"2019-10-15T07:30:27Z"}
{"client":"github","component":"tide","controller":"status-update","level":"info","msg":"Throttle(400, 200)","time":"2019-10-15T07:30:27Z"}
{"client":"github","component":"tide","level":"info","msg":"User()","time":"2019-10-15T07:30:28Z"}
{"component":"tide","error":"error getting bot name: fetching bot name from GitHub: status code 401 not one of [200], body: {\"message\":\"Bad credentials\",\"documentation_url\":\"https://developer.github.com/v3\"}","level":"fatal","msg":"Error getting Git client.","time":"2019-10-15T07:30:35Z"}