Jx: Git Source build step failing

Created on 28 Jun 2019 · 13Comments · Source: jenkins-x/jx

Summary

The Git Source build step is failing with the following error using the docker-helm buildpack

getting the log for build [Repository/PR-XXXX] #1 serverless-jenkins stage from build pack and container build-step-git-source-[repository]-pr-xxxx-serverle-lx6f9
{"level":"warn","ts":1561570892.948143,"logger":"fallback-logger","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/master\" is not a valid GitHub commit ID"}
{"level":"error","ts":1561570893.0224252,"logger":"fallback-logger","caller":"git/git.go:35","msg":"Error running git [fetch --depth=1 --recurse-submodules=yes origin 20f64c45b8c26adb157ce8aaaeb90085891db47e]: exit status 128\nfatal: could not read Username for 'https://github.com': No such device or address\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:35\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:88\nmain.main\n\t/workspace/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:36\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
{"level":"error","ts":1561570893.0982769,"logger":"fallback-logger","caller":"git/git.go:35","msg":"Error running git [pull --recurse-submodules=yes origin]: exit status 1\nfatal: could not read Username for 'https://github.com': No such device or address\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:35\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:91\nmain.main\n\t/workspace/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:36\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
{"level":"warn","ts":1561570893.0983508,"logger":"fallback-logger","caller":"git/git.go:92","msg":"Failed to pull origin : exit status 1"}
{"level":"error","ts":1561570893.099357,"logger":"fallback-logger","caller":"git/git.go:35","msg":"Error running git [checkout 20f64c45b8c26adb157ce8aaaeb90085891db47e]: exit status 128\nfatal: reference is not a tree: 20f64c45b8c26adb157ce8aaaeb90085891db47e\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:35\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:94\nmain.main\n\t/workspace/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:36\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
{"level":"fatal","ts":1561570893.0994601,"logger":"fallback-logger","caller":"git-init/main.go:37","msg":"Error fetching git repository: exit status 128","stacktrace":"main.main\n\t/workspace/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:37\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}

Steps to reproduce the behavior

Import project that uses docker-helm buildpack
Build fails on Git Source step

Expected behavior

Git Source step should succeed

Actual behavior

Git Source step fails with error message fatal: could not read Username for 'https://github.com': No such device or address

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.328
jenkins x platform 2.0.708
Kubernetes cluster v1.12.6-eks-d69f1b
kubectl            v1.14.0
helm client        Client: v2.9.1+g20adb27
git                git version 2.16.3
Operating System   Mac OS X 10.14.4 build 18E226

Jenkins type

[x] Next Generation (Tekton + Prow)
[ ] Classic Jenkins
[ ] Serverless Jenkins (JenkinsFileRunner + Prow)

Kubernetes cluster

EKS

Operating system / Environment

MacOS

areeks arevault kinbug prioritimportant-soon

Source

Wizofgoz

Most helpful comment

Hi @Wizofgoz a couple folks at CloudBees are working on overhauling JX EKS support over the net few weeks. This includes addressing issues with Vault support on EKS.

deanesmith on 23 Sep 2019

🎉3

All 13 comments

this is a regression that was recently introduced - its fixed by https://github.com/jenkins-x/jx/pull/4507

once thats merged + released, upgrade jx and either, recreate your cluster or run jx create env which should correctly create your pipeline secret used to run pipelines.

you can verify the workaround via:

jx step verify install

which should show its now got a valid pipeline secret

jstrachan on 1 Jul 2019

ok we've a new jx binary which fixes this issue - 2.0.370 - sorry about the regression!

jstrachan on 1 Jul 2019

@jstrachan I'm still getting the same error after updating and trying first jx create env and then uninstalling and running jx install. Below is some more information:

jx version

NAME               VERSION
jx                 2.0.372
jenkins x platform 2.0.718
Kubernetes cluster v1.12.6-eks-d69f1b
kubectl            v1.14.0
helm client        Client: v2.9.1+g20adb27
git                git version 2.16.3
Operating System   Mac OS X 10.14.4 build 18E226

jx step verify install

No --namespace option specified or $DEPLOY_NAMESPACE environment variable available so defaulting to using namespace jx
2019/07/01 11:44:15 [INFO] verifying the Jenkins X installation in namespace jx
2019/07/01 11:44:15 [INFO] verifying pods
Checking pod statuses
POD                                                                STATUS
crier-749f96fb4d-n88hm                                             Running
deck-696f77d746-9fmdm                                              Running
deck-696f77d746-lwxv5                                              Running
hook-6d9859bb47-stbgb                                              Running
hook-6d9859bb47-xzdxj                                              Running
horologium-6bc57b5f9-5fl54                                         Running
jenkins-x-chartmuseum-75d45b6d7f-dxjdd                             Running
jenkins-x-controllerbuild-6ddb75f89d-lbl2z                         Running
jenkins-x-controllerrole-5b6d489775-xf727                          Running
jenkins-x-controllerteam-6c67c985cd-nzbhc                          Running
jenkins-x-gcactivities-1562004000-dpvjj                            Succeeded
jenkins-x-gcactivities-1562005800-9dl4p                            Succeeded
jenkins-x-gcpods-1562004000-btt6m                                  Succeeded
jenkins-x-gcpods-1562005800-ncppx                                  Succeeded
jenkins-x-gcpreviews-1562004000-4f5pb                              Succeeded
jenkins-x-heapster-7777b7d7d8-zxzkt                                Running
jenkins-x-nexus-6ccd45c57c-rjq57                                   Running
pipeline-5f85b8df5b-fl7tz                                          Running
pipelinerunner-67958fb978-sg6md                                    Running
plank-8849d9d67-4lkht                                              Running
sinker-85ff54bd9b-vwl7b                                            Running
[Repository-Branch]-from-build-pack-g29pv-pod-76a899           Failed
[Repository-PR]-serverle-1-from-build-pack-jq5px-pod-2421cc Failed
[Repository-PR]-serverle-1-from-build-pack-cxjts-pod-dd6a8f Failed
[Repository-PR]-serverle-1-from-build-pack-5f8fc-pod-08707c Failed
tekton-pipelines-controller-687cfbcc89-25zkl                       Running
tekton-pipelines-webhook-7fd7f8cdcc-pxfj5                          Running
tide-5f8fb5964c-9ss2d                                              Running
vault-operator-55885856f8-6x47l                                    Running
2019/07/01 11:44:16 [INFO] verifying the git Secrets
2019/07/01 11:44:16 [INFO] verifying git Secret jx-pipeline-git-github-github
2019/07/01 11:44:16 [INFO] verifying git server GitHub at https://github.com
2019/07/01 11:44:16 [INFO] found 1 organisations in git server https://github.com: [Organization]
2019/07/01 11:44:16 [INFO] git tokens seem to be setup correctly
error: could not find the Secret kaniko-secret in the namespace: jx: secrets "kaniko-secret" not found

jx get build logs

getting the log for build [Repository/PR] #1 serverless-jenkins stage from build pack and container build-step-git-source-sportsit-stack-pr-4295-serverle-jcx4p
{"level":"warn","ts":1562006461.8414652,"logger":"fallback-logger","caller":"logging/config.go:65","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/master\" is not a valid GitHub commit ID"}
{"level":"error","ts":1562006461.9259896,"logger":"fallback-logger","caller":"git/git.go:35","msg":"Error running git [fetch --depth=1 --recurse-submodules=yes origin 56c26c57643380c7382868b8c0b75a9631e28024]: exit status 128\nfatal: could not read Username for 'https://github.com': No such device or address\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:35\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:88\nmain.main\n\t/workspace/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:36\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
{"level":"error","ts":1562006462.0001397,"logger":"fallback-logger","caller":"git/git.go:35","msg":"Error running git [pull --recurse-submodules=yes origin]: exit status 1\nfatal: could not read Username for 'https://github.com': No such device or address\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:35\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:91\nmain.main\n\t/workspace/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:36\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
{"level":"warn","ts":1562006462.0002198,"logger":"fallback-logger","caller":"git/git.go:92","msg":"Failed to pull origin : exit status 1"}
{"level":"error","ts":1562006462.001663,"logger":"fallback-logger","caller":"git/git.go:35","msg":"Error running git [checkout 56c26c57643380c7382868b8c0b75a9631e28024]: exit status 128\nfatal: reference is not a tree: 56c26c57643380c7382868b8c0b75a9631e28024\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:35\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\t/workspace/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:94\nmain.main\n\t/workspace/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:36\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}
{"level":"fatal","ts":1562006462.0017211,"logger":"fallback-logger","caller":"git-init/main.go:37","msg":"Error fetching git repository: exit status 128","stacktrace":"main.main\n\t/workspace/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:37\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}

Wizofgoz on 1 Jul 2019

the git secret issue looks fixed from the output of the jx step verify install - the error about the kaniko-secret turns out to be an overzealous test - we only need that secret when using GKE - that error should go away for you when this PR is merged: https://github.com/jenkins-x/jx/pull/4514

jstrachan on 2 Jul 2019

@jstrachan the builds are still failing with the same error though. Could we reopen this issue?

Wizofgoz on 2 Jul 2019

Running jx step git credentials also seems to output good credentials:

https://[username]:[apiToken]@github.com
http://[username]:[apiToken]@github.com

So I'm at a loss as to why it's not working in the build.

Wizofgoz on 2 Jul 2019

could you upgrade your jx CLI via jx upgrade cli and re-run jx step verify install - it should be good which means the pipeline secret is setup.

The only thing I can think of is the knative-git-user-pass is not populated...

try kubectl get secret -oyaml knative-git-user-pass | ksd if you have ksd on your path and see if thats populated - if not try edit it.

Failing that you could try another jx install?

jstrachan on 3 Jul 2019

New version info:

NAME               VERSION
jx                 2.0.399
jenkins x platform 2.0.744
Kubernetes cluster v1.12.6-eks-d69f1b
kubectl            v1.14.0
helm client        Client: v2.9.1+g20adb27
git                git version 2.16.3
Operating System   Mac OS X 10.14.4 build 18E226

Output of jx step verify install:

No --namespace option specified or $DEPLOY_NAMESPACE environment variable available so defaulting to using namespace jx
verifying the Jenkins X installation in namespace jx
verifying pods
Checking pod statuses
POD                                               STATUS
crier-749f96fb4d-5l29k                            Running
deck-696f77d746-wzx8z                             Running
deck-696f77d746-xfxj7                             Running
hook-6d9859bb47-7wwpr                             Running
hook-6d9859bb47-gczcq                             Running
horologium-6bc57b5f9-mpxtb                        Running
jenkins-x-chartmuseum-75d45b6d7f-vtnrm            Running
jenkins-x-controllerbuild-b767548db-266zf         Running
jenkins-x-controllerrole-749d85678c-l6brt         Running
jenkins-x-controllerteam-64b46fcfd6-87bw5         Running
jenkins-x-gcactivities-1562166000-zdz57           Succeeded
jenkins-x-gcactivities-1562167800-cgn2d           Succeeded
jenkins-x-gcpods-1562166000-b72jj                 Succeeded
jenkins-x-gcpods-1562167800-n9b4w                 Succeeded
jenkins-x-gcpreviews-1562166000-7s4f4             Succeeded
jenkins-x-heapster-7777b7d7d8-kpvvq               Running
jenkins-x-nexus-6ccd45c57c-5r5mv                  Running
jx-vault-dash-us-west-0                           Running
jx-vault-dash-us-west-configurer-5549b4fbfd-zl5nt Running
pipeline-5f85b8df5b-6p9vr                         Running
pipelinerunner-67958fb978-5hfdn                   Running
plank-8849d9d67-tpbwg                             Running
sinker-85ff54bd9b-6sq4c                           Running
tekton-pipelines-controller-687cfbcc89-xqrxr      Running
tekton-pipelines-webhook-7fd7f8cdcc-9rkc2         Running
tide-5f8fb5964c-fdmw6                             Running
vault-operator-55885856f8-7nrfm                   Running
WARNING: the following pods are not Ready:
Running: jx-vault-dash-us-west-0

Waiting 1s for the pods to become Ready...

WARNING: retrying after error: the following pods are not Ready:
Running: jx-vault-dash-us-west-0
POD                                               STATUS
crier-749f96fb4d-5l29k                            Running
deck-696f77d746-wzx8z                             Running
deck-696f77d746-xfxj7                             Running
hook-6d9859bb47-7wwpr                             Running
hook-6d9859bb47-gczcq                             Running
horologium-6bc57b5f9-mpxtb                        Running
jenkins-x-chartmuseum-75d45b6d7f-vtnrm            Running
jenkins-x-controllerbuild-b767548db-266zf         Running
jenkins-x-controllerrole-749d85678c-l6brt         Running
jenkins-x-controllerteam-64b46fcfd6-87bw5         Running
jenkins-x-gcactivities-1562166000-zdz57           Succeeded
jenkins-x-gcactivities-1562167800-cgn2d           Succeeded
jenkins-x-gcpods-1562166000-b72jj                 Succeeded
jenkins-x-gcpods-1562167800-n9b4w                 Succeeded
jenkins-x-gcpreviews-1562166000-7s4f4             Succeeded
jenkins-x-heapster-7777b7d7d8-kpvvq               Running
jenkins-x-nexus-6ccd45c57c-5r5mv                  Running
jx-vault-dash-us-west-0                           Running
jx-vault-dash-us-west-configurer-5549b4fbfd-zl5nt Running
pipeline-5f85b8df5b-6p9vr                         Running
pipelinerunner-67958fb978-5hfdn                   Running
plank-8849d9d67-tpbwg                             Running
sinker-85ff54bd9b-6sq4c                           Running
tekton-pipelines-controller-687cfbcc89-xqrxr      Running
tekton-pipelines-webhook-7fd7f8cdcc-9rkc2         Running
tide-5f8fb5964c-fdmw6                             Running
vault-operator-55885856f8-7nrfm                   Running
error: Timed out after 1s, last error: the following pods are not Ready:
Running: jx-vault-dash-us-west-0

The vault not being ready is a separate issue that has already been reported #3787 for EKS clusters (at least I'm assuming it is).

Output of kubectl get secret -oyaml knative-git-user-pass | ksd seems normal too:

apiVersion: v1
data:
  password: [gitToken]
  username: [gitUsername]
kind: Secret
metadata:
  annotations:
    jenkins.io/chart: tekton
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"password":"[base64EncodedGitToken]","username":"[base64EncodedGitUsername]"},"kind":"Secret","metadata":{"annotations":{"jenkins.io/chart":"tekton","tekton.dev/git-0":"https://github.com"},"labels":{"jenkins.io/chart-release":"tekton","jenkins.io/namespace":"jx","jenkins.io/version":"0.0.39"},"name":"knative-git-user-pass","namespace":"jx"},"type":"kubernetes.io/basic-auth"}
    tekton.dev/git-0: https://github.com
  creationTimestamp: "2019-07-02T18:01:04Z"
  labels:
    jenkins.io/chart-release: tekton
    jenkins.io/namespace: jx
    jenkins.io/version: 0.0.39
  name: knative-git-user-pass
  namespace: jx
  resourceVersion: "7008692"
  selfLink: /api/v1/namespaces/jx/secrets/knative-git-user-pass
  uid: 5bd448fb-9cf3-11e9-ac0a-060422fc6a80
type: kubernetes.io/basic-auth

I'll try install again and report back.

Wizofgoz on 3 Jul 2019

New install still shows the same behavior. Could it be that vault isn't working correctly on EKS? I'm not exactly sure what all the dependencies are for features.

Wizofgoz on 3 Jul 2019

Any updates on this? We're doing a major migration and I'll have to start looking at other options pretty soon. I'd rather not have to since Jenkins X checks all the right boxes.

Wizofgoz on 25 Jul 2019

Hi @Wizofgoz a couple folks at CloudBees are working on overhauling JX EKS support over the net few weeks. This includes addressing issues with Vault support on EKS.

deanesmith on 23 Sep 2019

🎉3

@Wizofgoz I am closing this since now the EKS is supported in boot which is the recommended way to install Jenkins X.