Jx: fatal: could not read Username for 'https://github.com': No such device or address

[stevef1uk]

Hi,

Using minikube on Mac set-up just tried to use the GO quickstart:

jx create quickstart -l go

Pipeline gets to make tag stage with the following logged:

[golang-http] Running shell script

• make tag

sed -i -e "s/version:.*/version: 0.0.1/" Chart.yaml

sed -i -e "s/repository: ./repository: 10.106.31.115:5000/*/golang-http/" values.yaml

sed -i -e "s/tag: .*/tag: 0.0.1/" values.yaml

git add --all

git commit -m "release 0.0.1" --allow-empty # if first release then no verion update is performed

[master 711314a] release 0.0.1

5 files changed, 11 insertions(+), 3 deletions(-)

create mode 100644 VERSION

create mode 100644 charts/golang-http@tmp/durable-569ca6bd/jenkins-log.txt

create mode 100755 charts/golang-http@tmp/durable-569ca6bd/script.sh

git tag -fa v0.0.1 -m "Release version 0.0.1"

git push origin v0.0.1

fatal: could not read Username for 'https://github.com': No such device or address

make: * [tag] Error 128

script returned exit code 2

[sauravnz]

Having similar issues but with my kubernetes cluster on AWS

The problem comes up when trying to build a sample application using jx create spring -d web -d actuator

There was no problem earlier when the cluster is getting created, used jx to create the cluster. Both the staging and production environment were created and the respective git repos were also created.

It fails on the third stage in the pipeline after it builds.

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18.911 s
[INFO] Finished at: 2018-04-17T02:28:58+00:00
[INFO] Final Memory: 18M/25M
[INFO] ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // container
[Pipeline] dir
Running in /home/jenkins/workspace/nkinsx-test-demo_master-H4XBLSGKW3XED2FZ2BLWU3K3STQVARK5V6TR2CI3SSZ4MPXKABBA/charts/jenkinsx-test-demo
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] sh
[jenkinsx-test-demo] Running shell script
+ make tag
sed -i -e "s/version:.*/version: 0.0.1/" Chart.yaml
sed -i -e "s/repository: .*/repository: 100.71.214.52:5000\/jenkinsx\/jenkinsx-test-demo/" values.yaml
sed -i -e "s/tag: .*/tag: 0.0.1/" values.yaml
git add --all
git commit -m "release 0.0.1" --allow-empty # if first release then no verion update is performed
[master 97b2a95] release 0.0.1
 7 files changed, 62 insertions(+), 4 deletions(-)
 create mode 100644 VERSION
 create mode 100644 charts/jenkinsx-test-demo@tmp/durable-4a54d2ec/jenkins-log.txt
 create mode 100755 charts/jenkinsx-test-demo@tmp/durable-4a54d2ec/script.sh
 create mode 100644 pom.xml.versionsBackup
git tag -fa v0.0.1 -m "Release version 0.0.1"
git push origin v0.0.1
fatal: could not read Username for 'https://github.com': No such device or address
make: *** [tag] Error 128
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Promote to Environments)
Stage 'Promote to Environments' skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done

[jstrachan]

It looks like the jenkins-git-credentials secret has a missing user for your git user/token - this should have been setup correctly during install.

If you type

kubectl get secret -oyaml jenkins-git-credentials

Then the data value needs to be base64 decoded.

For now you could manually try to edit this value via

kubectl edit secret jenkins-git-credentials

FWIW am working on an approach that dynamically generates this file so it’s easier to use jx to add/remove git servers & tokens

[sauravnz]

SOLVED
Thankyou @jstrachan . The problem was with my configuration and not JenkinsX. I validated this by creating a new account and creating a single organization. To what I understand, I was using the email id as username, where as the username was different for my github account.

Also i validated it with my ~/.jx/gitAuth.yaml configuration

If you can point out as to how these credential are passed to the running container which is pushing it to the github staging repo. IT would be very helpful.

Thanks again.

[jstrachan]

@sauravdevops ah great thanks for the heads up!

We should really add better validation that the credentials provided actually work to help catch issues where folks use their email instead of their user name for github etc.
https://github.com/jenkins-x/jx/issues/647

We're moving to a simpler approach inside Jenkins X of using annotated Secrets for git credentials that should be easier to manage and incrementally change via kubectl or the jx CLI - will add more docs soon...

[stevef1uk]

James,

I trashed everything and tried again this time remembering to set my global git id, as before my sabbatical I had been using my company's enterprise git repo

git config --global user.name 'myid'

I get the same issue as before and as far as I can see the kubectl get secret -oyaml jenkins-git-credentials output decoded looks ok. It is the form of:

https://(myid)(myapitoken)@github.com
http://(myid>

So still blocked :-(

[jstrachan]

The new release is out now so we don't use the jenkins-git-credentials that is setup at install time any more; so hopefully we'll better use the current Secrets for your git provider on the fly

[jstrachan]

@stevef1uk could you try trash your cluster and reinstall one more time? The new configuration should not use the jenkins-git-credentials secret any more now

[stevef1uk]

ok, not got anything better to do right now :-)

[jstrachan]

@stevef1uk yay - good luck!!!

[stevef1uk]

I used jx upgrade platform then minikube delete before jx create cluster minikube
$jx version
NAME VERSION
jx 1.1.30
Jenkins X 0.0.678
Kubernetes v1.9.4
Helm Client v2.8.2+ga802316
Helm Server v2.8.2+ga802316
Kubectl Client v1.7.0
Git git version 2.15.1 (Apple Git-101)

Sadly same error as before on creating go demo :-(

git tag -fa v0.0.1 -m "Release version 0.0.1"
git push origin v0.0.1
fatal: could not read Username for 'https://github.com': No such device or address
make: * [tag] Error 128

[jstrachan]

could you try a recent download of jx please? Starting with 1.2.x

[jstrachan]

BTW in newer jx versions the jx version command now prompts you to upgrade if your jx version is old

[stevef1uk]

Gosh I was so last week :-)

Upgraded and ...

Finished: SUCCESS

[jstrachan]

@stevef1uk yay!!! thanks for letting us know!

[jstrachan]

we're hoping to use CI/CD to keep the CI/CD platform and client upgraded continuously btw ;)

[stevef1uk]

Bother, not sure what was successful as I was using the command line, however, when looking at the web GUI Jenkins job has failed but further along the process:

• jx step changelog --version v0.0.1
  Using batch mode as inside a pipeline
  Generating change log from git ref 03748f1d9caeb700bcc26c0c180987eeba3d7bf6 => fe0d97fc128af0b8e864ad02aea158ea7ca2c8fe
  WARNING: Could not create GitProvide so cannot update the release notes: Could not create Git provider for host https://github.com as no user auths could be found

[stevef1uk]

Just tried from scratch again as I set my email address in git globally to be correct. This is a more interesting error:

• jx step changelog --version v0.0.1
  Using batch mode as inside a pipeline
  Generating change log from git ref 2b29ff9ce99633325f7caf26116924b965d81d4f => 35d60a68244105f6d22adbaaf3a2599623d4287d
  Finding issues in commit messages using git format
  EXITCODE 0panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa80f1d]

goroutine 1 [running]:
github.com/jenkins-x/jx/pkg/gits.(GitHubProvider).UpdateRelease(0xc420141440, 0xc4200e8433, 0x9, 0xc4200e843d, 0xb, 0x7fff4f7d97c4, 0x6, 0xc420323320, 0x47, 0x0)
/home/jenkins/go/src/github.com/jenkins-x/jx/pkg/gits/github.go:521 +0xcd
github.com/jenkins-x/jx/pkg/jx/cmd.(StepChangelogOptions).Run(0xc420146380, 0xc42032fc50, 0x0)
/home/jenkins/go/src/github.com/jenkins-x/jx/pkg/jx/cmd/step_changelog.go:356 +0x25ba
github.com/jenkins-x/jx/pkg/jx/cmd.NewCmdStepChangelog.func1(0xc4201fd900, 0xc4206690e0, 0x0, 0x2)
/home/jenkins/go/src/github.com/jenkins-x/jx/pkg/jx/cmd/step_changelog.go:154 +0x73
github.com/jenkins-x/jx/vendor/github.com/spf13/cobra.(Command).execute(0xc4201fd900, 0xc4206690c0, 0x2, 0x2, 0xc4201fd900, 0xc4206690c0)
/home/jenkins/go/src/github.com/jenkins-x/jx/vendor/github.com/spf13/cobra/command.go:766 +0x2c1
github.com/jenkins-x/jx/vendor/github.com/spf13/cobra.(Command).ExecuteC(0xc4200b8f00, 0xc4202cac80, 0xc4204b6a00, 0xc420366000)
/home/jenkins/go/src/github.com/jenkins-x/jx/vendor/github.com/spf13/cobra/command.go:852 +0x334
github.com/jenkins-x/jx/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200b8f00, 0xc420668420, 0x25c9540)
/home/jenkins/go/src/github.com/jenkins-x/jx/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/jenkins-x/jx/cmd/jx/app.Run(0xc420071f70, 0x1607239)
/home/jenkins/go/src/github.com/jenkins-x/jx/cmd/jx/app/jx.go:18 +0x90
main.main()
/home/jenkins/go/src/github.com/jenkins-x/jx/cmd/jx/jx.go:10 +0x22
[Pipeline] }

[stevef1uk]

Also, when I try the Spring demo I get this error:

jx step git credentials
error: Failed to write to /home/jenkins/git/credentials: open /home/jenkins/git/credentials: read-only file system

[jstrachan]

you sure thats on a new cluster? Whats the output of jx version? Only we've moved away from mounting the jenkins-git-credentials secret at /home/jenkins/git

[stevef1uk]

James, pretty sure but I could have left the local cluster running.

I have just recreated everything. Do I still need to manually delete the repos on GitHub created by jx e.g. environment_something_production? Previously I found I needed to or it wouldn't create a cluster for the 2nd time. This is a tedious task.

$ jx version
NAME VERSION
jx 1.2.2
jenkins x platform 0.0.684
kubernetes cluster v1.9.4
kubectl v1.7.0
helm client v2.8.2+ga802316
helm server v2.8.2+ga802316
git git version 2.15.1 (Apple Git-101)

On creating the Spring demo I still get the message:

• jx step git credentials
  error: Failed to write to /home/jenkins/git/credentials: open /home/jenkins/git/credentials: read-only file system

[rawlingsj]

The new cluster will create new staging and production environment repos using the cluster name, so they should be unique and no need to delete old environment repos.

[jstrachan]

@stevef1uk I wonder if you edit your Jenkinsfile for the line thats got

sh "jx step git credentials"

and replace it with:

sh "sudo jx step git credentials"

it works any better? could be a permissions thing on the home dir

[stevef1uk]

JamesR,

Thanks for that update.

The golang demo has got further now!

I see lots of messages like this:

Using pipeline: stevef1uk/golang-http/master build: #1
WARNING: Failed to query the Pull Request last commit status for https://github.com/stevef1uk/environment-pythonfog-staging/pull/1 ref bcca878c5cfd82d1bc2084a9f7c40206b33effb5 Could not find a status for repository

I clicked on the URL and approved the merge but now I have:

Merge commit has not yet any statuses on repo stevef1uk/environment-pythonfog-staging merge sha 779e365a51863b1a592f47e9daf439c3741cd47d

[rawlingsj]

That PR should have been merged automatically but it can take a few mins for the CI job to run and then the merge happens after all checks have passed. Next time I wonder if you wait a little longer it should merge without you manually doing so.

Similarly the merge to master job can take a min or two to start, so hopefully you'll see something happen shortly.

Another thing to check is if the webhooks are triggering the jenkins jobs, there's some help on an issue here about that https://github.com/jenkins-x/jx/issues/648#issuecomment-381935746.

[stevef1uk]

JamesS,

sudo jx step git credentials
/home/jenkins/workspace/stevef1uk_demo_master-7HNBP7CJ2645ML6H2LFD5TJZD4SIVV57SXT6IBX63UNCVHQZ4C6Q@tmp/durable-c56008ce/script.sh: line 2: sudo: command not found

Not sure how to check if sudo installed in the relevant container, but I would assume not by default as a few security issues there :-)

[jstrachan]

@stevef1uk so which quickstarts/pipelines are working past the jx step git credentials step?

The messages like WARNING: Failed to query the Pull Request last commit status for stevef1uk/environment-pythonfog-staging#1 ref bcca878c5cfd82d1bc2084a9f7c40206b33effb5 Could not find a status for repository are just that its waiting for a PR pipeline to complete and run, then an update to run etc. Often those can take a while if you're cluster is a bit low on reasources/CPUs?

[jstrachan]

@stevef1uk ah I think the /home/jenkins/git/credentials is the same as this issue #680 which am just testing a fix for...

[jstrachan]

I think this is fixed in 0.0.722 of the platform. Lemme know and we can reopen (or create a new issue) if there are still problems

[stevef1uk]

James just tried again with a completely new set-up and the 'jx create quickstart -l go' seems to have an issue:

I used 'jx upgrade cli'

$ jx version
NAME VERSION
jx 1.2.9
jenkins x platform 0.0.722
kubernetes cluster v1.9.4
kubectl v1.7.0
helm client v2.8.2+ga802316
helm server v2.8.2+ga802316
git git version 2.15.1 (Apple Git-101)

My cluster in on minikube and I gave it 12Mb and 3 CPUs so it shouldn't be resource constrained.

Loads of the following messages on the Jenkins GUI. I have only copied a few lines:

stevef1uk/environment-bellyspring-staging with ref 34c7cafcaede0457d7b31ea7cead756bbb97811f
WARNING: Failed to query the Pull Request last commit status for https://github.com/stevef1uk/environment-bellyspring-staging/pull/1 ref 34c7cafcaede0457d7b31ea7cead756bbb97811f Could not find a status for repository stevef1uk/environment-bellyspring-staging with ref 34c7cafcaede0457d7b31ea7cead756bbb97811f
WARNING: Failed to query the Pull Request last commit status for https://github.com/stevef1uk/environment-bellyspring-staging/pull/1 ref 34c7cafcaede0457d7b31ea7cead756bbb97811f Could not find a status for repository stevef1uk/environment-bellyspring-staging with ref 34c7cafcaede0457d7b31ea7cead756bbb97811f
WARNING: Failed to query the Pull Request last commit status for https://github.com/stevef1uk/environment-bellyspring-staging/pull/1 ref 34c7cafcaede0457d7b31ea7cead756bbb97811f Could not find a status for repository stevef1uk/environment-bellyspring-staging with ref 34c7cafcaede0457d7b31ea7cead756bbb97811f
Pull Request https://github.com/stevef1uk/environment-bellyspring-staging/pull/1 is merged at sha 560343c1e8b4af286408617207094d85fb2982db

At this point I manually handled the Pull request and it stuck here:

Merge commit has not yet any statuses on repo stevef1uk/environment-bellyspring-staging merge sha 560343c1e8b4af286408617207094d85fb2982db

for a while before continuing! Finally have something built 😀

merge status: pending for URL https://api.github.com/repos/stevef1uk/environment-bellyspring-staging/statuses/560343c1e8b4af286408617207094d85fb2982db with target: http://unconfigured-jenkins-location/job/stevef1uk/job/environment-bellyspring-staging/job/master/2/display/redirect description: This commit is being built
EXITCODE 0merge status: success for URL https://api.github.com/repos/stevef1uk/environment-bellyspring-staging/statuses/560343c1e8b4af286408617207094d85fb2982db with target: http://unconfigured-jenkins-location/job/stevef1uk/job/environment-bellyspring-staging/job/master/2/display/redirect description: This commit looks good
Merge status checks all passed so the promotion worked!
Application is available at: http://jx-staging-golang-http.jx-staging.192.168.64.19.nip.io
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline

[jstrachan]

Yay! That’s basically working! I tested on minishift yesterday - with GitHub & mini* there is no webhooks so there is a background poll instead - which is very slow - took about 22 minutes to promote!!!

We could maybe try a short timeout on mini* - or see if `jx promote - which knows about the PR & merge to master - could just trigger the builds directly

[jstrachan]

I raised https://github.com/jenkins-x/jx/issues/687

[CobraFlow]

also LGTM. Thanks!

[randymay]

I am on the latest version of jx (1.2.47), and I am seeing this same error. I started seeing it on the previous version as well (1.2.46).

Please note that I can manually execute the git push origin command from the command line in the pod, but this fails in Jenkins.

Any help would be much appreciated.

[jstrachan]

@randymay you mean you are getting the fatal: could not read Username for 'https://github.com': No such device or address error?

We've a pending issue to validate the github API token better before we just assume you entered it correctly https://github.com/jenkins-x/jx/issues/651

Is the failure happening when the pipeline tries to do a git push?

[tdcox]

Now seeing the ‘could not read Username’ error here, on a project that was previously building. Not sure if this is due to a server upgrade or because a different user has taken on the project. Will investigate.

[randymay]

@jstrachan, Yes, that is the error I am seeing, and yes, it is happening on git push.

I have tried re-creating and changing the API token, as well as recreating my cluster, and I am seeing the same result. This project was building properly last week, but it was using a different user name. I deleted all git credentials, as well as git servers, all with no luck.

It's also interesting that I can use kubectl to initiate the git push through the command line directly in the pod, and it works properly.

[tdcox]

In reference to our instance of this error, it was due to a developer cutting and pasting configuration from a very stale example project that was pre-skaffold. Deleting the Jenkinsfile and running jx import again fixed the problem here.

[randymay]

@tdcox, thanks for the suggestion, but it didn't work for me. What is interesting is that the JenkinsFile was recreated, and successfully pushed to GitHub using the user that I was using last week even though I have removed it (using jx delete git token | server).

What can't be pushed is the tag...

[randymay]

I even tried re-importing using the User that was working last week (including using this User for the pipeline), and it still failed at the same point.

[Wizofgoz]

@jstrachan strangely, I just started encountering the fatal: could not read Username for 'https://github.com': No such device or address error for the latest version.

Jenkins X Version:
 NAME               VERSION
jx                 2.0.328
jenkins x platform 2.0.708
Kubernetes cluster v1.12.6-eks-d69f1b
kubectl            v1.14.0
helm client        Client: v2.9.1+g20adb27
git                git version 2.16.3
Operating System   Mac OS X 10.14.4 build 18E226

EKS cluster running Serverless Jenkins, Tekton, and Prow.

I'm guessing that the above approaches no longer apply as I'm not seeing a secret called jenkins-git-credentials.

[martin-patsov]

Just experienced this after executing the steps to reproduce in this issue: https://github.com/jenkins-x/jx/issues/5532