Jx: No pipeline "retrying after error:Failed to create the folder in jenkins: 500 Server Error" .

[scheung38]

Summary

After jx create cluster aws

"retrying after error:Failed to create the scheung38 folder in jenkins: 500 Server Error" . 
error: failed to create staging environment: after 10 attempts, last error: Failed to create the 
scheung38 folder in jenkins: 500 Server Error

This is similar to issue https://github.com/jenkins-x/jx/issues/1184 but I cannot locate where this line
https://kubernetes.default is to be found from

kubectl edit configmap jenkins

From @rawlingsj

"Try to kubectl edit configmap jenkins search for

https://kubernetes.default

and replace with

https://kubernetes.default.svc"

Steps to reproduce the behavior

jx create cluster aws

Jx version

1.3.79

Kubernetes cluster provider / Version

aws --version
aws-cli/1.15.55 Python/3.6.4 Darwin/17.6.0 botocore/1.10.54

Operating system / Environment

Mac OSX high sierra 10.13.5

Expected behavior

pipelines in Jenkins to be created

Actual behavior

retrying after error:Failed to create the scheung38 folder in jenkins: 500 Server Error

[jstrachan]

the failed to create folder error can happen if there's a bad jenkins token in your ~/.jx/jenkinsAuth.yaml - removing that file (or using jx delete jenkins token) should fix it?

[jstrachan]

relates to #1234

[darthcabs]

I was going through the same error, decided to kill the jenkins pod with kubectl delete pod jenkins-857765b5b7-fhp8k --namespace jx and let its service recreate it for me. When it came back, problem solved.

I suspect it is because early on I decided to click on jenkins console's notifications tab and update all the plugins, and that came back to bite my ankle

[jp-gorman]

Getting same error - details below:

Pushed Git repository to https://github.com/xxxxxxx/ksdemo

retrying after error:Failed to create the xxxxxxx folder in Jenkins: 500 Server Error
retrying after error:Failed to create the xxxxxxx folder in Jenkins: 500 Server Error
retrying after error:Failed to create the xxxxxxx folder in Jenkins: 500 Server Error
retrying after error:Failed to create the xxxxxxx folder in Jenkins: 500 Server Error
retrying after error:Failed to create the xxxxxxx folder in Jenkins: 500 Server Error

retrying after error:Failed to create the xxxxxxx folder in Jenkins: 500 Server Error
retrying after error:Failed to create the xxxxxxx folder in Jenkins: 500 Server Error
retrying after error:Failed to create the xxxxxxx folder in Jenkins: 500 Server Error
retrying after error:Failed to create the xxxxxxx folder in Jenkins: 500 Server Error
error: after 10 attempts, last error: Failed to create the xxxxxxx folder in Jenkins: 500 Server Error
bash-4.4$
bash-4.4$ jx version
NAME VERSION
jx 1.3.620
jenkins x platform 0.0.2940
Kubernetes cluster v1.11.3-gke.18
kubectl v1.10.7
helm client v2.9.1+g20adb27
helm server v2.9.1+g20adb27
git git version 2.10.1
Operating System Mac OS X 10.13.6 build 17G65

I tried removing yaml authentication for git, tried removing the jenkins pod as sugested before, all still has same issue.

[mrkolb]

I made repeated attempts to install JenkinsX on GKE with cloudshell but everytime I tried I ended up with the folder error for Jenkins....

retrying after error:Failed to create the xxxxxxx folder in Jenkins: 500 Server Error

Because of that the jx import dies and cannot create the GIt environments and ultimately deploy the app.

jx --version = 1.3.688

Remediation attempted:
*recreate cluster
*reinstall jx after removing all components in homedir
*create folder complained about manually in Jenkins admin console on cluster.

This same installation worked on AWS eks with local jx installation on MacOS. So close but yet so far.

[jstrachan]

@mrkolb I wonder if you hit this issue: https://jenkins-x.io/faq/issues/#error-creating-jenkins-credential-jenkins-x-chartmuseum-500-server-error

e.g. I wonder what happens if you try rm -rf ~/.jx then trying again inside Google Cloud Shell?

[jstrachan]

here's a FAQ or this issue btw: https://jenkins-x.io/faq/issues/#error-creating-jenkins-credential-jenkins-x-chartmuseum-500-server-error it seems to be a general issue where your ~/.jx/jenkinsAuth.yaml has an old Jenkins token which no longer maps to a token inside the Jenkins server causing install issues setting up new jenkins projects

[mrkolb]

Thanks for the response...
I did try blowing away the .jx directory and reinstalling but the issue persists. I'm beginning to sense a communication issue between cloudshell and the K8s cluster that Jenkins is on. I can hit the console and login but with my meager experience with Jenkins it's hard to tell if something is out of wack.

A auth flowchart would go a long way to debugging this type issue. If you have any specific debugging protocol I can give it a go. Creating cluster on GKE is a snap.

[jstrachan]

you are running jx create cluster gke inside Google Cloud Shell right? In which case its creating a new cluster with a new IP address and setting up ~/.kube/config to point to the new cluster along with then populating ~/.jx/jenkinsAuth.yaml to point to the new jenkins server

[mrkolb]

specifically what I have been doing with Google cloudshell
1) jx create cluster gke --skip-login=true --project-id=jxtest3 --zone=us-east4-a --skip-installation=true
2) jx install --provider=gke --domain=gke.weblaminar.com --default-admin-password xxxxxxxxxx
In the middle of step two I note the cluster IP to my DNS wildcard and insure it's propagated.

Google has a regex constraint for the --name=clustername which I determined later. Doesn't like caps. Other than that it asks for the Jenkins key and I can login to the Jenkins console.

Then I import the app and it dies with folder creation error.
3) jx import --url=https://github.com/mrkolb/pygke.git

[mrkolb]

hmmm.... I added the -n jxtest switch to my jx create cluster gke command and was able to get past the Jenkins create folder error.
jx create cluster gke --skip-login=true --project-id=jxtest3 --zone=us-east4-a -n jxtest --skip-installation=true

When doing the jx import of a github.com repo logs show it fails with a
"Failed to query the Pull Request last commit status for http://github.com/mrkolb......

I checked the webhook on my staging environment and there were no failures on the github.com end. Jenkins just shows failure of job.

[jstrachan]

@mrkolb that sounds like things are working - you usually get that message a few times until the pipeline for the PR and merge to master of the Environment git repos trigger. Did the PR pipeline complete on the staging environment?

[mrkolb]

No unfortunately....
Pipeline hangs and never completes. Looking in Google console I see two workloads fail for exposing jx-staging and jx-production. I'm assuming that the workloads fail due to the pipeline never completing.

Checking the firewall rules things seem ok. I could be missing something as the K8s cluster networking is new to me.

[jstrachan]

whats jx version - there was an issue with exposecontroller configuration in Staging/Production git repositories lately btw. See https://jenkins-x.io/about/status/#5-jan-2019-environment-git-repository-issue

[mrkolb]

jx version 1.3.688

[mrkolb]

thanks! will test this evening EST. Have to ask an ignorant question. Where does one make this edit? In the ~/.jx directory structure?

[jstrachan]

I tried to be clear in https://jenkins-x.io/about/status/#5-jan-2019-environment-git-repository-issue - its the git repository for your Staging environment you need to look at the env/values.yaml file and sure its using expose: and not exposecontroller:

[mrkolb]

James you were clear.....I was a little fuzzy. After updating github repo env/value.yaml I ran jx import but still see the same error.

"Failed to query the Pull Request last commit status for http://github.com/mrkolb......
Thanks for all the support to date!

[mrkolb]

Update....
Was able to successfully install after making DNS changes inside of Google Cloud DNS. Without that the pipeline would fail when I made DNS changes in my own domain DNS.
Applications published successfully. Wooohooo!

[ccojocar]

Closing this issue since there is a follow to improve the UX in #1234