Charts: [stable/jenkins] JCasC doesn't work if master.sidecars.configAutoReload.enabled is set to true

Created on 14 Sep 2019 · 12Comments · Source: helm/charts

Describe the bug
JCasC not loading up while enabling the sidecars.

Version of Helm and Kubernetes:
helm version
Client: &version.Version{SemVer:"v2.14.0", GitCommit:"05811b84a3f93603dd6c2fcfe57944dfa7ab7fd0", GitTreeState:"clean"}

kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T18:56:40Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.10", GitCommit:"37d169313237cb4ceb2cc4bef300f2ae3053c1a2", GitTreeState:"clean", BuildDate:"2019-08-19T10:44:49Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
Jenkins

What happened:
While using values.yaml to enable JCasC. When I set master.sidecars.configAutoReload.enabled : true. JCasC doesn't work. I am deploying it on AKS cluster.

What you expected to happen:
Expected to JCasC to work. Jenkins Server should have come up with welcome message.

How to reproduce it (as minimally and precisely as possible):

Run below command and use below values.yaml.

helm install --name jenkins --namespace jenkins -f values.yaml stable/jenkins

master:
  componentName: "jenkins-master"
  image: "jenkins/jenkins"
  tag: "lts"
  adminUser:  admin
  adminPassword:  asdfg
  version: 0.0.1
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "2000m"
      memory: "4Gi"
  servicePort: 8080
  serviceAnnotations : 
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
  targetPort : 8080
  slaveListenerPort : 50000
  loadBalancerIP: "10.114.121.90"
  serviceType: LoadBalancer

  installPlugins:
    - kubernetes:1.18.1
    - workflow-job:2.33
    - workflow-aggregator:2.6
    - credentials-binding:1.19
    - git:3.11.0

  JCasC:
    enabled: true
    PluginVersion: 1.27
    SupportPluginVersion: 1.18
    configScripts:
      welcome-message: |
        jenkins:
          systemMessage: Price and Promotions Jenkins Server. This Jenkins is configured and managed 'as code'.   

  sidecars:
    configAutoReload:
      enabled: true
      image: shadwell/k8s-sidecar:0.0.2
      imagePullPolicy: IfNotPresent
      resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 50m
            memory: 50Mi


agent:
  enabled: true
  image: $REPOSITORY.azure.io/jenkins-slave
  tag: latest
  alwaysPullImage: false
  podName: jenkins-slave
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "2000m"
      memory: "4Gi"

Output:

NAME:   jenkins
LAST DEPLOYED: Sun Sep 15 02:18:32 2019
NAMESPACE: jenkins
STATUS: DEPLOYED

RESOURCES:
==> v1/ConfigMap
NAME                                    DATA  AGE
jenkins                                 6     1s
jenkins-jenkins-config-welcome-message  1     1s
jenkins-tests                           1     1s

==> v1/Deployment
NAME     READY  UP-TO-DATE  AVAILABLE  AGE
jenkins  0/1    1           0          1s

==> v1/PersistentVolumeClaim
NAME     STATUS   VOLUME   CAPACITY  ACCESS MODES  STORAGECLASS  AGE
jenkins  Pending  default  1s

==> v1/Pod(related)
NAME                      READY  STATUS   RESTARTS  AGE
jenkins-6bc4ddc4d9-8jpp7  0/2    Pending  0         0s

==> v1/Role
NAME                     AGE
jenkins-casc-reload      1s
jenkins-schedule-agents  1s

==> v1/RoleBinding
NAME                      AGE
jenkins-schedule-agents   1s
jenkins-watch-configmaps  1s

==> v1/Secret
NAME     TYPE    DATA  AGE
jenkins  Opaque  3     1s

==> v1/Service
NAME           TYPE          CLUSTER-IP      EXTERNAL-IP  PORT(S)         AGE
jenkins        LoadBalancer  10.114.119.231  <pending>    8080:31100/TCP  1s
jenkins-agent  ClusterIP     10.114.119.234  <none>       50000/TCP       1s

==> v1/ServiceAccount
NAME     SECRETS  AGE
jenkins  1        1s


NOTES:
1. Get your 'admin' user password by running:
  printf $(kubectl get secret --namespace jenkins jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
2. Get the Jenkins URL to visit by running these commands in the same shell:
  NOTE: It may take a few minutes for the LoadBalancer IP to be available.
        You can watch the status of by running 'kubectl get svc --namespace jenkins -w jenkins'
  export SERVICE_IP=$(kubectl get svc --namespace jenkins jenkins --template "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}")
  echo http://$SERVICE_IP:8080/login

3. Login with the password from step 1 and the username: admin

4. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: http:///configuration-as-code and examples: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos

For more information on running Jenkins on Kubernetes, visit:
https://cloud.google.com/solutions/jenkins-on-container-engine
For more information about Jenkins Configuration as Code, visit:
https://jenkins.io/projects/jcasc/

Anything else we need to know:

When I set master.sidecars.configAutoReload.enabled : false. JCasC work as expected.

Source

SUNIL23891YADAV

👍5

Most helpful comment

It's worth mentioning that applying Bozmak's fix means we don't see the aforementioned error in the Init Container logs. We also don't see the ERROR in the jenkins logs. And crucially the JCasC config is applied successfully. If anyone needs it the syntax for the values file is as follows:
```master:
initScripts:
- |
import jenkins.security.*
import hudson.model.User
import jenkins.model.Jenkins
User user = User.getOrCreateByIdOrFullName("admin")
if (user == null) {
System.err.println("ERROR: user 'admin' not found! Can't configure SSH key which is needed to reload JCasC config!")
} else {
String sshKeyString = new File('/var/jenkins_home/key.pub').text
keys_param = new org.jenkinsci.main.modules.cli.auth.ssh.UserPropertyImpl(sshKeyString)
user.addProperty(keys_param)
def inst = Jenkins.getInstance()
def sshDesc = inst.getDescriptor("org.jenkinsci.main.modules.sshd.SSHD")
sshDesc.setPort(1044)
sshDesc.getActualPort()
sshDesc.save()
}

arolls on 11 Oct 2019

🚀1 🎉1 👍1

All 12 comments

We have the same issue. The logs of the sidecar contain the following:
Traceback (most recent call last): File "/app/sidecar.py", line 188, in <module> main() File "/app/sidecar.py", line 183, in main ssh_port) File "/app/sidecar.py", line 120, in watchForChanges jenkinsReloadConfig(admin_private_key, admin_user, ssh_port, logger) File "/app/sidecar.py", line 22, in jenkinsReloadConfig ssh_client.connect('127.0.0.1', port=ssh_port, username=admin_user, pkey=private_key) File "/usr/local/lib/python3.6/site-packages/paramiko/client.py", line 362, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 1044 on 127.0.0.1

kristoflemmens on 17 Sep 2019

In templates jcasc-config.yaml. Below is being mentioned. I am not sure how JCasC loads up when master.sidecars.configAutoReload.enabled is set to false.

{{- if and (.Values.master.JCasC.enabled) (.Values.master.sidecars.configAutoReload.enabled) }}

SUNIL23891YADAV on 17 Sep 2019

I'm facing the same issue, but without errors, after debugging, I found out where the problem came from, but not how to exactly how to fix it:

the init script is executed to early, and User.get fails, the logs show

ERROR: user 'admin' not found! Can't configure SSH key which is needed to reload JCasC config!

Once jenkins is completely up, I run the script that creates the ssh key manually (from jenkins GUI), and it passes.

yelhouti on 20 Sep 2019

I am facing the same problem, and I have the same error as mentioned by @yelhouti.

After debugging, I found that before chart version 1.6.1, the init-add-ssh-key-to-admin.groovy script used User u = User.get("admin") which returns the user "admin". This method is now deprecated.

From chart version 1.6.1 and beyond, the script uses User user = User.get("admin", false) which returns false for some reason.

I could not figure out why that happens.

reference:
https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/User.java

hazim1093 on 20 Sep 2019

@hazim1093 Maybe it is returning null because the user doesn't exist at that point, I'm unsure where the admin user is currently created but maybe this didn't fail in the past because of the side effect that made the function deprecated in the first place.

I created an init script on my values file copying the current script and just changed the User.get("admin", false) to User.getOrCreateByIdOrFullName("admin") which worked.

On further testing, I tried to use the current User.get("admin", false) in a loop to test, it always returned null as long as the script can run, after it finishes Jenkins can finally start correctly and running the command on the script console returns the user correctly.

I don't mind opening a PR for that but it seems that the only solution is to change the function from get to getOrCreateByIdOrFullName which is basically the old get function.

@lachie83, @viglesiasce, @maorfr, @torstenwalter, @mogaal is this solution acceptable?

bosmak on 1 Oct 2019

👍1

@Bosmak Happy to review a PR.

torstenwalter on 2 Oct 2019

Having this same issue using chart 1.7.4 with none of our JCasC based config being set. Noticed in the logs of the Init container:
cp: overwrite '/var/jenkins_home/init.groovy.d/init-add-ssh-key-to-admin.groovy'? cp: overwrite '/var/jenkins_home/jobs/job_creator/config.xml'?

The init container seems to stop at this point. The job_creator job is our DSL job which generates all of our jenkins jobs. This job is defined in our Helm chart values file as the only job to create on installation.

arolls on 11 Oct 2019

👎1 👍1

arolls on 11 Oct 2019

🚀1 🎉1 👍1

@arolls Could you create a PR which fixes this? I am happy to merge. I am just away from my laptop for a couple of days...

torstenwalter on 11 Oct 2019

@torstenwalter I just did it, Sorry for the delay.
Let me known if you need any changes.

@arolls That's exactly the function that I have on my Values until this is solved.
Just curious do you get some warning when this is set?
After I put it there I started to see this warning like 6 times

2019/10/11 14:37:27 Warning: Merging destination map for chart 'jenkins'. The destination item 'initScripts' is a table and ignoring the source 'initScripts' as it has a non-table value of: <nil>

bosmak on 11 Oct 2019

@Bosmak Thanks for the fix!

torstenwalter on 11 Oct 2019

🎉1

@bosmak i dont see that but i do see that same for jcasc section of the values files. Doesn't seem to affect functionality.

Either way, thanks for the fix. Saved me from 2 days of headache. Hopefully other helm jenkins jcasc users see this one.

arolls on 11 Oct 2019

Was this page helpful?

0 / 5 - 0 ratings