Configuration-as-code-plugin: Helm setup refuses to load YAML file from ConfigMap mount

[dbendelman]

B"H

Hi,

I'm trying to run a helm setup based on the example, except I'm using an external ConfigMap outside the helm chart to store CASC configs and mounting them into the Jenkins container.

(The docs in your example suggest making a custom configmap, but the implementation of that isn't so clean in the stable/jenkins chart, and during upgrades it'll require me to monkeypatch any future changes they make to their reference configmap content.)

I'm seeing very strange behaviour: the CASC plugin recognizes the config file on the disk, but doesn't seem to load it. Here's what my http://jenkins/configuration-as-code/ screen shows:

Configuration as Code

Configuration loaded from :

/var/jenkins_home/casc_configs/jenkins.yaml

Last time applied : 1-Jan-1970 12:00:00 UTC AM

Even clicking Reload doesn't work. I entered the container and verified that that file is readable to the Jenkins user:

$ kubectl exec -it jenkins-5b644b574f-vkcsf -- bash
root@jenkins-5b644b574f-vkcsf:/# su - jenkins
jenkins@jenkins-5b644b574f-vkcsf:~$ cd /var/jenkins_home/casc_configs
jenkins@jenkins-5b644b574f-vkcsf:~/casc_configs$ ls -lah
total 32K
drwxrwxrwx.  3 root root 4.0K Aug  6 16:35 .
drwxrwxrwx. 15 root root 4.0K Aug  6 16:38 ..
drwxr-xr-x.  2 root root 4.0K Aug  6 16:35 ..2018_08_06_16_35_45.759292262
lrwxrwxrwx.  1 root root   31 Aug  6 16:35 ..data -> ..2018_08_06_16_35_45.759292262
lrwxrwxrwx.  1 root root   19 Aug  6 16:35 jenkins.yaml -> ..data/jenkins.yaml
jenkins@jenkins-5b644b574f-vkcsf:~/casc_configs$ cat jenkins.yaml 
jenkins:
  systemMessage: "CASC Jenkins is cool"
  numExecutors: 1
jenkins@jenkins-5b644b574f-vkcsf:~/casc_configs$

Any help would be appreciated. Here's my full helm config and jenkins log:

jenkins-casc-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: jenkins-casc-config
data:
  jenkins.yaml: |
    jenkins:
      systemMessage: "CASC Jenkins is cool"
      numExecutors: 1

jenkins-helm-chart-values.yaml

Master:
  ImageTag: 2.121.2
  UseSecurity: false
  ServiceType: ClusterIP
  ServicePort: 80

  # Enable the experimental update center (needed for the configuration-as-code
  # plugin):
  ContainerEnv:
  - name: JENKINS_UC
    value: https://updates.jenkins.io
  - name: JENKINS_UC_EXPERIMENTAL
    value: https://updates.jenkins.io/experimental
  - name: JENKINS_INCREMENTALS_REPO_MIRROR
    value: https://repo.jenkins-ci.org/incrementals
  - name: CASC_JENKINS_CONFIG
    value: /var/jenkins_home/casc_configs/jenkins.yaml
  InitContainerEnv:
  - name: JENKINS_UC
    value: https://updates.jenkins.io
  - name: JENKINS_UC_EXPERIMENTAL
    value: https://updates.jenkins.io/experimental
  - name: JENKINS_INCREMENTALS_REPO_MIRROR
    value: https://repo.jenkins-ci.org/incrementals

  InstallPlugins:
  - kubernetes:1.12.0
  - workflow-aggregator:2.5
  - workflow-job:2.23
  - credentials-binding:1.16
  - git:3.9.1
  - configuration-as-code:0.11-alpha

Persistence:
  Enabled: false

  volumes:
  - name: casc-config
    configMap:
      name: jenkins-casc-config
  mounts:
  - mountPath: /var/jenkins_home/casc_configs
    name: casc-config
    readOnly: true

rbac:
  install: true

jenkins-container.log

Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Aug 06, 2018 4:36:22 PM org.eclipse.jetty.util.log.Log initialized
INFO: Logging initialized @1601ms to org.eclipse.jetty.util.log.JavaUtilLog
Aug 06, 2018 4:36:23 PM winstone.Logger logInternal
INFO: Beginning extraction from war file
Aug 06, 2018 4:36:28 PM org.eclipse.jetty.server.handler.ContextHandler setContextPath
WARNING: Empty contextPath
Aug 06, 2018 4:36:28 PM org.eclipse.jetty.server.Server doStart
INFO: jetty-9.4.z-SNAPSHOT, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
Aug 06, 2018 4:36:29 PM org.eclipse.jetty.webapp.StandardDescriptorProcessor visitServlet
INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
Aug 06, 2018 4:36:29 PM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: DefaultSessionIdManager workerName=node0
Aug 06, 2018 4:36:29 PM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: No SessionScavenger set, using defaults
Aug 06, 2018 4:36:29 PM org.eclipse.jetty.server.session.HouseKeeper startScavenging
INFO: Scavenging every 600000ms
Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Aug 06, 2018 4:36:31 PM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started w.@66971f6b{/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
Aug 06, 2018 4:36:31 PM org.eclipse.jetty.server.AbstractConnector doStart
INFO: Started ServerConnector@770d0ea6{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
Aug 06, 2018 4:36:31 PM org.eclipse.jetty.server.Server doStart
INFO: Started @10329ms
Aug 06, 2018 4:36:31 PM winstone.Logger logInternal
INFO: Winstone Servlet Engine v4.0 running: controlPort=disabled
Aug 06, 2018 4:36:35 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
Aug 06, 2018 4:36:44 PM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
Aug 06, 2018 4:37:22 PM jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
Aug 06, 2018 4:37:22 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
Aug 06, 2018 4:37:33 PM jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
Aug 06, 2018 4:37:34 PM jenkins.model.Jenkins setBuildsAndWorkspacesDir
INFO: Using non default workspaces directories: ${JENKINS_HOME}/workspace/${ITEM_FULLNAME}.
Aug 06, 2018 4:37:34 PM jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
Aug 06, 2018 4:37:34 PM jenkins.util.groovy.GroovyHookScript execute
INFO: Executing /var/jenkins_home/init.groovy.d/tcp-slave-agent-port.groovy
Aug 06, 2018 4:37:34 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Download metadata
Aug 06, 2018 4:37:34 PM hudson.model.AllView migrateLegacyPrimaryAllViewLocalizedName
INFO: JENKINS-38606 detected for AllView in hudson.model.Hudson@122c90e8; renaming view from All to all
Aug 06, 2018 4:37:38 PM jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
Aug 06, 2018 4:37:38 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
--> setting agent port for jnlp
--> setting agent port for jnlp... done
Aug 06, 2018 4:38:12 PM hudson.model.UpdateSite updateData
INFO: Obtained the latest update center data file for UpdateSource default
Aug 06, 2018 4:38:17 PM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
Aug 06, 2018 4:38:17 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Download metadata. 43,171 ms

[dbendelman]

@HerrmannHinz @ndeloof I see you guys are the authors of that Helm example -- any idea why the plugin might "see" the config yaml but won't load it when the config is mounted from a kubernetes ConfigMap?

[HerrmannHinz]

@dbendelman no idea. will try it on my end as well. but that sounds weird.

so, you are saying:

• plugin is loaded fine
• the yaml gets recognised by the plugin fine because in the CASC page it says the yaml is loaded correct, right?
• the applied configuration (systemMessage) will not be applied?

[dbendelman]

@HerrmannHinz yes, exactly.

Maybe it doesn't like that the yaml file is actually a symlink? See the "ls" dir listing in my post above -- that's how kubernetes implements mounting ConfigMap files.

Other things I've tried that didn't work:

• Mount the yaml file outside $JENKINS_HOME.
• Mount it read/write.

[HerrmannHinz]

@dbendelman got it working finaly:

• i have putted it outside of jenkins home as well (not sure if this affecting it actually but...)
• configmap is like yours:

apiVersion: v1
kind: ConfigMap
metadata:
  name: jenkins-casc-config
  namespace: cicd
data:
  jenkins.yaml: |
    jenkins:
      systemMessage: "CASC Jenkins is cool"
      numExecutors: 1

• mount path stays the same like in your example:

  volumes:
  - name: casc-config
    configMap:
      name: jenkins-casc-config
  mounts:
  - mountPath: /var/opt/casc_configs
    name: casc-config
    readOnly: true

• modified the values yaml, check this (referencing the maybe real file?):

- name: CASC_JENKINS_CONFIG
   value: /var/opt/casc_configs/..data/jenkins.yaml
   # mind the `..data/` path. 

• now the message appears: "CASC Jenkins is cool"

really seems to be an issue of the symlink'd file?

here is the log of the jenkins init:

$ k logs -f jenkins-64c6d76b6c-tmfdc
Picked up JAVA_TOOL_OPTIONS:
Running from: /usr/share/jenkins/jenkins.war
webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")
Aug 07, 2018 7:53:10 PM org.eclipse.jetty.util.log.Log initialized
INFO: Logging initialized @618ms to org.eclipse.jetty.util.log.JavaUtilLog
Aug 07, 2018 7:53:10 PM winstone.Logger logInternal
INFO: Beginning extraction from war file
Aug 07, 2018 7:53:12 PM org.eclipse.jetty.server.handler.ContextHandler setContextPath
WARNING: Empty contextPath
Aug 07, 2018 7:53:12 PM org.eclipse.jetty.server.Server doStart
INFO: jetty-9.4.z-SNAPSHOT; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_171-b11
Aug 07, 2018 7:53:12 PM org.eclipse.jetty.webapp.StandardDescriptorProcessor visitServlet
INFO: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
Aug 07, 2018 7:53:12 PM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: DefaultSessionIdManager workerName=node0
Aug 07, 2018 7:53:12 PM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: No SessionScavenger set, using defaults
Aug 07, 2018 7:53:12 PM org.eclipse.jetty.server.session.HouseKeeper startScavenging
INFO: node0 Scavenging every 660000ms
Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Aug 07, 2018 7:53:13 PM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started w.@18cc679e{Jenkins v2.136,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
Aug 07, 2018 7:53:14 PM org.eclipse.jetty.server.AbstractConnector doStart
INFO: Started ServerConnector@1ac85b0c{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
Aug 07, 2018 7:53:14 PM org.eclipse.jetty.server.Server doStart
INFO: Started @4059ms
Aug 07, 2018 7:53:14 PM winstone.Logger logInternal
INFO: Winstone Servlet Engine v4.0 running: controlPort=disabled
Aug 07, 2018 7:53:15 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
Aug 07, 2018 7:53:16 PM hudson.ClassicPluginStrategy createClassJarFromWebInfClasses
WARNING: Created /var/jenkins_home/plugins/backup/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness
Aug 07, 2018 7:53:17 PM hudson.ClassicPluginStrategy createClassJarFromWebInfClasses
WARNING: Created /var/jenkins_home/plugins/oauth-credentials/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness
Aug 07, 2018 7:53:17 PM hudson.ClassicPluginStrategy createClassJarFromWebInfClasses
WARNING: Created /var/jenkins_home/plugins/openid4java/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness
Aug 07, 2018 7:53:17 PM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
Aug 07, 2018 7:53:17 PM org.jvnet.hudson.plugins.backup.BackupPluginImpl loadConfiguration
INFO: Loading configuration...
Aug 07, 2018 7:53:17 PM org.jvnet.hudson.plugins.backup.BackupPluginImpl start
SEVERE: Error loading configuration file : /var/jenkins_home/backup.xml
Aug 07, 2018 7:53:25 PM jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
Aug 07, 2018 7:53:25 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
Aug 07, 2018 7:53:28 PM org.openid4java.server.RealmVerifier setEnforceRpId
WARNING: RP discovery / realm validation disabled;
Aug 07, 2018 7:53:28 PM jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions
Aug 07, 2018 7:53:28 PM jenkins.model.Jenkins setBuildsAndWorkspacesDir
INFO: Using non default workspaces directories: ${JENKINS_HOME}/workspace/${ITEM_FULLNAME}.
Aug 07, 2018 7:53:28 PM org.jenkinsci.plugins.casc.BaseConfigurator configure
INFO: Setting [email protected] = CASC Jenkins is cool
Aug 07, 2018 7:53:28 PM org.jenkinsci.plugins.casc.BaseConfigurator configure
INFO: Setting [email protected] = 1
Aug 07, 2018 7:53:28 PM jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs
Aug 07, 2018 7:53:28 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Started Download metadata
Aug 07, 2018 7:53:28 PM jenkins.util.groovy.GroovyHookScript execute
INFO: Executing /var/jenkins_home/init.groovy.d/tcp-slave-agent-port.groovy
Aug 07, 2018 7:53:28 PM hudson.model.AllView migrateLegacyPrimaryAllViewLocalizedName
INFO: JENKINS-38606 detected for AllView in hudson.model.Hudson@402e9cc0; renaming view from All to all
Aug 07, 2018 7:53:29 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.web.context.support.StaticWebApplicationContext@5414347f: display name [Root WebApplicationContext]; startup date [Tue Aug 07 19:53:29 GMT 2018]; root of context hierarchy
Aug 07, 2018 7:53:29 PM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
INFO: Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@5414347f]: org.springframework.beans.factory.support.DefaultListableBeanFactory@580a6159
Aug 07, 2018 7:53:29 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@580a6159: defining beans [filter,legacy]; root of factory hierarchy
Aug 07, 2018 7:53:30 PM jenkins.InitReactorRunner$1 onAttained
INFO: Completed initialization
Aug 07, 2018 7:53:30 PM jenkins.plugins.slack.SlackNotifier$Migrator onLoaded
INFO: Starting Settings Migration Process
Aug 07, 2018 7:53:30 PM jenkins.plugins.slack.SlackNotifier$Migrator onLoaded
INFO: Completed Settings Migration Process
Aug 07, 2018 7:53:30 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running
Aug 07, 2018 7:53:33 PM hudson.model.UpdateSite updateData
INFO: Obtained the latest update center data file for UpdateSource default
Aug 07, 2018 7:53:34 PM hudson.model.DownloadService$Downloadable load
INFO: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller
Aug 07, 2018 7:53:34 PM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished Download metadata. 5,428 ms
--> setting agent port for jnlp
--> setting agent port for jnlp... done

@ndeloof any idea maybe?

[ndeloof]

thanks for this diagnostic, will try to debug this tomorrow

[dbendelman]

Thanks @HerrmannHinz for the workaround! :+1:

I'll see if I can test it this week.

@ndeloof I appreciate you looking into this bug.

[dbendelman]

@HerrmannHinz I can confirm your tricks worked for me. Looking forward to a proper fix though, so this can work out of the box for new users.

I would also recommend updating the Helm example to show this external ConfigMap pattern instead of suggesting to customize the stable/jenkins chart's ConfigMap, for the reason I mentioned in my OP.

Thanks!

[HerrmannHinz]

@dbendelman mind adding your approach (which is totally valid and which i will follow as well in the future) as PR to the documentation?

[sshepel]

@ndeloof same for me workaround worked... because ..data/jenkins.yaml has following permissions:

-rw-r--r-- 1 root jenkins 17642 Aug 10 19:50 /var/jenkins_config/..data/jenkins.yaml

whereas jenkins.yaml from /var/jenkins_config folder has following:

lrwxrwxrwx  1 root    root      19 Aug 10 13:12 jenkins.yaml -> ..data/jenkins.yaml

As I've mentioned in gitter, it could be related to https://github.com/kubernetes/kubernetes/issues/2630
cc @carlossg

[dbendelman]

@HerrmannHinz done :)

[ndeloof]

I guess I've found the cause for this:
ConfigurationAsCode#configs do check if the configured file is a RegularFile, so won't follow a symlink

[HerrmannHinz]

I guess I've found the cause for this:
ConfigurationAsCode#configs do check if the configured file is a RegularFile, so won't follow a symlink

@ndeloof it works fine if you just point to the directory. then this ..data hack is not necessary.

anyway. fix would be nice. i'm quite sure there will be situations when ppl do symlink in the casc.yaml's

[ndeloof]

for the record: this has been introduced by https://github.com/jenkinsci/configuration-as-code-plugin/commit/eadd066225c5f5dc6ca8e41f9b18393d48845d4e

[eh-am]

Hey @ndeloof, 3fee9fa882fa53736bb7f23d8dffe79980744383 seems to have broken for me, now that symlinks are being used, I get an error about conflicting keys, which I am assuming is the result of symlink + the original file being used.
May be obvious, but I am using:

- name: CASC_JENKINS_CONFIG
  value: /var/jenkins_home/casc_configs

Sorry, I don't have the exact logs in hands now.

[ndeloof]

can't you use value: /var/jenkins_home/casc_configs/jenkins.yaml ?
I don't think we can offer any fix if your CASC_JENKINS_CONFIG do both include the jenkins.yaml symlink + the ..data and ..<timestamp> directories it actually resolves to.
Anyway we feel the need to revisit this code, so any suggestion / feedback from the trenches would help