Configuration-as-code-plugin: Race condition for dsl job queue() and dsl script security

[benfiedler]

Hi,

I have multiple .yaml files in my CASC_JENKINS_CONFIG folder. One primarily does all the heavy lifting including disabling dsl security, e.g.:

security:
  globaljobdslsecurityconfiguration:
    useScriptSecurity: false

In a separate CASC_JENKINS_CONFIG file I am creating a seed dsl job that pulls in more dsl groovy scripts over git, e.g.:

jobs:
  - script: >
      pipelineJob('dsl-seed-job') {
         definition {
           cps {
             script('''node {
               stage('some_config') {
                  git branch: SOME_BRANCH, credentialsId: SOME_CREDENTIAL_ID, url: SOME_URL
                  jobDsl targets: 'job_dsl/*.groovy'
               ..................................
      }
      queue('dsl-seed-job')

Because this job itself executes wildcard jobDsl scripts, script security must be disabled before it is queued in order for it to run unattended. But from the jenkins log file, it unfortunately consistently queues the job just before the script security parameter is updated, and so it fails to actually run without manual permission / intervention:

Oct 24, 2018 6:41:17 PM javaposse.jobdsl.plugin.JenkinsJobManagement createOrUpdateConfig
INFO: createOrUpdateConfig for dsl-seed-job
Oct 24, 2018 6:41:18 PM hudson.plugins.promoted_builds.integrations.jobdsl.PromotionsExtensionPoint notifyItemCreated
INFO: Creating promotions for dsl-seed-job
Oct 24, 2018 6:41:18 PM io.jenkins.plugins.casc.Attribute setValue
INFO: Setting [e-mail] = false

If I go login into CasC interface within jenkins management after the container has started and hit 'Reload Existing Configuration' without making any yaml changes, the seed job successfully queues.

Is there a way to specify a prerequisite change in CasC so this doesn't happen, or enforce lexicographical filename order for yaml changes? My current workaround is to use a Datetime() delta and cron schedule the seed job to run in 1 minute which gives the config enough time to finish updating, but it feels like a sloppy workaround.

• Jenkins version 2.138.2
• Plugin version 1.11
• docker jenkins/jenkins:lts

Thanks

[jetersen]

Just linking #280 🙈

[nbutkowski]

I'm also seeing this behavior.

[jetersen]

Suggestion, create another job that you queue instead and in this other job you delay the execution of the jobdsl seed job?

[nbutkowski]

Hi Casz, Thank you for the suggestion. I gave it a try and I'm not certain I'm doing it correctly. Either way, I'm not asking for help debugging this, just putting my information in the open in hopes that it helps someone:

# Root level of my casc yaml
jobs:
  - script: >
      freeStyleJob('my-seed-job') {
        displayName('My Seed Job')
        steps {
            customWorkspace('$JOBS_DSL_JENKINS_CONFIG_DIR')
            dsl(['ansibleJob.groovy'], 'DELETE')
        }
      }
  - script: >
      freeStyleJob('test-wait') {
          displayName('Test Wait Load')
          steps {
            shell('sleep 30')
            queue('my-seed-job')
          }
      }
  - script: queue('test-wait')

# Startup Logs

Setting [email protected] = true
Mar 06, 2019 10:15:24 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting [email protected] = jenkins.security.s2m.AdminWhitelistRule@7f4116c3
Mar 06, 2019 10:15:25 AM INFO javaposse.jobdsl.plugin.JenkinsJobManagement createOrUpdateConfig
createOrUpdateConfig for my-seed-job
Mar 06, 2019 10:15:25 AM INFO javaposse.jobdsl.plugin.JenkinsJobManagement createOrUpdateConfig
createOrUpdateConfig for test-wait
Mar 06, 2019 10:15:25 AM INFO javaposse.jobdsl.plugin.JenkinsJobManagement queueJob
Scheduling build of my-seed-job from my-seed-job
Mar 06, 2019 10:15:25 AM INFO javaposse.jobdsl.plugin.JenkinsJobManagement queueJob
Scheduling build of test-wait from test-wait
Mar 06, 2019 10:15:25 AM WARNING io.jenkins.plugins.casc.BaseConfigurator createAttribute
Can't handle class javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration#metaClass: type is abstract but not Describable.
Mar 06, 2019 10:15:25 AM WARNING io.jenkins.plugins.casc.BaseConfigurator createAttribute
Can't handle class javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration#metaClass: type is abstract but not Describable.
Mar 06, 2019 10:15:25 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting [e-mail] = false
Mar 06, 2019 10:15:25 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting jenkins.model.GlobalConfigurationCategory$[e-mail]iguration = javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration@7e54e444

To me it appears that this config is being applied after the 'jobs: block' from above.

security:
  globalJobDslSecurityConfiguration:
    useScriptSecurity: false

If I re-apply the casc config here's what I see:

# Reload log snippet
Mar 06, 2019 10:28:51 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting [email protected] = true
Mar 06, 2019 10:28:51 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting [email protected] = jenkins.security.s2m.AdminWhitelistRule@4d88b13a
Mar 06, 2019 10:28:51 AM INFO javaposse.jobdsl.plugin.JenkinsJobManagement createOrUpdateConfig
createOrUpdateConfig for my-seed-job
Mar 06, 2019 10:28:52 AM INFO javaposse.jobdsl.plugin.JenkinsJobManagement createOrUpdateConfig
createOrUpdateConfig for test-wait
Mar 06, 2019 10:28:52 AM INFO javaposse.jobdsl.plugin.JenkinsJobManagement queueJob
Scheduling build of my-seed-job from my-seed-job
Mar 06, 2019 10:28:52 AM INFO javaposse.jobdsl.plugin.JenkinsJobManagement queueJob
Scheduling build of test-wait from test-wait
Mar 06, 2019 10:28:52 AM WARNING io.jenkins.plugins.casc.BaseConfigurator createAttribute
Can't handle class javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration#metaClass: type is abstract but not Describable.
Mar 06, 2019 10:28:52 AM WARNING io.jenkins.plugins.casc.BaseConfigurator createAttribute
Can't handle class javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration#metaClass: type is abstract but not Describable.
Mar 06, 2019 10:28:52 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting [e-mail] = false
Mar 06, 2019 10:28:52 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting jenkins.model.GlobalConfigurationCategory$[e-mail]iguration = javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration@3a19a17
Mar 06, 2019 10:28:52 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting [email protected] = [email protected]
Mar 06, 2019 10:28:52 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting [email protected] = https://jenkins-sb.os.chnonprod.net
Mar 06, 2019 10:28:52 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting jenkins.model.GlobalConfigurationCategory$[e-mail] = jenkins.model.JenkinsLocationConfiguration@4fad2ee1
Mar 06, 2019 10:28:58 AM INFO javaposse.jobdsl.plugin.JenkinsJobManagement createOrUpdateConfig
createOrUpdateConfig for ansible-job
Mar 06, 2019 10:28:58 AM INFO hudson.model.Run execute
my-seed-job #1 main build action completed: SUCCESS
Mar 06, 2019 10:29:28 AM INFO hudson.model.Run execute
test-wait #1 main build action completed: SUCCESS

[benfiedler]

That looks exactly like the test I was going to run. Curious as to why it schedules both jobs at the same timestamp instead of having a 30 second delay..

[benfiedler]

@nbutkowski If you are interesting in my ugly-as-sin workaround, it is:

jobs:
  - script: >
      // put the job on a cron to run in 1 minute after the rest of the casc configuration finishes loading
      import groovy.time.TimeCategory
      currentDate =  new Date()
      use( TimeCategory ) {
        delay = currentDate + 1.minutes
      }

      pipelineJob('dsl-seed-job') {
        description('Creates scripts/jobs from git DSL scripts')
        triggers {
          cron('' + delay.getMinutes() + ' ' + delay.getHours() + ' ' + delay[Calendar.DAY_OF_MONTH] + ' ' + (delay.getMonth()+1) + ' *' )
        }
        ...............................................................
      }

I would expect a freeStyleJob to work with triggers{} as well.

[jetersen]

There is properly something we can do here to have jobs be the last thing we configure. I would prefer JCasC configurator was moved to job dsl plugin instead of being located here in this plugin.

[jetersen]

I added a commit to a PR that should fix the issue. This ensures that JobDSL configurator is run last, that should properly ensure global security issues.

please take a look @benfiedler and @nbutkowski you could try and use this version of the plugin ahead of time to confirm: https://repo.jenkins-ci.org/incrementals/io/jenkins/configuration-as-code/1.9-rc706.1715987b2c9c/configuration-as-code-1.9-rc706.1715987b2c9c.hpi
This is an incremental hpi build from #674 with my potential fix :smile:

Uploaded as a part of this build: https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fconfiguration-as-code-plugin/detail/PR-674/26/pipeline

[jetersen]

I actually managed to implement a full test, so I am confident that we have a fix for the race condition:
see #780 :muscle:

[jetersen]

@benfiedler, please give v1.9 a spin 😄

[weirdbiz]

@casz, Hi, just found that with JCasC v.1.22 and Job-DSL v.1.74 this issue still exists. I use freestyleJob to create 'Process Job DSL' step. On first clean startup queue() part does not create new build, but if I reload configuration it's OK. I'm running jenkins/jenkins:lts in Docker and have single .yaml JCasC config file, with the following code:

security:
  globalJobDslSecurityConfiguration:
    useScriptSecurity: false

And then:

jobs:
  - script: >
      freeStyleJob('seed-job') {
          displayName('Seed Job')
          scm {
              git {
                  remote {
                      credentials('git_credentials')
                      url('http://<git_url>/DSL-Jobs.git')
                  }
              }
          }
          steps {
              dsl {
                  external('dsl/*.groovy')
                  removeAction('DELETE')
                  ignoreExisting(false)
                  }
               }
      }
  - script: >
      queue('seed-job')

System log:

Jul 05, 2019 10:18:39 AM INFO javaposse.jobdsl.plugin.JenkinsJobManagement createOrUpdateConfig
createOrUpdateConfig for seed-job
Jul 05, 2019 10:18:44 AM INFO javaposse.jobdsl.plugin.JenkinsJobManagement queueJob
Scheduling build of seed-job from seed-job
Jul 05, 2019 10:18:45 AM WARNING io.jenkins.plugins.casc.BaseConfigurator createAttribute
Can't handle class javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration#metaClass: type is abstract but not Describable.
Jul 05, 2019 10:18:45 AM WARNING io.jenkins.plugins.casc.BaseConfigurator createAttribute
Can't handle class javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration#metaClass: type is abstract but not Describable.
Jul 05, 2019 10:18:45 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting [e-mail] = false
Jul 05, 2019 10:18:46 AM INFO io.jenkins.plugins.casc.Attribute setValue
Setting jenkins.model.GlobalConfigurationCategory$[e-mail]iguration = javaposse.jobdsl.plugin.GlobalJobDslSecurityConfiguration@2e8a1554

[jetersen]

1394 may actually finally fix the load order.