Docker: Use jenkins.war PluginManager to resolve plugins and dependencies

[ndeloof]

plugin.sh is used to download a set of plugin but does not handle dependencies, and trying to do so will result in re-implementing jenkins' PluginManager.

Would be great to execute the PluginManager outside jenkins so we rely on the exact same codebase. Something like: java -cp jenkins.war hudson.PluginManager --install <some-plugin>

[jglick]

:-1: as explained in JENKINS-30361. New plugin releases can arise between calls to docker build, so the result would be nondeterministic, breaking a fundamental assumption of the Docker model.

What I _would_ be in favor of is a _verification_ step that fails the build if the plugin list you have specified is internally inconsistent:

• plugin A depends on plugin B but B is not listed
• plugin A depends on plugin B in version x but B is listed in version y where x > y
• plugin A depends on core in version x but actual core version is y where x > y

plugins.sh could also attempt to connect to the update site (if online) and print an informational message in case some of your listed plugins have updates available.

[dweomer]

I have "solved" this via a ${JENKINS_HOME}/init.groovy.d/install-plugins-with-dependencies.groovy that looks like:

import hudson.model.*
import hudson.remoting.Future
import jenkins.model.*
import java.util.concurrent.TimeUnit

{ String msg = getClass().protectionDomain.codeSource.location.path ->
    println "--> ${msg}"

    Jenkins.instance.getPluginManager().getPlugins()
    Jenkins.instance.getUpdateCenter().updateAllSites()

    def Map<String,Future<UpdateCenter.UpdateCenterJob>> updateCenterJobs = [:]
    /* plugins */ [
            'authentication-tokens',
            'build-monitor-plugin',
            'build-name-setter',
            'build-with-parameters',
            'cloudbees-folder',
            'credentials',
            'docker-workflow',
            'durable-task',
            'git',
            'github',
            'gravatar',
            'greenballs',
            'groovy',
            'groovy-postbuild',
            'job-dsl',
            'ldap',
            'matrix-auth',
            'matrix-project',
            'parameterized-trigger',
            'pipeline-utility-steps',
            'project-description-setter',
            'rebuild',
            'ssh-agent',
            'ssh-credentials',
            'timestamper',
            'workflow-aggregator',
            'workflow-multibranch'
    ].each { pluginId ->
        try {
            def plugin = Jenkins.instance.getUpdateCenter().getPlugin(pluginId)
            if (!plugin.installed || (!plugin.installed.isPinned() && plugin.installed.hasUpdate())) {
                updateCenterJobs[pluginId] = plugin.deploy(true)
            }
        } catch (Exception x) {
            x.printStackTrace()
        }
    }

    updateCenterJobs.each { entry ->
        entry.getValue().get(5, TimeUnit.MINUTES)
    }

    if (updateCenterJobs.size() > 0) {
        Jenkins.instance.safeRestart()
    }

    println "--> ${msg} ... done"
} ()

As should be obvious, this is a boot-time, and not build-time, solution. What it also fails to address is determinacy in the versions that are brought down from the top level requirements specified. You should be able to specify a plugin with it's version, e.g. workflow-aggregator:1.13 to be more determinant.

UPDATE: specifying plugin-name:version or plugin-name@version doesn't work as expected.

[hypergig]

@jglick is safe to assume jenkins plugins are backwards compatible? If so, then grabbing the latest should be good enough. If not, then you can determine the minimal required version of a dependency pretty easily, then grab those versions. That should be somewhere between what we have now and best case? Circular dependencies would be tricky...

[hypergig]

update

I was able to do this in python using the json from https://updates.jenkins-ci.org/current/update-center.json (thanks @jnbnyc for finding that) but there is some jank.

First, the json that is returned isn't valid and you need to do some stripping. That was a bit annoying.

Second, the json only considers the latest version of any plugin. At first I was grabbing all dependencies and their versions, then from that list only the latest version of any one dependency.

So for the github plugin, it would look like this:

>>> final_plugins = set()
>>> 
>>> get_plugin_dependencies(('github', 'latest'), final_plugins, update_center_post)
>>> 
>>> pprint.pprint(final_plugins)
set([(u'conditional-buildstep', u'1.3.1'),
     (u'credentials', u'1.10'),
     (u'credentials', u'1.21'),
     (u'credentials', u'1.22'),
     (u'credentials', u'1.24'),
     (u'git', u'2.4.0'),
     (u'git-client', u'1.19.0'),
     ('github', 'latest'),
     (u'github-api', u'1.69'),
     (u'icon-shim', u'2.0.2'),
     (u'javadoc', u'1.0'),
     (u'junit', u'1.2'),
     (u'junit', u'1.6'),
     (u'mailer', u'1.16'),
     (u'mailer', u'1.7'),
     (u'mapdb-api', u'1.0.1.0'),
     (u'matrix-project', u'1.4'),
     (u'matrix-project', u'1.6'),
     (u'maven-plugin', u'1.480.3'),
     (u'maven-plugin', u'2.0'),
     (u'parameterized-trigger', u'2.4'),
     (u'plain-credentials', u'1.1'),
     (u'project-inheritance', u'1.5.3'),
     (u'promoted-builds', u'2.10'),
     (u'promoted-builds', u'2.24.1'),
     (u'rebuild', u'1.16'),
     (u'rebuild', u'1.22'),
     (u'run-condition', u'1.0'),
     (u'scm-api', u'0.2'),
     (u'scm-api', u'1.0'),
     (u'script-security', u'1.13'),
     (u'ssh-credentials', u'1.11'),
     (u'ssh-credentials', u'1.6.1'),
     (u'subversion', u'1.38'),
     (u'token-macro', u'1.1'),
     (u'token-macro', u'1.10'),
     (u'token-macro', u'1.11'),
     (u'token-macro', u'1.5.1')])
>>> 
>>> 

As you can see, multiple versions of the same plugin are in this list. But because the update center post json doesn't contain information on any plugin version but the latest, this list is irrelevant. You can't check dependencies for older plugins. I might as well just get latest for all dependencies.

But it is possible and I will clean up this code and create a PR. It will contain a new version of plugin.sh that will call this python process instead, thus resolving all dependencies too. My question is, would the PR even go through? Or would the powers that be reject it because I am using python instead of bash?

[hypergig]

@ndeloof take a look here
https://github.com/hypergig/jenkins-plugin-resolver

[freme]

Thanks @hypergig, that way I can keep a file with only the top-level plugins and do not have to deal with dependencies

#!/usr/bin/python

import sys
sys.path.insert(0, "/path/to/hypergigs/jenkins-plugin-resolver")
from JenkinsPluginResolver.JenkinsPluginResolver import JenkinsPluginResolver
jpr = JenkinsPluginResolver()

with open('plugins-no-deps.txt') as f:
    for line in f:
        jpr.load(line.split(':')[0], line.split(':')[1])

with open('plugins.txt','w') as f:
    for k, v in jpr.dump().items():
        f.write('%s:%s\n' % (k, v.rstrip()))

[ndeloof]

I added an 'install-plugins' script to latest image to offer a comparable
solution for plugins installation.
Le 31 mai 2016 4:01 PM, "freme" [email protected] a écrit :

Thanks @hypergig https://github.com/hypergig, that way I can keep a
file with only the top-level plugins and do not have to deal with
dependencies

!/usr/bin/python

import sys
sys.path.insert(0, "/path/to/hypergigs/jenkins-plugin-resolver")from JenkinsPluginResolver.JenkinsPluginResolver import JenkinsPluginResolver
jpr = JenkinsPluginResolver()
with open('plugins-no-deps.txt') as f:
for line in f:
jpr.load(line.split(':')[0], line.split(':')[1])
with open('plugins.txt','w') as f:
for k, v in jpr.dump().items():
f.write('%s:%s\n' % (k, v.rstrip()))

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jenkinsci/docker/issues/147#issuecomment-222697371,
or mute the thread
https://github.com/notifications/unsubscribe/AAIGlQnnjig3Lg_hpTpU0Aeh-ystd4oSks5qHD8QgaJpZM4F7zV6
.

[hypergig]

Awesome, glad folks are using it!

... I really need to integrate it with the Jenkins container at some point... @ndeloof is that what you are doing? Can you reference this issue in your commits so we can see?

[ndeloof]

see https://github.com/jenkinsci/blueocean-plugin/pull/214/files#diff-3254677a7917c6c01f55212f86c57fbfR12 for sample usage