Renovate: Support Jenkinsfile for Docker Tags

[Aghassi]

What would you like Renovate to be able to do?
Jenkins Blue Ocean is a CI system that is becoming a more used use case (especially in enterprise). Ideally, we would like to replicate the current CircleCI support Renovate provides in our Jenkinsfile. By this I mean, if we specify a docker container in the Jenkinsfile, Renovate will do things like pin, issue PRs for updated tags, etc in a standard manner.

Describe the solution you'd like
If I specify a docker container in my Jenkinsfile (either through def or some other means), Renovate should recognize it and handle it as a use case for digests

Describe alternatives you've considered
I have not considered alternatives

Additional context
There is a large push in our organization to move to Jenkins (Blue Ocean plugin + Kubernetes) and we don't want to lose the current benefit we have on CircleCI of having digests handled by Renovate.

[rarkins]

Need to collect examples of Jenkinsfile examples with docker images

[narayanan]

Jenkins Kubernetes plugin provides the ability to declare Pod inside Jenkinsfile for executing the builds. Each Pod specification contains one or more docker container image ids in them. We need the ability to monitor the version changes of these containers and raise a PR when there is a change.

Jenkins Kubernetes plugin exposes multiple methods to declare Pod with in Jenkinsfile. Here are different methods and examples in each method.

• Declarative Pipelines
  
  
  
  • https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/declarative.groovy
  
  
  • https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/declarative-multiple-containers.groovy
  
  
  • https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/kaniko-declarative.groovy
  
  
• Declarative Pipelines W/External Yaml
  
  
  
  • https://github.com/jenkinsci/kubernetes-plugin/tree/master/examples/declarative_from_yaml_file
  
  
• Scripted Pipelines w/Container Template
  
  
  
  • https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/maven-with-cache.groovy
  
  
  • https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/maven.groovy
  
  
• Scripted Pipelines w/Yaml Declaration
  
  
  
  • https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/multi-container.groovy
  
  
  • https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/containerLog.groovy
  
  
  • https://github.com/jenkinsci/kubernetes-plugin/blob/master/examples/selenium.groovy
  
  

More Examples: https://github.com/jenkinsci/kubernetes-plugin/tree/master/examples

[narayanan]

All of our Jenkinsfile's uses the declarative pipeline syntax. So it will be nice if the support for Jenkinsfile declarative pipeline is prioritized.

[viceice]

Jenkins files are written in groovy language

[rarkins]

The challenge with an executable language for package management is that it's not as simple as parsing it with the language itself. A declaration in theory could have if/else statements for example, and only one of those will be executed when we evaluate the code, leaving dependencies in the other logic branch undetected. So in general we're better if we can parse the declarations ourselves, even if it's a little crude.

[viceice]

Because you can declare arbitrary variables, where you can declare your images, i think we can add some special comments to make the parsing more easy. like it is planned for the dockerfiles.

[Aghassi]

@rarkins If the Jenkinsfile uses a shared library with parameters, would that helps?

We currently have it set such that our team just do

#!/usr/bin/env groovy

// Shared library
@Library('shared-libraries') _

sharedFunction {
    tag = "1.2.3"
    image = "docker.enterprise.com/path/to/image"
}

Without compiling anything, is that a "structure" that could be used to have renovate do the lookup and update?

[krohrsb]

Or maybe some scoped variable name to search for like def RENOVATE_DOCKER_IMAGE = '...' or def DOCKER_IMAGE = '...'? Obviously the tricky part is there is no pre-existing standard, so you kinda have to key off of something specific.

[ZyanKLee]

Some rather simple example on how a docker image could be declared without any special cases:

pipeline {
    agent none
    stages {
        stage('JSON lint') {
            agent {
                docker {
                    image 'pipelinecomponents/jsonlint:0.1.4'
                }
            }
            steps {
                sh '''
                    find . \
                        -not -path './.git/*' \
                        -name '*.json' \
                        -type f \
                        -print0 \
                    | parallel \
                        --will-cite -k -0 -n1 \
                            jsonlint -cq
                '''
            }
        }
    }
}

[ZyanKLee]

docker binary might be used in shell scripts inline like such:

pipeline {
    agent none
    stages {
         stage('Docker Build') {
            when {
                beforeAgent true;
                branch 'master'
            }

            agent {
                label 'docker'
            }

            steps {

                withDockerRegistry([
                        url          : 'https://private.registry.invalid',
                        credentialsId: 'private_registry_credentials',
                ]) {
                    sh '''
                        docker pull private.registry.invalid:443/some/image:tag
                    '''
                }
            }
        }
    }
}

[ZyanKLee]

I'd say the easiest first step would be to follow the suggestion of @krohrsb and search for defined variables beginning with the pattern of "DOCKER_IMAGE" which may then be extended by a custom identifier to allow multiple images per Jenkinsfile.
For example:

def DOCKER_IMAGE = "node:12.16.1"

or

def DOCKER_IMAGE_NODE12 = "node:12.16.1"
def DOCKER_IMAGE_NODE13 = "node:13.0.0"

[viceice]

@ZyanKLee I would prefer to a more general way, see #1938

[ZyanKLee]

@viceice I wasn't aware about that idea. It would be even better, indeed.

[rarkins]

We just released "regex" manager support, allowing for custom search patterns within files. Docs: https://docs.renovatebot.com/modules/manager/regex/

[viceice]

here is a sample i use:

  ...
regexManagers: [
    {
      fileMatch: ['^Jenkinsfile$'],
      matchStrings: ['image \'(?<depName>[a-z/-]+):(?<currentValue>[a-z0-9.-]+)@(?<currentDigest>sha256:[a-f0-9]+)\''],
      datasourceTemplate: 'docker',
    },
  ],
  ...

It works for digest updates too. 🎉

[ZyanKLee]

EDIT: fixed the regex

my approach is a little more general (but does not yet support digests):

  regexManagers: [
    {
      fileMatch: ["(^|/)Jenkinsfile.*$"],
      matchStrings: [
        String.raw`def\s+(?<datasource>.+?)_version_(?<depName>.+?)\s*=\s*["'](?<currentValue>[^"'@]+)(?:@(?<currentDigest>sha256:[a-f0-9]+))?["'](?:\s*\/\/\s*(?<lookupName>[^\s]+))?`
      ]
    }
  ]

In Jenkinsfile before the start of the pipeline we can def variables named by a specific scheme and assign them the version number. If needed a comment with a lookupName can be added at the end of the same line.

The name of this variable has to consist of three parts divided by underscores (_): _version_

"datasource" defines what is being versioned here (i.e. docker, npm, maven, etc: https://github.com/renovatebot/renovate/tree/master/lib/datasource)

"depName" defines the object that is to be found in this datasource and to be used in the branch or PR for example.

The optional comment defines an alternative "lookupName" which will be used to find the object in the datasource if it diverts from the "depName". This is useful for example if the object in the datasource has special characters in its name, that may not be used in the variable name (for example docker image renovate/renovate)

Example:

def docker_version_renovate = "19.167.0" // renovate/renovate

[rarkins]

This is really great. I'd like to:

• Start putting these approaches into presets that can be shared/imported instead of copy/pasted
• Create a new set of example/showcase repos that demonstrate the code/config/results

[ZyanKLee]

after we fixed the issue with the regex (I got confused due to the fact that the regex would match the whole file, not line by line), you may now add it to whatever docs and preset you like.

[ZyanKLee]

variant with docker digest:

String.raw`def\s+(?<datasource>.+?)_version_(?<depName>.+?)\s*=\s*["'](?<currentValue>[^"'@]+)(?:@(?<currentDigest>sha256:[a-f0-9]+))?["'](?:\s*\/\/\s*(?<lookupName>[^\s]+))?`