Gradle: Add offline instrumentation support to Gradle Jacoco plugin

[bdarwin]

Currently it is not possible to get coverage report for the test classes which use powermock. In order to get the coverage, those classes must be instrumented before the test executes.

Gradle Jacoco plugin doesn't support this natively.

Expected Behavior

Gradle Jacoco plugin should have another parameter to instrument before test

jacoco {
offlineInstrumentation true
}

Current Behavior

Currently Gradle doesn't support such a functionality.

Workaround

task instrument(dependsOn: [classes, project.configurations.jacocoAnt]) { 

        inputs.files classes.outputs.files
        File outputDir = new File(project.buildDir, 'instrumentedClasses')
        outputs.dir outputDir
        doFirst {
            project.delete(outputDir)
            ant.taskdef(
                    resource: 'org/jacoco/ant/antlib.xml',
                    classpath: project.configurations.jacocoAnt.asPath,
                    uri: 'jacoco'
            )
            def instrumented = false
            jacocoOfflineSourceSets.each { sourceSetName ->
                if (file(sourceSets[sourceSetName].output.classesDir).exists()) {
                    def instrumentedClassedDir = "${outputDir}/${sourceSetName}"
                    ant.'jacoco:instrument'(destdir: instrumentedClassedDir) {
                        fileset(dir: sourceSets[sourceSetName].output.classesDir, includes: '**/*.class')
                    }
                    //Replace the classes dir in the test classpath with the instrumented one
                    sourceSets.test.runtimeClasspath -= files(sourceSets[sourceSetName].output.classesDir)
                    sourceSets.test.runtimeClasspath += files(instrumentedClassedDir)
                    instrumented = true
                }
            }
            if (instrumented) {
                //Disable class verification based on https://github.com/jayway/powermock/issues/375
                test.jvmArgs += '-noverify'
            }
        }
    }
    test.dependsOn instrument

[bmuschko]

Just wondering: Couldn't you apply the JaCoCo plugin and add test.jvmArgs += '-noverify' to any project that needs it?

[Ihsahn]

There are other scenarios where we would like to use offline instrumentation (eg my team reports that Jacoco has serious cpu and time impact on running integration tests, so I would rather instrument and run coverage report task on demand, not on each run)

[softcoder]

Any action on this? This is a big headache for us

[voznenkoD]

any update on that?

[aleksandarsusnjar]

Just another vote on this... To add a bit info this is the symptom:

[ant:jacocoReport] Classes in bundle '...' do no match with execution data. For report generation the same class files must be used as at runtime.
[ant:jacocoReport] Execution data for class ... does not match.

[Stvad]

Any updates or plans regarding this one?

[yaroslavbaymak]

Do you have any full working examples with Gradle project using offline instrumentation?

[Johnlon]

Wasn't able to make the workaround here work, however the example provided by Tschasmine here worked for me ok.
https://github.com/esdk/g30l0/commit/82af4c9aad50aadc40d940471fe1b934473170c7

+1 please please give us a working featuree in the plugin.

[Johnlon]

Excuse my ignorance but with offline instrumentation is there any risk that instrumented classes find their way into prod?

[ChaminW]

Any update on this issue?

[afrancoc2000]

Hi, can you give a little more details on how to make this work? I tried the workaround but the tests created with powermock aren't showing in the jacoco report yet, and I get the ant errors that @aleksandarsusnjar talks about.

This is what I'm doing:

1. I added the instrumentation task from here: instrumentation task.

2. I configured the destination of my reports in the test task to send to sonar, like this:

test{
    jacoco{
        append = false
    destinationFile = file("jacoco/jacocoTest.exec")
    }
    reports.junitXml.destination = file("jacoco/test-results")  
}

1. I configured a dependency on tests for the jacoco test report, like this:

jacocoTestReport.dependsOn(test)

1. I ran the jacocoTestReport task:

gradlew jacocoTestReport

I'm getting this logs:

> Task :domain:jacocoTestReport
[ant:jacocoReport] Classes in bundle ... do no match with execution data. For report generation the same class files must be used as at runtime.
[ant:jacocoReport] Execution data for class ... does not match.

This is the class that I'm testing with powermock and the report isn't showing the coverage in this class.

What am I doing wrong?

Thanks

[Vampire]

Here a work-around that allows to pick single classes for offline instrumentation, the rest is done as usual, especially as the on-the-fly instrumentation is the recommended way as per JaCoCo documentation:

val jacoco by configurations.registering
dependencies {
    "jacoco"("org.jacoco:org.jacoco.ant:0.8.4")
}
tasks.test {
    val main by sourceSets
    @Suppress("UnstableApiUsage")
    val mainJavaOutputDir = main.java.outputDir
    val instrumentedDirectory = layout.buildDirectory.dir("$mainJavaOutputDir-instrumented").get()
    classpath = layout.files(instrumentedDirectory.asFile, classpath)

    val preinstrumentedClasses = listOf(
            "Test"
    )
    @Suppress("UnstableApiUsage")
    extensions.configure<JacocoTaskExtension> {
        excludes = preinstrumentedClasses
    }

    doFirst {
        ant.withGroovyBuilder {
            "taskdef"(
                    "name" to "instrument",
                    "classname" to "org.jacoco.ant.InstrumentTask",
                    "classpath" to jacoco.get().asPath)
            instrumentedDirectory.asFile.deleteRecursively()
            "instrument"("destDir" to instrumentedDirectory.asFile) {
                "fileset"("dir" to mainJavaOutputDir) {
                    preinstrumentedClasses.onEach {
                        "include"("name" to "${it.replace('.', '/')}.class")
                    }
                }
            }
        }
    }
}

[AureaMohammadAlavi]

I found a script on StackOverflow that works with gradle 4.x and I tweaked it to make it work with gradle 5.x. I tested this with gradle 6.0.1 and it worked perfectly.
To generate jacoco report you just need to run ./gradlew report.

configurations {
    jacoco
    jacocoRuntime
}

dependencies {
    jacoco group: 'org.jacoco', name: 'org.jacoco.ant', version: '0.8.4', classifier: 'nodeps'
    jacocoRuntime group: 'org.jacoco', name: 'org.jacoco.agent', version: '0.8.4', classifier: 'runtime'
}


task instrument(dependsOn: ['classes']) {
    ext.outputDir = buildDir.path + '/classes-instrumented'
    doLast {
        print "sourceSets.main.output.classesDirs: ${sourceSets.main.output.classesDirs}"
        ant.taskdef(name: 'instrument',
                classname: 'org.jacoco.ant.InstrumentTask',
                classpath: configurations.jacoco.asPath)
        ant.instrument(destdir: outputDir) {
            sourceSets.main.output.classesDirs.each { fileset(dir: it) }
        }
    }
}

gradle.taskGraph.whenReady { graph ->
    if (graph.hasTask(instrument)) {
        tasks.withType(Test) {
            doFirst {
                systemProperty 'jacoco-agent.destfile', buildDir.path + '/jacoco/tests.exec'
                classpath = files(instrument.outputDir) + classpath + configurations.jacocoRuntime
            }
        }
    }
}

task report(dependsOn: ['instrument', 'test']) {
    doLast {
        ant.taskdef(name: 'report',
                classname: 'org.jacoco.ant.ReportTask',
                classpath: configurations.jacoco.asPath)
        ant.report() {
            executiondata {
                ant.file(file: buildDir.path + '/jacoco/tests.exec')
            }
            structure(name: 'Example') {
                classfiles {
                    sourceSets.main.output.classesDirs.each { fileset(dir: it) }
                }
                sourcefiles {
                    fileset(dir: 'src/main/java')
                    //uncomment this if you use groovy
                    //fileset(dir: 'src/main/groovy')
                }
            }
            html(destdir: buildDir.path + '/reports/jacoco')
        }
    }
}

[dmonizer]

@AureaMohammadAlavi :

Could not find method jacoco() for arguments [{group=org.jacoco, name=org.jacoco.ant, version=0.8.4, classifier=nodeps}] on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

could you maybe publish your whole build.gradle (somewhere) ? I'd very much would like to get it to work.

(btw, offtopic: what are the chances to search for a solution of gradle jacoco offline, and get to a solution posted 23 minutes ago? :) )

[AureaMohammadAlavi]

@dmonizer you must count yourself lucky :)

I think you need to add mavenCentral()or jcenter() to the list of repositories. I even tested this with gradle 4.x and it worked without any problem.

Here is my whole build.gradle file:

plugins {
  id 'java'
}

repositories {
  mavenCentral()
}


ext {
    powerMockVersion = "2.0.2"
}

dependencies {
  testImplementation 'junit:junit:4.12'
  testImplementation 'org.assertj:assertj-core:3.13.2'
  testImplementation "org.powermock:powermock-module-junit4:${powerMockVersion}"
  testImplementation "org.powermock:powermock-core:${powerMockVersion}"
  testImplementation "org.powermock:powermock-api-mockito2:${powerMockVersion}"
}

configurations {
    jacoco
    jacocoRuntime
}

dependencies {
    jacoco group: 'org.jacoco', name: 'org.jacoco.ant', version: '0.8.4', classifier: 'nodeps'
    jacocoRuntime group: 'org.jacoco', name: 'org.jacoco.agent', version: '0.8.4', classifier: 'runtime'
}


task instrument(dependsOn: ['classes']) {
    ext.outputDir = buildDir.path + '/classes-instrumented'
    doLast {
        ant.taskdef(name: 'instrument',
                classname: 'org.jacoco.ant.InstrumentTask',
                classpath: configurations.jacoco.asPath)
        ant.instrument(destdir: outputDir) {
            sourceSets.main.output.classesDirs.each { fileset(dir: it) }
        }
    }
}

gradle.taskGraph.whenReady { graph ->
    if (graph.hasTask(instrument)) {
        tasks.withType(Test) {
            doFirst {
                systemProperty 'jacoco-agent.destfile', buildDir.path + '/jacoco/tests.exec'
                classpath = files(instrument.outputDir) + classpath + configurations.jacocoRuntime
            }
        }
    }
}

task report(dependsOn: ['instrument', 'test']) {
    doLast {
        ant.taskdef(name: 'report',
                classname: 'org.jacoco.ant.ReportTask',
                classpath: configurations.jacoco.asPath)
        ant.report() {
            executiondata {
                ant.file(file: buildDir.path + '/jacoco/tests.exec')
            }
            structure(name: 'Example') {
                classfiles {
                    sourceSets.main.output.classesDirs.each { fileset(dir: it) }
                }
                sourcefiles {
                    fileset(dir: 'src/main/java')
                    //uncomment this if you use groovy
                    //fileset(dir: 'src/main/groovy')
                }
            }
            html(destdir: buildDir.path + '/reports/jacoco')
        }
    }
}

[Vampire]

Doing it is much easier actually, no custom report task necessary or similar, but still should be supported natively.
Here is a way in Kotlin DSL to instrument only the specific classes that need pre-instrumentation due to incompatible Powermock usage:

val jacoco by configurations.registering

dependencies {
    jacoco("org.jacoco:org.jacoco.ant:0.8.4")
}

tasks.test {
    val main by sourceSets
    val mainJavaOutputDir = main.java.outputDir
    val instrumentedDirectory = layout.buildDirectory.dir("$mainJavaOutputDir-instrumented").get()
    classpath = layout.files(instrumentedDirectory.asFile, classpath)

    val preinstrumentedClasses = listOf(
            "Test"
    )
    extensions.configure<JacocoTaskExtension> {
        excludes = preinstrumentedClasses
    }

    doFirst {
        ant.withGroovyBuilder {
            "taskdef"(
                    "name" to "instrument",
                    "classname" to "org.jacoco.ant.InstrumentTask",
                    "classpath" to jacoco.get().asPath)
            instrumentedDirectory.asFile.deleteRecursively()
            "instrument"("destDir" to instrumentedDirectory.asFile) {
                "fileset"("dir" to mainJavaOutputDir) {
                    preinstrumentedClasses.onEach {
                        "include"("name" to "${it.replace('.', '/')}.class")
                    }
                }
            }
        }
    }
}

[AureaMohammadAlavi]

@dmonizer:
Here is a sample project that uses jacoco offline instrumentation to measure test coverage for tests that use powermock:
https://github.com/AureaMohammadAlavi/gradle-jacoo-offline-instrumentation

[igor-dmitriev]

@bdarwin jacocoOfflineSourceSets is not defined, please share the variable value

[mkdev-cloud]

@dmonizer:
Here is a sample project that uses jacoco offline instrumentation to measure test coverage for tests that use powermock:
https://github.com/AureaMohammadAlavi/gradle-jacoo-offline-instrumentation

Hi
I have clone and ran the a project in my local and all test cases are passed but could not see the code coverage result on the build folder and only able to see the test folder under reports folder.
Also not able see any below folders under build/report folder
classes-instrumented
jacoco
Kindly help me to generate the jacoco code coverage for powermock test cases using gradle build.

[SurpSG]

Hi @mkdev-cloud
Check this one example
https://github.com/SurpSG/jacoco-offline-instrumentation

[AureaMohammadAlavi]

Hi @mkdev-cloud
You need to run ./gradlew report first and then you'll find the html report in build/reports/jacoco/ directory.

[mkdev-cloud]

Thanks for the response, it's working now.

Power mock taking more time me, once class taking more than 30 minutes to
run the test class which are having only 2 or test methods and entire have
up to 50 lines and it's taking more time each test classes. Kindly let me
know, how to resolve this issue.

On Sun, May 10, 2020, 6:59 PM Mohammad Alavi [e-mail]
wrote:

Hi @mkdev-cloud https://github.com/mkdev-cloud
You need to run ./gradlew report first and then you'll find the html
report in build/reports/jacoco/ directory.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gradle/gradle/issues/2429#issuecomment-626328280, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ALF6JYXLEGP3J56PCFD2MFDRQ2T4PANCNFSM4DSKYCRQ
.

[AureaMohammadAlavi]

@mkdev-cloud powermock adds a bit overhead but not 30 minutes!. Your tests methods must be doing some time-consuming things I can not help without seeing your code.

[mkdev-cloud]

I found a script on StackOverflow that works with gradle 4.x and I tweaked it to make it work with gradle 5.x. I tested this with gradle 6.0.1 and it worked perfectly.
To generate jacoco report you just need to run ./gradlew report.

configurations {
    jacoco
    jacocoRuntime
}

dependencies {
    jacoco group: 'org.jacoco', name: 'org.jacoco.ant', version: '0.8.4', classifier: 'nodeps'
    jacocoRuntime group: 'org.jacoco', name: 'org.jacoco.agent', version: '0.8.4', classifier: 'runtime'
}


task instrument(dependsOn: ['classes']) {
    ext.outputDir = buildDir.path + '/classes-instrumented'
    doLast {
        print "sourceSets.main.output.classesDirs: ${sourceSets.main.output.classesDirs}"
        ant.taskdef(name: 'instrument',
                classname: 'org.jacoco.ant.InstrumentTask',
                classpath: configurations.jacoco.asPath)
        ant.instrument(destdir: outputDir) {
            sourceSets.main.output.classesDirs.each { fileset(dir: it) }
        }
    }
}

gradle.taskGraph.whenReady { graph ->
    if (graph.hasTask(instrument)) {
        tasks.withType(Test) {
            doFirst {
                systemProperty 'jacoco-agent.destfile', buildDir.path + '/jacoco/tests.exec'
                classpath = files(instrument.outputDir) + classpath + configurations.jacocoRuntime
            }
        }
    }
}

task report(dependsOn: ['instrument', 'test']) {
    doLast {
        ant.taskdef(name: 'report',
                classname: 'org.jacoco.ant.ReportTask',
                classpath: configurations.jacoco.asPath)
        ant.report() {
            executiondata {
                ant.file(file: buildDir.path + '/jacoco/tests.exec')
            }
            structure(name: 'Example') {
                classfiles {
                    sourceSets.main.output.classesDirs.each { fileset(dir: it) }
                }
                sourcefiles {
                    fileset(dir: 'src/main/java')
                    //uncomment this if you use groovy
                    //fileset(dir: 'src/main/groovy')
                }
            }
            html(destdir: buildDir.path + '/reports/jacoco')
        }
    }
}

Hi
This is not working in junit platform runner with junit 5, and jacco folder and file is not created, kindly help me out for the same.

[AureaMohammadAlavi]

@mkdev-cloud please check out https://github.com/AureaMohammadAlavi/gradle-jacoo-offline-instrumentation. I added sample junit5 tests and after running report gradle task, jacoco report folder is created as expected. You probably need to specify in build.gradle file that you are using junit5:

test {
    useJUnitPlatform()
}

[mkdev-cloud]

Thanks for the reply.

It's working while running through junit 4
but not working in junit 5 platform runner due to system properties not
created test.exc file with Jacoco folder while running test, kindly advise

On Fri, 17 Jul, 2020, 12:15 pm Mohammad Alavi, [e-mail]
wrote:

@mkdev-cloud https://github.com/mkdev-cloud please check out
https://github.com/AureaMohammadAlavi/gradle-jacoo-offline-instrumentation.
I added sample junit5 tests and after running report gradle task, jacoco
report folder is created as expected. You probably need to specify in
build.gradle file that you are using junit5:

test {
useJUnitPlatform()
}

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gradle/gradle/issues/2429#issuecomment-659898774, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ALF6JYWXEP7IFBMLDFDZYJDR37XRNANCNFSM4DSKYCRQ
.

[AureaMohammadAlavi]

@mkdev-cloud The sample repo contains both junit4 and junit5 tests and jacoco report is created for both types of tests.
Please compare your build script with the one in the sample repo and make sure you include both vintage and jupiter test engines.

[mkdev-cloud]

Thanks for your reply.

Now jococo folder getting created after vintage dependency added but
getting power mock related exception.
Like suppress is not working...

On Fri, 17 Jul, 2020, 2:45 pm Mohammad Alavi, [e-mail]
wrote:

@mkdev-cloud https://github.com/mkdev-cloud The sample repo
https://github.com/AureaMohammadAlavi/gradle-jacoo-offline-instrumentation
contains both junit4 and junit5 tests and jacoco report is created for
both types of tests.
Please compare your build script with the one in the sample repo and make
sure you include both vintage and jupiter test engines.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gradle/gradle/issues/2429#issuecomment-659985362, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ALF6JYQ7XM3DWQNC5IKTK7TR4AJEXANCNFSM4DSKYCRQ
.

[AureaMohammadAlavi]

@mkdev-cloud PowerMock is not compatible with junit 5 and you can only use it with junit 4.

[Vampire]

That is untrue, you just have to know how to trick it and I described it on several issues in the powermock repo ;-)

[mkdev-cloud]

Can you share repo url please?

On Sat, 18 Jul, 2020, 3:49 am Björn Kautler, [e-mail]
wrote:

That is untrue, you just have to know how to trick it and I described it
on several issues in the powermock repo ;-)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gradle/gradle/issues/2429#issuecomment-660360926, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ALF6JYTOVCTEVJRRZSJLK4TR4DE6XANCNFSM4DSKYCRQ
.

[Vampire]

https://github.com/powermock/powermock

[mkdev-cloud]

It's fine. Suppress is not working for junit 5 with power mock delegates
using junit platform runner.

We are looking for solution that, generate jococo offline instrumentation
report using power mock runner with power mock delegates using junit
platform runner

Please advise...

On Sat, 18 Jul, 2020, 2:13 pm Björn Kautler, [e-mail]
wrote:

https://github.com/powermock/powermock

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gradle/gradle/issues/2429#issuecomment-660450827, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ALF6JYT4NFUPCWUIGHQS7YLR4FOC5ANCNFSM4DSKYCRQ
.

[Vampire]

I don't know what you mean by "suppress", but using power mock runner with power mock delegates using junit platform runner works fine if you do it right as I described it. This should work fine with offline instrumentation.

[mkdev-cloud]

Can you give me repo url which is generating jococo report using power mock
with junit platform runner in Gradle script?

On Sat, 18 Jul, 2020, 10:33 pm Björn Kautler, [e-mail]
wrote:

I don't know what you mean by "suppress", but using power mock runner with
power mock delegates using junit platform runner works fine if you do it
right as I described it. This should work fine with offline instrumentation.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gradle/gradle/issues/2429#issuecomment-660511422, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ALF6JYR4I6FA52XO3DYUZKLR4HIUXANCNFSM4DSKYCRQ
.

[Vampire]

I don't have a public one I can provide, no.
But as I said, I explained how to make PowerMock and JUnit 5 work together in the according issues and adding offline instrumentation is just the same, no matter how you run the classes.

[jGleitz]

Having offline instrumentation implemented in the Gradle Jacoco plugin is especially interesting since the instrumentation done for configuration caching breaks coverage for Gradle plugin tests. This could be easily accommodated if offline instrumentation was available in the Jacoco plugin (see https://github.com/gradle/gradle/issues/13614#issuecomment-658046647).