jib doesn't update images when included Gradle projects change

[emersonf]

Description of the issue:
gradle jibDockerBuild isn't creating new Docker images when code in Gradle included builds changes.

For the time being, I'm calling gradle build jibDockerBuild to work around this but that's counter-intuitive.

Expected behavior:
When code in an included build changes and gradle jibDockerBuild is re-run, a new image should be built containing the latest code from both the project and its included builds.

Steps to reproduce:
Assume a composite Gradle build where an app's build.gradle declares a dependency on a library using

    implementation("foo:some-lib:1.0.0") { changing = true }

and its settings.gradle contains

includeBuild '../some-lib'

1. If I intentionally introduce a compile-time error in some-lib and run gradle jibDockerBuild on the app, the build fails, so I know the right library code is being compiled.
2. If I make a non-breaking change in some-lib, jibDockerBuild succeeds but the Docker image it constructs is identical to the one before the change, it just has a new LastTagTime.
3. If I make a change to both the app and to the library and run gradle jibDockerBuild, a new image is built, but it still executes old some-lib code.
4. If I open a shell, I can see some-lib.jar in /apps/libs, which isn't getting updated.
5. If I run gradle build before gradle jibDockerBuild, the library is updated.

Environment: macOS, JDK8, Gradle

jib-gradle-plugin Configuration:

jib {
    from {
        image = "gcr.io/distroless/java:debug"
    }
    to {
        image = "foo/bar"
    }
    container {
        jvmFlags = ["-Duser.timezone=UTC"]
        ports = ["80"]
    }
}

[loosebazooka]

Any chance you give provide a simple project on which to test this out?

[loosebazooka]

I suspect that you could not be triggering a new build of some-lib somehow in your gradle configuration. Jib is not aware of what's being built underneath and will just work with what it's given from gradle.

Usually for multiproject builds, I expect to see the compile project('some-lib'), what's the reason you're using the composite build syntax?

[emersonf]

@loosebazooka composite builds improve the workflow of working with interacting projects, e.g. app and some-lib in multi-repo environments. In short, they provide quick feedback loops during development, while still preserving the isolation and other benefits of separate repos. Gradle's Stefan Oehme has a good 3 minute explanation here.

jibDockerBuild is definitely compiling some-lib, but "what it's given from Gradle" must not be including the compilation output from those libraries. We've never had an issue with bootJar targets, for example.

It may well be something problematic in our projects, but as a starting point it would be good to know if it's been tested against composite builds.

[emersonf]

@loosebazooka as for sample projects, Gradle has a few composite build samples on GitHub to bootstrap with. If that's not good enough, let me know and I'll put something together.

[loosebazooka]

@emersonf, right, I was just curious why you were using it. We sometimes encounter users making confusion decisions, it's a little presumptuous of me to make that assumption wholesale, but we usually need to make sure people know what they're doing before investing time on strange issues.

It turns out we expect gradle to behave predictably and it doesn't seem to be doing so here: For whatever reason gradle is not resolving substituted dependencies as projectDependencies but instead still as externalModuleDependencies when we read them, so our logic to handle projectDependencies isn't kicking in. I don't know at what point it does make this substitution. (<- turns out this is irrelevant except for skaffold intergration) I will take a deeper look today.

[loosebazooka]

So we definitely have an issue with jibBuildTar (we don't do new builds if project deps change) but I can't seem to recreate your issue with jibDockerBuild (in my test changes are always reflected). As long as your build is correctly rebuilding its dependencies, jib should just pull them from wherever gradle is building them to.

Can you run your build with --console=plain so we can see what tasks are being run (for the case that doesn't seem to be working)?

[emersonf]

So I followed this process...

1. Run the Gradle command in question.
2. Make a change to the code of a dependency. The dependency of interest is interesting-lib. Creativity at its best.
3. Re-run the Gradle command. The output below is from step 3.

Here's the output when the Gradle command is ./gradlew jibDockerBuild -Djib.to.tags=branch-develop --console=plain:

> Task :versionInfo
> Task :processResources UP-TO-DATE
> Task :some-lib-6:compileJava UP-TO-DATE
> Task :some-lib-4:compileJava UP-TO-DATE
> Task :some-lib-5:compileJava UP-TO-DATE
> Task :some-lib-3:compileJava UP-TO-DATE

> Task :interesting-lib:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :some-lib-1:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :compileJava
> Task :classes

> Task :jibDockerBuild

Containerizing application to Docker daemon as foo...
Getting base image gcr.io/distroless/java:8...
Building dependencies layer...
Building resources layer...
Building classes layer...

Container entrypoint set to [java, -Duser.timezone=UTC, -cp, /app/resources:/app/classes:/app/libs/*, com.foo.Application]
Loading to Docker daemon...

Built image to Docker daemon as foo

BUILD SUCCESSFUL in 6s
4 actionable tasks: 3 executed, 1 up-to-date

Here's the output when the Gradle command is ./gradlew assemble jibDockerBuild -Djib.to.tags=branch-develop --console=plain. I'm using assemble as it's narrower than build and also results in updated images:

> Task :versionInfo
> Task :some-lib-1:processResources UP-TO-DATE
> Task :processResources UP-TO-DATE
> Task :interesting-lib:processResources NO-SOURCE
> Task :some-lib-3:processResources UP-TO-DATE
> Task :some-lib-4:compileJava UP-TO-DATE
> Task :some-lib-4:processResources NO-SOURCE
> Task :some-lib-4:classes UP-TO-DATE
> Task :some-lib-5:compileJava UP-TO-DATE
> Task :some-lib-5:processResources NO-SOURCE
> Task :some-lib-5:classes UP-TO-DATE
> Task :some-lib-6:compileJava UP-TO-DATE
> Task :some-lib-4:jar UP-TO-DATE
> Task :some-lib-6:processResources UP-TO-DATE
> Task :some-lib-6:classes UP-TO-DATE
> Task :some-lib-5:jar UP-TO-DATE
> Task :some-lib-6:jar UP-TO-DATE
> Task :some-lib-3:compileJava UP-TO-DATE
> Task :some-lib-3:classes UP-TO-DATE
> Task :some-lib-3:jar UP-TO-DATE

> Task :interesting-lib:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :interesting-lib:classes
> Task :interesting-lib:jar

> Task :some-lib-1:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :some-lib-1:classes
> Task :some-lib-1:jar UP-TO-DATE
> Task :compileJava
> Task :classes
> Task :bootJar SKIPPED
> Task :jar SKIPPED
> Task :assemble UP-TO-DATE

> Task :jibDockerBuild

Containerizing application to Docker daemon as foo...
Getting base image gcr.io/distroless/java:8...
Building dependencies layer...
Building resources layer...
Building classes layer...

Container entrypoint set to [java, -Duser.timezone=UTC, -cp, /app/resources:/app/classes:/app/libs/*, com.foo.Application]
Loading to Docker daemon...

Built image to Docker daemon as foo

BUILD SUCCESSFUL in 11s
4 actionable tasks: 3 executed, 1 up-to-date

Happy diffing.

[loosebazooka]

So I'm experiencing a slightly different result. :jibDockerBuild is triggering the :jar tasks on the dependency projects, and that is enough for jib to pick up the changes.

So I've just modified the basic project from the examples: https://github.com/gradle/gradle/tree/master/subprojects/docs/src/samples/compositeBuilds/basic/groovy

I just add the com.google.cloud.tools.jib version 1.1.2 plugin and a simple jib.to.image config on my-app and ran the command

gradle --include-build ../my-utils/ jibDockerBuild --console=plain

and I get this output

> Task :processResources NO-SOURCE
> Task :my-utils:string-utils:compileJava UP-TO-DATE
> Task :my-utils:string-utils:processResources NO-SOURCE
> Task :my-utils:string-utils:classes UP-TO-DATE
> Task :my-utils:string-utils:jar UP-TO-DATE
> Task :my-utils:number-utils:compileJava UP-TO-DATE
> Task :my-utils:number-utils:processResources NO-SOURCE
> Task :my-utils:number-utils:classes UP-TO-DATE
> Task :my-utils:number-utils:jar UP-TO-DATE
> Task :compileJava UP-TO-DATE
> Task :classes UP-TO-DATE

> Task :jibDockerBuild

Containerizing application to Docker daemon as gcr.io/appu-learn/composite-test...
Getting base image gcr.io/distroless/java:8...
Building dependencies layer...
Building classes layer...

Container entrypoint set to [java, -cp, /app/resources:/app/classes:/app/libs/*, org.sample.myapp.Main]
Loading to Docker daemon...

Built image to Docker daemon as gcr.io/appu-learn/composite-test

BUILD SUCCESSFUL in 1s
2 actionable tasks: 1 executed, 1 up-to-date

And as you can see the difference between your build and mine is that the subproject dependencies are being built as full jars (so jib can pick up any changes).

Even just running gradle --include-build ../my-utils compileJava --console=plain triggers the :jars on the subprojects.

What version of gradle are you using? Maybe this is a gradle issue? I'm on 5.4.1

[loosebazooka]

@emersonf Any update here?

[emersonf]

@loosebazooka I haven't had time to try to continue evaluating, was planning to try again over the weekend. I'm on 5.4 as well.

My (only) suspicion is that Spring Boot's Gradle plugin is futzing with JAR behaviour, so I'll dig into that in the near future. Feel free to tag this issue as blocked on me if that helps.

[emersonf]

I think I found the culprit, and it's not Spring.

Our some-libs use the Java Library plugin instead of the JAR plugin to reduce compile classpath sizes and to avoid inadvertent transient dependencies, as recommended by Gradle. During compilation, If i'm reading this right and judging by the console output, JARs aren't built when using the Java Library plugin ... "consumers will use the output classes directory of this project directly on their compile classpath, instead of the jar file if the project uses the Java plugin". I assume "use" means reference, not actually copy.

My guess is that jibDockerBuild isn't seeing the updated .class files of the included builds during the "Building classes layer..." step and creating stale images as a result. But it is seeing the updated JAR files of the included builds that are created by a build or assemble task during the Building dependencies layer... step and updating the image correctly.

If that's right, this could be a bigger issue, because it would affect all projects using Java Library plugins, irrespective of included builds.

[emersonf]

To add a bit more confusion to the scenario, it seems like assemble is also not creating JARs for the dependent projects when the Java Library plugin is used, but Spring Boot's Gradle plugin is indeed futzing with things and creating them in the assemble task.

I think we should ignore all that; it would help to figure out how to get jibDockerBuild to work with Java Library plugin dependencies first.

[chanseokoh]

For the record:

The jibBuildTar input dependency issue (actually #1757) was fixed by #1754 at least. The fix is in v1.3.0.

[loosebazooka]

@emersonf, we haven't looked at this issue in a little because of some other things we're working on, but one temporary solution might be to put this in your build:

tasks.all { task ->
  if(task.name.startsWith('jib')) {
    task.dependsOn project.tasks.jar // or project.tasks.assemble
  }
}

if you have an idea of the correct dependsOn, that would be a real help to us, we can then verify your suggestion and include it.

[garyp]

I have a similar issue to #1776, where running jibDockerBuild fails because it can't find the jar files for sub-project dependencies. I tried the solution suggested by @loosebazooka above and could only get the build to pass when I added a dependency on tasks.assemble. Depending on tasks.jar was not sufficient.

From the logs, it seems the reason is that with the java-library plugin tasks.jar only builds the current project's jar file and does not create any dependencies on the jar tasks of sub-project dependencies. Whereas tasks.assemble does create a dependency on the jar tasks of all sub-project dependencies.

For reference, this is what I ended up adding in my build.gradle (modified from @loosebazooka's suggestion in order to work with gradle's new task configuration avoidance API):

tasks.matching {
    it.name.startsWith('jib')
}.configureEach {
    it.dependsOn tasks.named('assemble')
}

[loosebazooka]

Fixed by #2247