Jib: Too many tasks executed on jib build

[bs-git]

Description of the issue:
While using the jib gradle task (jib) the Plugin configures the jib tasks to depend on ALL assemble tasks on all upstream projects in a multi project build. This causes lots of unecessary artifacts being build even if only the jar (default dependency) of any upstream project is used.

Expected behavior:
If the plugin is applied only the default dependency should be used, which in fact is already evaluated for the build of the project (jar/classes). So there should be no need at all to depend on anything on the parent projects. If specific artifacts are needed by a specific project the project should configure this dependency itself, it should not be hard coded within the Jib Plugin itself.

Steps to reproduce:

Its in the original code:
`
COPIED FROM JibPlugin.java:

        // Find project dependencies and add a dependency to their assemble task. We make sure
        // to only add the dependency after BasePlugin is evaluated as otherwise the assemble
        // task may not be available yet.
        List<Project> computedDependencies = getProjectDependencies(projectAfterEvaluation);
        for (Project dependencyProject : computedDependencies) {
          dependencyProject
              .getPlugins()
              .withType(
                  BasePlugin.class,
                  unused -> {
                    TaskProvider<Task> assembleTask =
                        dependencyProject.getTasks().named(BasePlugin.ASSEMBLE_TASK_NAME);
                    buildImageTask.configure(task -> task.dependsOn(assembleTask));
                    buildDockerTask.configure(task -> task.dependsOn(assembleTask));
                    buildTarTask.configure(task -> task.dependsOn(assembleTask));
                  });
        }

`
If some kind of dependency is really required, there should be a minimal dependency setup from any required project. Normally this should only be the "default" gradle artifact - including the classpath.

[loosebazooka]

This was added in for: https://github.com/GoogleContainerTools/jib/issues/815. It basically is working as intended:

• The jib module only invokes classes since that's all that is required for the build.
• Because of this, the dependency modules also only invoke classes - and they will not generate a jar that we can include as a dependency, so we go invoke assemble explicitly

^^^ yes this is a consequence of how we do builds.

There are a few potential solutions to explore here:

• Perhaps we can find the exact task on the subproject that builds the artifact. We might be able to query which task builds the artifact that we are importing. From the project dependency -> configuration -> artifact -> builtBy
• Just depend on assemble instead of classes on the jib module for the multimodule case only
  
  
  
  • optionally disable the jar, war task on the jib module when in EXPLODED mode (a la springboot or whatever)
  
  

If we can agree on a solution... we love receiving pull requests from external contributors :)

[bs-git]

Thank you for your explaination. I see what the current solution tries to archive. I still do not understand the full complexity of this regarding exploded or non exploded dependencies used in spring boot, boot war, jar, application single/multi project... use cases. So my trivial "idea" of simply relying on the basic dependencies task "outputs.files" or "configuration.files" may not work as intended. On the other hand the complexity "explodes" if this tries to fulfill use cases with one "magic bullet". At last - I will try to look at your code again if there is anything that gives a simple solution for this.

[bs-git]

I am not quiet sure what was the original problem in #815 - the linked source does not show any problems and I can't reproduce the original issue the author of the patch was referring to. Also the new test case only validates the effect of the patch - not the original problem, so it does not test anything useful other than the patch code is executing.

On the other hand the described problem of #815 that only "compileJava" is executed is (probably) wrong. Any project dependency typically builds also the jar of the project. I have to verify this again - but from my point of view the general dependency on the assemble task is not needed at all.
I commented out the code and used that plugin in my multi project build without any sign of a problem - but of course this is only my project.

What would be the next step? Create a test case that reproduces the original problem and verifies it is (still) a problem? Or to introduce a patch to "disable" the assemble task dependency behavior on demand?

[loosebazooka]

@bs-git, sorry took a bit to get back to this. After a little more investigation it's specifically about the java-library plugin that interacts poorly. Where running classes on the main project wouldn't build a jar on the subproject that had the java-library plugin applied. A solution for this would require us to continue to work with java-library while also not triggering unnecessary tasks.

[seanabraham]

Chiming in here to say, that I agree depending on assemble is often overzealous and can result in cases where much more work is happening than necessary. One concrete example of this is when projects happen to apply the shadow plugin which is quite popular. The shadowJar task produces an artifact that is added to the archives configuration and therefore will run when assemble is called but this is entirely extraneous in the context of a jib run and winds up costing a lot of build time.

One alternative I'll propose (and I can also try to make a PR with this) is rather than trying to pick out the right tasks to depend on, we instead try and hook into the task graph in a way that matches the intention of what we're trying to do. Gradle gives you the option of allowing a task to depend on a Collection and it will automatically depend on the tasks which populate that file collection. In this example, I believe we can depend on the runtimeClasspath configuration (configurations are FileCollections under the hood) of the project which is exactly the set of files we need. In this way, we can express our intention more clearly to the build system (I want every artifact necessary to run this project — i.e. everything on the runtimeClasspath) and let it do the work of selecting the minimal set of tasks to satisfy that requirement. I'm fairly certain this will work and produce the jars we need, even if the java-library plugin is in use.

Let me know if you think I'm misunderstanding anything. If this is an option worth exploring I'm happy to try and put up a PR.

[loosebazooka]

Right it's expensive and there solutions here. Feel free to take a shot at it. In addition to the options presented in https://github.com/GoogleContainerTools/jib/issues/2184#issuecomment-561889549, I think you can also just check if a project has the "java-library" plugin applied (the core of the issue) and then do the math to calculate the correct build task to depend on.

[loosebazooka]

@seanabraham I actually got curious and was playing around with this. Lots of what I want is gradle internal, and the public api seems to make it pretty hard to the perfect thing. #2246 is a good stopgap for now.

[seanabraham]

@loosebazooka I took a shot at it and put it up at https://github.com/GoogleContainerTools/jib/pull/2247. I think public apis give you enough actually to get it done pretty cleanly as long as you depend on the FileCollection and let Gradle do the rest (FileCollections maintain mappings of which tasks produce which artifacts).

[chanseokoh]

@bs-git we've released Jib 2.1.0 with the fix.

@seanabraham thanks for your great contribution!