Gradle: Allow nested composite builds

[gocursor]

I have a Java project A which depends on project B (module), and project B dependes on project C (another module). For project A I would like to setup "includeBuild ../projectB" and for project B I would like too setup "includeBuild ../projectC" so that I could develop everything in Eclipse + Buildship 2.0 without the need to run Gradle for every small change in each of the projecta A, B and C.

But if I setup this I get: "Included build '%s' cannot have included builds.".

Expected Behavior

Recursive "includeBuild" would recursively include dependent projects.

Current Behavior

I get "Included build '%s' cannot have included builds.".

Your Environment

Gradle 3.5, Buildship 2.0, Eclipse 3.6

[big-guy]

You can put all of the included builds into "A" if you want to work on it like that.

[gocursor]

Hi! I just found out the same... This workaround works for me now, but it is not as nice, because I need to define a large flat list of all dependencies which are actually deep down in the dependency hierarchy. And whenever I change dependency of project B I need to update this list in project A...

So the proposed feature would be nice-to-have, but it is not critical for me anymore...

[Sam-Fairbanks]

Can you give an example of how this workaround actually works? This issue I'm having is getting changes in B to see the changes made in C... is that possible?

[gocursor]

In settings,gradle of project A include builds of both: project B and C:
includeBuild ../projectB
includeBuild ../projectC

After importing Gradle projects A, B and C in Eclipse the changes in C will be visible to both - project A and B.

[Sam-Fairbanks]

Hmmm, I am definitely not seeing this behavior, but maybe it has something to do with my environment:

Gradle 3.2.1, Buildship 2.0, STS 3.8.2 (based on Eclipse Neon 4.6.1)

[gocursor]

It works for me (Gradle 3.5, Buildship 2.0, Eclipse Neon 4.6).

[Sam-Fairbanks]

My bad! This does appear to work correctly, even in Gradle 3.2.1. Thanks for this post, this helps a lot!

[fusr]

@bmuschko Hi, just out of curiosity, what's the reason for rejecting this feature?

[bmuschko]

@fusr As afar as I understand @Sam-Fairbanks's last comment this is a non-issue.

[big-guy]

@bmuschko We will eventually need to support something like this to fully replace buildSrc, so I think it's OK to leave this open (although, we're not doing anything immediately towards fixing this).

[gocursor]

Nested composite builds are "very nice to have", but currently we can live with a workaround described above...

[emersonf]

This workaround fixes project A but breaks project B.

If project A includes project B, and B includes project C, moving the C include from B to A means A will now work, but B is now broken and can't be built independently of A.

Is it somehow possible to ignore child includes (B including C) when building a parent project (A)? Or better yet just ignore them by default? This would prevent breaking B by removing C from B's settings.gradle. And let you build both A and B directly, without the dreaded Included build 'B' cannot have included builds. error.

[gocursor]

@emersonf Note that C is still included as dependency in B, but not as "included build" in settings.gradle! You can still build B independently of A, but C will not be rebuilt in this case - because it is not "included" - but the last built version will be taken from repository.

[emersonf]

@gocursor that defeats the purpose. The primary benefit of using included builds is to not pull dependencies from the repository.

I want to be able to test B against C the same way I test A against B and C.

Until nested composite builds are supported, B's dependency on C should be ignored when building A. That delegates to A to figure out what version of C to build against. B should be completely unaware that A exists, and shouldn't be written differently in case a parent comes along. (This exact use case just bit us.)

[gocursor]

@emersonf For my use case the purpose of composite builds is not ignoring the repository, but always using the latest sources of A, B and C during development, and also - with Eclipse Buildship 2.0 Plugin - browsing/showing/editing sources of dependencies.

[calvertdw]

Has anyone found a hack to know if you are the first build in the settings.gradle? Desired effect:

settings.gradle
if (!isIncludedBuild) {
includeBuild "../projectB"
...
}

I want to have all my projects usable independently, but use each other from source if present.

[gocursor]

@calvertdw I don't understand what exactly are you asking. Can you elaborate please?

Regarding "I want to have all my projects usable independently, but use each other from source if present." With the workaround described above you get the independency you mention and you can still use each other sources.

[calvertdw]

@gocursor Very specifically, how does a settings.gradle know if it's the first project to be evaluated?

I'm searching for a method to automate the includeBuilds such that I don't have to comment the others when I change working directories.

Example: I have 100 projects who all includeBuild each other for their dependencies. I have 1000s of users who will at random select one project to work on, but need to modify the source of some others at the same time. With your workaround they would have to modify all the settings.gradles to get it working.

[gocursor]

@calvertdw I think this is not possible at the moment,, because nested composite builds are not yet supported. What you can to today with the workaround proposed by @big-guy above is to put all included builds into one project - for all other projects (including nested dependencies).

[calvertdw]

It appears you can use startParameter.searchUpwards (whatever it means) to filter the first evaluation from the others. This way, no modification of settings.gradles are necessary when switching roots.

rootProject.name = 'projA'

if (startParameter.searchUpwards)
{
includeBuild ../projB
includeBuild ../projC
}

[valter-silva-au]

Hi everyone!
I'm facing the same issue. I have a 2 projects for multi-modules with Gradle, let's call them modules-technical, and modules-business;

This is my setting.gradle for modules-business:

rootProject.name = 'modules-business'
include 'business-core'
.. snip business-* modules ..
includeBuild '../modules-technical/'

Which means, that modules-business depends on modules-technical.

Now, I want to develop an app that depends on modules-business.

rootProject.name = 'my-app'
includeBuild '../modules-business'
includeBuild '../modules-technical'

Due to the current limitations of Gradle 4.2.1 I would need to comment the includeBuild statement on my modules-business project. Like this:

rootProject.name = 'modules-business'
include 'business-core'
.. snip business-* modules ..
// includeBuild '../modules-technical/'

Is there a better way to manage this limitatio for now? Meaning, how can I develop multi-modules and applications using Gradle composite?

[calvertdw]

To get around this I implemented a very custom build plugin for my company. It's open source though. Feel free to steal code. It's written in Kotlin. Main class of interest is the IHMCCompositeBuildAssembler. I'm planning on making this plugin more general in the future and rebranding as seperate from IHMC.

https://github.com/ihmcrobotics/ihmc-build/blob/master/src/main/kotlin/us/ihmc/build/IHMCCompositeBuildAssembler.kt

This functionality is pretty complicated as it turns out. You have to actually parse dependencies of other projects at settings.gradle evaluation time. This is probably why Gradle is having trouble with it.

I use it like this (settings.gradle):

buildscript {
   repositories {
      maven { url "https://plugins.gradle.org/m2/" }
      mavenLocal()
   }
   dependencies {
      classpath "gradle.plugin.us.ihmc:ihmc-build:0.9.9"
   }
}

import us.ihmc.build.IHMCSettingsConfigurator

/**
 * Browse source at https://github.com/ihmcrobotics/ihmc-build
 */
def ihmcSettingsConfigurator = new IHMCSettingsConfigurator(settings, logger, ext)
ihmcSettingsConfigurator.checkRequiredPropertiesAreSet() // Give friendly errors for unset properties
ihmcSettingsConfigurator.configureProjectName(hyphenatedName) // rootProject.name = hyphenatedName
ihmcSettingsConfigurator.configureExtraSourceSets(extraSourceSets) // Setup source sets to be multi-project subprojects

if (Boolean.valueOf(includeBuildsFromWorkspace))
{
   ihmcSettingsConfigurator.findAndIncludeCompositeBuilds() // Search workspace and `includeBuild` matches
}

[eliezio]

@calvertdw just set

isIncludedBuild = gradle.parent != null

from: https://stackoverflow.com/a/46450198/346545

this was the simplest/best solution I've found

[adammurdoch]

Initial support for this added in https://github.com/gradle/gradle/pull/5890

[gocursor]

Thank you!

[calvertdw]

@nddipiazza Does ignite-datastore depend on connector-plugin-ext's main or test source set? If it depends on the test source set then you have a circular dependency.

[nddipiazza]

@calvertdw i moved this over to an issue: https://github.com/gradle/gradle/issues/6147
so i deleted the existing comment to not double-up

[mabaoqing]

Hi! I just found out the same... This workaround works for me now, but it is not as nice, because I need to define a large flat list of all dependencies which are actually deep down in the dependency hierarchy. And whenever I change dependency of project B I need to update this list in project A...

So the proposed feature would be nice-to-have, but it is not critical for me anymore...

Hi, I have the same problem as you. My gradle version is 4.6. How did you solve it?

[rgoldberg]

@mabaoqing: upgrade Gradle to any version 4.10 or greater.

[mabaoqing]

@rgoldberg This version cannot be changed

[lacasseio]

@mabaoqing Unfortunately we won't backport nested composite builds. If there is any significant bugs preventing you from upgrading, please open an issue so we can address them.

[casidiablo]

I'm having some problems getting this to work with composite builds that are used as plugins.

For instance, if I have a plugin in my-plugin I can use it/apply it in a different project like this:

// my-project/settings.gradle
includeBuild("../my-plugin")

// my-project/build.gradle
buildscript {
  dependencies {
    classpath("my.plugin:the-plugin")
  }
}

apply(plugin: "the.plugin.id")

So far so good. However, if I have a super project that depends on my-project:

// super-project/settings.gradle
includeBuild("../my-project")

// super-project/build.gradle
dependencies {
  implementation("my.project:my-project")
}

Gradle seems to fail to find/apply the nested project:

* Where:
Build file '/foo/bar/my-project/build.gradle.kts' line: 3

* What went wrong:
Plugin [id: 'the.plugin.id'] was not found in any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (plugin dependency must include a version number for this source)

Any ideas of how to fix this? Was this supposed to work at all?

[rgoldberg]

@casidiablo: From what I know, applying a plugin to one project shouldn’t apply it to any other projects, including super projects.

So, your super project should use a plugins block to apply the plugin, and it should specify a version when applying the plugin (the version that you specify in the build file for the plugin).

Is there any reason why you’re not using the plugins block instead of apply?

[casidiablo]

Thanks for taking a look. I _was_ using the plugins block, and it does not work with it.

I finally managed to make it work, but seems still like a bug in Gradle somewhere. The way I was able to make it work was by using apply(plugin: the.class.Name) instead of using apply(plugin: "the-id") or plugins { id("the-id") }

I got the idea from https://github.com/gradle/gradle/issues/1262 where they have a similar issue but with buildSrc plugins

[rgoldberg]

@casidiablo: Sorry, I saw your question on my phone while on the subway, and misread it on the small screen and in my haste to answer before I exited to meet with people (plus I think you have the wrong directories in the comments of your second code block).

I’ve never tried to get an included plugin project working in a nested composite build, but got it working using the Gradle 5.5 Kotlin DSL in a normal single-level composite build using includeBuild(../my-plugin) in my composite project & plugins in my equivalent of your my-project. I did not need to reference anything for the plugin in any dependencies.

I imagine you need to supply a version when you add the plugin dependency to the classpath of the buildSrc of my-project, which must match the version in the build.gradle in the included plugin project.

Why do you reference the plugin from dependencies in your super project?

[casidiablo]

Sorry, I had a couple of typos in the example. I just updated it.

I do not reference the plugin from dependencies in my super project. I reference my-project, which, in turn, uses the plugin.

[rupebac]

I am also having problems, still with 6.0.1. I have reported it in: https://github.com/gradle/gradle/issues/11651

[sirinath]

Anything below the 1st nested include build is not built. I have reported a related issue with Gradle script convention plugins here: https://github.com/gradle/gradle/issues/14859