Renovate: Support Android Gradle projects

[rarkins]

Gradle support was added in #2610, but according to its author @corecanarias we still have some work to do in order to support Android-specific projects. This issue is a placeholder to assess community need for this support as well as collect requirements.

[DCKcode]

I tested Renovate's Gradle support on an Android project. It seems the first issue that is encountered is a lack of an Android SDK in the Docker image that runs the Gradle build.

To reproduce that issue:

$ git clone https://github.com/TeamNewPipe/NewPipe.git
$ cd NewPipe
### The Gradle checker adds an init.gradle in lib/manager/gradle/index.js , which you manually need to add here
$ cat init.gradle
gradle.projectsLoaded {
rootProject.allprojects {
    buildscript {
        repositories {
            maven {
              url "https://plugins.gradle.org/m2/"
            }
        }
       dependencies {
        classpath "gradle.plugin.se.patrikerdes:gradle-use-latest-versions-plugin:0.2.3"
        classpath 'com.github.ben-manes:gradle-versions-plugin:0.17.0'
       }
     }
     afterEvaluate { project ->
       project.apply plugin: 'com.github.ben-manes.versions'
       project.apply plugin: 'se.patrikerdes.use-latest-versions'
    }
  }
}
$ docker run --rm -v $(pwd):/mnt/renovate/gh/newpipe -w /mnt/renovate/gh/newpipe renovate/gradle gradle --init-script init.gradle dependencyUpdates -revision=release

This results in an error stating

A problem occurred configuring project ':app'.
> SDK location not found. Define location with sdk.dir in the local.properties file or with an ANDROID_HOME environment variable.`

The Android SDK is simply not configured in the current building container.

[rarkins]

Thanks @DCKcode for that info. I'm not a Gradle user myself so not 100% sure we need to add. I noticed that @alexisduque created a Docker Gradle image here that sounds like what you describe we need.

[DCKcode]

Configuring the Android SDK in the container can be quite tricky - I had a brief go at this myself to build it into the existing renovate/gradle container but got bogged down on what seem to be Java issues (it seems for now Android strongly prefers JDK 8). The current Gradle container uses Java 10.

Perhaps it makes sense to maintain a different infrastructure for Android projects - this will allow maintaining specific Java versions and Android SDK setups. I'm guessing you can autodetect Android projects based on the android top level block in the build.gradle file. The other logic around it should be the same though.

[rarkins]

I don't really have any preference towards Java 10 vs 8, so it's quite possible we could use 8 for everything for now until some point later when someone really requires 10 and at that point I need to support customisable images/versions. And yes of course it would be good to autodetect it whenever possible at that time, but for now I think we're hopefully OK with JDK 8. What do you think @corecanarias ?

[DCKcode]

I spent some time checking out whether the Android SDK is really the only problem. I created a modified version of the renovate/docker-gradle project that installs and configures the necessary Android build tools.

Using this local image I was able to run the following (NewPipe project as set up in my first comment in the thread):

docker run --rm -v $(pwd):/mnt/renovate/gh/newpipe -w /mnt/renovate/gh/newpipe renovate_android gradle --init-script init.gradle dependencyUpdates
Starting a Gradle Daemon (subsequent builds will be faster)
Parallel execution with configuration on demand is an incubating feature.
> Task :app:dependencyUpdates
> Task :dependencyUpdates

------------------------------------------------------------
: Project Dependency Updates (report to plain text file)
------------------------------------------------------------

The following dependencies are using the latest milestone version:
 - com.android.support:appcompat-v7:28.0.0
 - com.android.support:cardview-v7:28.0.0
 - de.hdodenhof:circleimageview:2.2.0
 - android.arch.persistence.room:compiler:1.1.1
 - com.android.support:design:28.0.0
 - com.android.support.test.espresso:espresso-core:3.0.2
 - com.nononsenseapps:filepicker:4.2.1
    ....

With the inclusion of the Android SDK it seems to work fine. Notice my modified local image, and the fact that I had to omit -revision=release from the command, which the Gradle command in Renovate now includes by default but is not accepted in the Android build environment. This is on the same Java 10 version (Ubuntu 18.04 base image default) as the current Gradle plugin is using.

There is one major caveat, however. I am including the Android SDK, all of its revisions, and all of the build tools for all Android versions in the Docker image build. The reason I did it this way was pure convenience, but it is also necessary to support as many Android build environments as possible. As a result, the image size balloons to a huge 9.33 GiB! For reference, the current plain renovate/gradle image is 629 MiB.

I can prepare a pull request for an updated Docker image on that repository if you're interested in a solution in this direction.

[rarkins]

@DCKcode that's really awesome!

I guess first thing we need to work out is - can we look at the gradle files and determine if they're Android gradle or not? I assume we can. This gives us the ability to change the -revision=release param dynamically.

The next question is: could we also detect the Android SDK version necessary, and if so then would we want to publish different Docker images and then dynamically select which one we need? 9.33GB is super big, however if the App's workers eventually end up caching all of the different SDK versions anyway then we don't really save any storage.

[corecanarias]

About the -revision=release can you provide the error you are getting? The actual parameter is -Drevision=release (note the -D at the beginning) Could be because of that?

The gradle-versions-plugin requires that parameter to filter only released versions.

[DCKcode]

Looks like it was a typo on my end indeed - -Drevision=release does work 😄

[DCKcode]

The next question is: could we also detect the Android SDK version necessary, and if so then would we want to publish different Docker images and then dynamically select which one we need? 9.33GB is super big, however if the App's workers eventually end up caching all of the different SDK versions anyway then we don't really save any storage.

The packages now include super old Android SDK platform versions and also SDK packages (e.g. Android Wear) that will be exceedingly rare in practice. So storing each SDK package on the build server is not efficient from a storage perspective, but necessary if you want to build (almost) any random Android project.

But sure, it can be dynamically detected in principle. E.g. the app/build.gradle file in NewPipe starts with this:

apply plugin: 'com.android.application'

android {
    compileSdkVersion 28
    buildToolsVersion '28.0.3'

For a tool with similar requirements, Travis CI will detect and download the Android SDK for a continuous integration build every single build and that is also not very efficient. That's just the unfortunate consequence of relying on a build tool that actually needs to build the project (instead of parsing the dependencies separately).

[DCKcode]

Of course that the only reason to include the Android SDK's is to make Gradle (in particular Gradle's Android plugin) happy - we don't actually care about building anything. It's just that a random Gradle file can declare a version of the Android SDK and Android build tools that need to be installed on the filesystem, and Gradle will not download these for you on its own.

Therefore it's worth noting I was also able to generate a dependency report on NewPipe using the existing renovate/gradle image with the following steps:

1. Comment out the android { } section in all app/build.gradle.
2. Replace apply plugin: 'com.android.application' with apply plugin: 'java'
3. Replace all Android specific dependency declarations (e.g. debugImplementation, androidTestImplementation) with plain implementation.

The resulting dependency report is word for word identical to a run in the Android SDK filled container. It's another route that could work, but of course it's a very hacky solution.

[corecanarias]

I would say that after finishing https://github.com/renovatebot/renovate/issues/2985 we would be in a better position to remove gradle entirely and parse the build.gradle files manually or at least try and see how it looks. Of course that will limit the way you can define dependencies in your project, but at the end of the day we are already limited because when we try to modify the dependencies we already expect to be in a particular format. Or maybe we could provide both options and see which one works better.

[DCKcode]

Ah - I see. If you are moving things to manual parsing of the build.gradle files already (which I guess is the more stable long term solution) then it will probably work right away if you take into account Android projects.

Let me know if you're still interested in the updated Docker image to support Android projects temporarily (on top of the current temporary regular Gradle support) until the other work is done. If you consider it worth the storage space, it does seem to be working out of the box. I'd be interested in using that at least 😃

[viceice]

Do we really need the sdk? For me renovate found my deps:

[viceice]

@corecanarias @DCKcode Have you tested the current renovate version? For me it works fine.

Maybe you need to upgrade gradle and the android gradle plugin to a recent version.

[viceice]

@rarkins If there are no new requirements we can close this.

[rarkins]

OK let's close, but reopen or create new if anyone has recent results.

[arhohuttunen]

An Android Gradle project with Renovate 23.68.0-slim Docker image, using Android SDK 29, build tools 29.0.3, and Gradle 6.7, Android Gradle build tools 4.1.0:

SDK location not found. Define location with an ANDROID_SDK_ROOT environment variable or by setting the sdk.dir path in your project's local properties file

[rarkins]

@arhohuttunen if you can create a minimal public repo to reproduce it, we can decide whether to reopen this issue or create a new one