Android-components: Decision task should not invoke gradle

[mitchhentges]

The decision task should not be a heavyweight operation - it collects enough information to figure out which tasks need to run, then delegates them.

However, right now, we invoke gradle to get module definitions and the current version. This increases the minimum size of instance we can use to run the decision task, and increases overall costs and waiting time.

(Inspired by a conversation with ncalexan in Orlando)

[mitchhentges]

There's two pieces of info we grab from gradle:

• List of projects from (source is settings.gradle)
• componentsVersion (source is Config.kt)

I'd like to tackle this by putting .buildconfig.yml in the root of the repository. settings.gradle will parse it for the list of projects to pipe into setupProject, and Config.kt will be modified to use use a more-flexible backing map (I'll leave the types in for autocomplete convenience)

The format of .buildconfig.yml will look like:

projects:
    concept-awesomebar:
        path: 'components/concept/awesomebar'
        description: 'An abstract definition of an awesomebar component.'
        avoidPublish: true
    # <snip>
    samples-browser:
        path: 'samples/browser'
        description: 'A simple browser composed from browser components.'

Benefits:

• Faster, less expensive Taskcluster build

Negatives:

• Slower local build (should be < 1 second, so should be significantly less of a change compared to the savings in Taskcluster, but local builds happen more frequently, so it's a tradeoff?)
• Gradle build needs the complexity of parsing YAML now
• Taskcluster build needs the complexity of parsing YAML (much less complexity than invoking gradle, but it's a "more obvious"/"more explicit" chunk of logic)

@ncalexan @pocmo I believe that this is worth the tradeoffs, but you guys handle a-c builds more frequently than I do. :+1:?

[pocmo]

I have mixed feelings about this.

Your approach sounds good and this looks good to me. What is a bit concerning to me is that we reverse authority here. Previously the Android build was the source of truth and automation read from there. Now the automation configuration is the source of truth and the Android build needs to process it:

• Currently it's only about the configuration values. What if the automation process needs to access data that gradle determines at runtime. For example for one of the parallelization builds I would like to know whether a project is a library or (sample) app. I would need to duplicate this data in the config if we are not reading from Gradle. Maybe not that problematic though.

• My experience from Fennec was that not having a "standard Android build" first and instead having the build following automation/mach/Firefox/... was always a bit painful and often a blocker for new Android SDK/build features that should have been just one-liners most of the time. Those often turned into bigger projects that required knowledge about our atypical setup. I would like to know what @ncalexan thinks who did most of the amazing work that enabled us to use standard tools like gradle with Fennec :)

[mitchhentges]

Really :thinking: thanks Sebastian :)
I absolutely agree that deviating from a "standard" gradle configuration makes maintenance much more difficult. I think it's important that if this was to be followed-through, we'd have to be careful that the gradle build just depends on raw data (e.g.: a yaml file). This way, we can avoid the extra work of "my build isn't working, and I have to debug all the python/automation logic to find out what's happening". Instead, it's: "my build isn't working, it looks like gradle is getting my data from <static file>".
So, the dream would be something like:

                                                                     |==============|
|--------------|                                                     |    Entire    |
| static files |   <-- minimal glue logic to inject into gradle -->  |    gradle    |
|--------------|                                                     |    build     |
                                                                     |    config    |
                                                                     |==============|

---

What if the automation process needs to access data that gradle determines at runtime.

Yes, "data creep" of additional things needing to be specified in the static files is not good. That'll definitely be a cost :thinking:

[ncalexan]

I have mixed feelings about this.

Your approach sounds good and this looks good to me. What is a bit concerning to me is that we reverse authority here. Previously the Android build was the source of truth and automation read from there. Now the automation configuration is the source of truth and the Android build needs to process it:

* Currently it's only about the configuration values. What if the automation process needs to access data that gradle determines at runtime. For example for one of the parallelization builds I would like to know whether a project is a library or (sample) app. I would need to duplicate this data in the config if we are not reading from Gradle. Maybe not that problematic though.

I must step out of the shadows: I had a conversation with Mitch and Johan in Florida (beside a frickin' rocket!) about doing exactly this. Gradle is a big ball of goo; invoking it at decision task time is, in my opinion, doing too much in the decision task. If we get into a situation where we have non-trivial computation _in settings.gradle_ -- i.e., in Gradle's very earliest stage when it's accumulating project descriptors but not yet configuring any projects -- then we're probably doing too much at that time.

* My experience from Fennec was that not having a "standard Android build" first and instead having the build following automation/mach/Firefox/... was always a bit painful and often a blocker for new Android SDK/build features that should have been just one-liners most of the time. Those often turned into bigger projects that required knowledge about our atypical setup. I would like to know what @ncalexan thinks who did most of the amazing work that enabled us to use standard tools like gradle with Fennec :)

As I said, it was my gentle preference to do less at decision task time by moving some configuration out of settings.gradle. The flip side is that we do very non-trivial things at project descriptor accumulation time for Fennec (and GeckoView)! We parse mozconfigs by evaluating mach commands in order to get things configured correctly. That's mostly for the source and object directories, which aren't configurable in any (sensible) Gradle build. So my perspective is that if we need that much configuration that early in the process, we've gone off the rails a little bit.

Hope that clarifies my position: I am generally positive on simplifying our decision task and hard-coding more things in strictly weaker expressive systems.

[pocmo]

Alright. Then let's move ahead with slimming down our decision task :)