Godot: On the future of Android exporting

[reduz]

Hi everyone! I want to open this issue to:

• Discuss problems and solutions on improving the Android export workflow
• See who might want to volunteer to implement the solutions discussed.

I wrote the original Android port of Godot, but I'm a bit disconnected from mobile development as I haven't worked or published a mobile game since 2015. Thankfully many contributors have kept the port up to date.

There is however, a serious problem with it, which is that the approach used for exporting is slowly becoming obsolete. Originally, the Android port used Ant for building, which was quite very limited, so many of the hacks done were thought around that.

As the build system required Java installed and many other dependencies, I just took a pre-genreated APK and hacked it (modified binary resources inside) for exporting. This used to work OK for a while, but afterwards we started seeing many limitations to this approach:

• It's a hack, basically
• It makes it difficult to include third party add-ons. Community is often in pain due to the difficulty of adding third party APIs such as Admob or proprietary monetization/ad/analytics/etc. APIs. This needs to be made easier.
• It seems Google does not like our approach very much, given #18192

So I suggest we use this issue to discuss on how the Android exporter could be changed. I see two ways of doing this.

Proposal 1:

Keep the current exporter, but create an Android Template Builder. This would get an different export template, unzip it to a directory, allow you to make changes (add third party API support without recompiling Godot, change permissions, manifest, etc), then you can zip it again and create a new export template which is used with the current system.

Pros: Exporting remains rather simple for those who just want to test code on Android (no need to download Android SDK, which is a hassle, unless you need to)
Cons: May still not solve all problems or future problems that may arise due to us hacking the APK.

Proposal 2:

Do away entirely with the current export system and make the UI build the APK every time exporting is requiered by calilng gradle. Allow installing extensions from UI (or asset library) for ADMob and other stuff with almost no effort.

Pros: A lot less code and no need to do APK hacking. If you have android building properly set-up, it's easier to download stuff and make it work transparently.
Cons: If you just want to test on Android, you will need to get Android SDK with the right components, which can be a hassle. Can this be automated by Godot by calling the Android tool?

In any case, one of the goals of doing this change, is that you can download third party APIs from our asset library and get them to just work.. something users have a hugely difficult time with currently.

NOTE: Please keep the discussion on the above topic, don't propose or discuss features that are not directly related to this.

[volzhs]

I vote for the proposal 2.

[xsellier]

If we are talking about refactoring the apk exporting mechanism I think it might important to know which feature Godot Engine is missing:

1. APK Dynamic content: Allowing to generate a lighter apk
2. Google Instant App: User can play your game/app without installing it
3. Adaptive icons: Latest Android (Android P, and some Android 7.1+) require adaptive icons.
4. Crashing handler: Godot Engine is missing a crashing API. Meaning when Godot Engine crashes, we should expose a function allowing users to send it to their favorite crash-analytics application
5. ANR now crashes, so when the game is starting, we should use AsyncTask instead of UIThread.

While we are refactoring the exporting mechanism we should also add those features to Godot Engine.

My vote => 2

[HeartoLazor]

Maybe an hybrid?
Use the old approach if sdk is not found for testing in debug mode, but requires the install of SDK for release package. If sdk is found, use it in debug and release mode.
I vote 2, because is always a requirement if you are working in android.

[reduz]

@xsellier I would appreciate the point of this argument does not go beyond what was proposed in the OP, If someone wants to implement what you mention is fine, but let's please not derail.

[reduz]

@HeartoLazor Less code to maintain is better imo, so if going for #2, Hybrid is a bad idea.

[aswinmohanme]

I also vote for proposal 2.

Once set up Android tooling makes a lot of the work pretty painless, like downloading new sdk's and building external libs.

Also adding 3rd party libraries would be easier.

[stonekase]

I vote for option 2

[antoniotorresm]

Option 2 sounds a lot more convenient. Gradle is a great tool, adding libraries and managing the building process is a lot easier than the current flow.

[adeluiz]

My vote is for proposal 2

[vinod8990]

I had once created an android template creator as an addon for 2.x
https://github.com/vinod8990/godot-addon-aetc

However I vote for proposal 2.
If we make a standard, then need to make sure a module remove api too. It is a huge PIA when we need to remove a native plugin (as in the case of Unity).

[swarnimarun]

Vote -> Proposal 2: It seems like the most logical choice, Godot is currently and in near future the best game engine for mobile development and better admob/tools integration is a must when it comes to exporting to Android and mobile platforms.

Does this also mean that iOS export will also be improved.? At-least in near future.

[Guaranapps]

Vote for propsal 2: Some library integration required to rebuild apk with application specific files like google-services.json for firebase or strings.xml customization for facebook api. On Godot 2, I had patch SConstruct/SCsub/methods.py and some templates file in order to properly integrate some lib.

[reduz]

There seems to be mostly agreement here.. So I guess APK hacking will have to go away, and export templates should be pretty much the Android directory before Gradle is run..

Anyone up to make a proposal or PR for option #2? :)

[juanpaexpedite]

Well at the moment I've suffered a lot of pain adding the share and the Admob modules, because at the moment there is a bug in the get_data method from the texture so after compiling and else you lose.

Also I've to have an specific configuration to make it all work together java, gradle and else which is fine but is hard.

Also knowing what shader methods are available and have a more fluid physics would be great because I have to rewrite many things to change the position instead applying simple velocities because some issues that will be change in the 3.1 so having a page with tips on Android would be also a great step.

So the #2 with some checkboxes for optimizing for android would be great.

Of course I appreciate the effort that creating all this export environment.

[noidexe]

Number 2 seems cleaner so I think it's better than keeping working on a hack that might bring future problems.
My only worry is deployment time. Currently testing on android is really fast, it basically depends mostly on how big your assets are. Will option 2 severely impact build time?
In any case I still think it's the way to go. It'd be a matter of relying more on scene/script syncing for testing.

[dns]

Proposal 2 is much better because Google Play always require the apk to be build with their latest Android SDK.

And also GDnative is pain when I try to integrate with third party IAP & ads from Amazon, Leadbolt, etc.

[Shin-NiL]

Proposal 2 for sure.

Similar approaches are already used in some big projects like Cordova and Flutter. A great option would be the possibility to open the project in Android Studio if the user wants to have more control over the APK creation.

[aurodev]

Option 2 seems cleaner and more like the rest of Godot... i.e. done properly.

[ZodmanPerth]

I just went through the pain of getting the android export to work. The most pain was getting the right (versions of) tools installed; configuring in Godot was easy.
In my book, if doing it right involves a similar amount of pain getting the right tools installed it's still a step in the right direction, so I also vote for #2.
I share concerns about speed of launching on Android for debugging, but if I can't release to the store at the end then that's a huge issue that needs addressing; I need to know the store will be available when I'm ready to release.

[NathanWarden]

Definitely #2 :)

[PaulHMason]

I'm for option 2 - installing the Android sdk isn't a problem, and it opens up a lot of other possibilities.

At a high level, it seems similar to the way Flutter does things, https://github.com/flutter/buildroot

[Geequlim]

Does the option #2 means we have to compile the entire engine include C++ and Java code to export an android apk file for each project everytime?

That is would be a disaster ! Cocos Creator use such a terrible workflow.

[volzhs]

@Geequlim If I understand right, not c++ but java.

[vipsbpig]

I'm for option #1 .When I do some experiment or debug workflow.I don't want such a Android Bundle to Debug A Small Project.Keep the hack way.Let the progammer do the SDK workflow.

[JFonS]

@Geequlim Yeah, that's my concern too. If the project needs to be recompiled and packed into an apk everytime I have a feeling it will take quite a lot of time.

[veloc1]

@Geequlim @JFonS Full recompilation is not needed. Gradle has a lot of build time optimization, so full compilation (c++ and Java) will be needed only for first time. Also, when you apply third party plugin, it will modify Java code, Gradle will see that few files changed and one dependency added, and will recompile only changed parts. Most of the time exporter will copy project resources (scenes, scripts, and assets) to android folder, Gradle will pack an apk from cached source files and this resources and run it on android device, this should take not so much time.

[Geequlim]

@veloc1 We still have to compile the hole engine for each project event for test demos from the asset lib right ?

The gradle build workflow may broken by a lot of reaseon：

• The wrong version SDK installed.
• The dependences can't be downloaded with a bad network connection (that is really ofen in China without proxy).
• The avaliable memory of the machine is less than 4GB but some third-party SDK have to enable multiDexEnabled true to compile.
• ...

And the gradle is not always recompile as expected.

[2dvision]

Maybe i'm wrong, But i think #1 is a glass of milk and #2 is a dairy cow. Am I correct?

[veloc1]

@Geequlim As I can imagine, we don't really need to recompile whole engine. All code can be packed in library (so, each Godot release should rebuild android library with Godot engine). When project created, Godot editor can unzip android template in "android" folder with one java class, and Gradle file with included Godot engine android dependency from maven (React Native works the same way, as I remembered).
This way we can minimize build time, memory usage and improve simplicity of android project.

But you're right, this also involves a lot of limitations.

And some comments to your list of possibly broken things:

• SDK doesn't play much role, because Godot engine doesn't use a lot of features from new sdk. For simple projects almost any sdk will be ok. Also, downloading appropriate sdk can be automated.
• Dependency problem, that you described can be a pain. We can include dependencies with links from maven, but also we can put libraries files (jar or aar) in project folder, and use dependency, so with bad network connection you can download all dependencies in other places, put them on USB stick and move to project. But this should be greatly described in export documentation.
• Multidex and memory doesn't affects each other. Multidex can be enabled always in template project. Memory will be a stronger limitation. I can't say exactly, which amount of available memory will be needed for that, but this should be greater than 2GB.

And there is a wrap up of changes that should be implemented in Godot (according to the way, that I described above):

• Create android library, that contains engine and plugin system. (this is hard)
• Automate building android library on release and put it on maven.
• Make android project template.
• Collect a set of scripts to validate environment, downloads Gradle, Java, Android Sdk
• Make changes in Godot engine, so when user runs project on android device, engine should copy all resources and call Gradle task to build and run project.
• Documentation for all the things.

[reduz]

@Geequlim

Does the option #2 means we have to compile the entire engine include C++ and Java code to export an android apk file for each project everytime?

No.. I don't really see any reason for that. This is the case now, but the new workflow would simply get the android project in pre-built state (with the Godot .so files). Adding external modules that you downloaded in other directories (like, Admob) is really easy with Gradle, so you could theorically just download that support from the asset library and it will magically work.

[reduz]

@veloc1

There _is_ already a plugin system which modifies Gradle build to add stuff, that will most likely need no or little change.

The new "run/export" workflow would consist on:
1) Building Gradle (if no changes happened, nothing will be done, that should be pretty quick
2) Likely run the current export logic to unzip the APK and add the files, gdnative plugins, etc. and zip it again.. so signing would still be done by Godot and not Gradle I guess.

[reduz]

Likewise, I guess Godot could do the required editing to AndroidManifest.xml before building (change name, adding permissions and other stuff), add icons, etc. to make it easier on users who may still not be friendly to how manifest works. The good thing is that this is done in a non destructive way, so you can still add your own stuff to Manifest.

[veloc1]

@reduz

Didn't sure that signing should be on Godot side. Android plugin for Gradle has great feature for that https://developer.android.com/studio/build/build-variants#signing, so on export we just check keystore and passwords, and Gradle do all the work.

And for Manifest. When I say about plugin system, I mean exactly that thing: each plugin declare permissions, and on export we collect all included plugins and generate AndroidManifest.xml. Also, this way we can include library Services and BraodcastReceivers.

For resources like name and icon - what about placeholder resources, and when configured in project settings, just replace placeholders with values from project settings? So, end users will not see all this terrible android things at all.

[LinuxUserGD]

Will it be possible with the new export workflow to have the game pack outside of the APK instead of being zipped again (folder or .pck)?

[volzhs]

@LinuxUserGD isn't Apk Expansion (.obb) for that?
it's already there.

[vinod8990]

Whatever we do, it should be flexible and need to make sure it is not like unity's way of things. It's a mess when we mix and match several plugins which have same dependencies.

As for manifest, I'd like to keep it manually editable too.

The current build is very flexible that I could edit the template and add/delete whatever I want and nothing will interfere when building the templates.

I think something like a hybrid system will work?

[freemanfromgodotengine]

No doubt proposal 2 sounds better, cleaner and more "proper".
@reduz Cons of proposal 2? I imagine it's a hassle only when fast Internet is not available.
Godot, like Android Studio could probably very easily (as a text file?) pass to sdkmanager a list of needed packages to download https://developer.android.com/studio/command-line/sdkmanager

[Montecri]

Proposal 2, for sure.

[FrostByteGER]

Proposal #2. Android SDK installation is quite easy nowadays!

[rdhafidh]

@reduz , I'm voting for proposal 2. But, please don't forget about #18141

[snakeskinsalamander]

proposal 2. Gradle is also documented so we'll have more resources. Most mobile devs will have android sdk installed somewhere on their system anyway.

[lobinuxsoft]

Proposal 2

[gamergamesz]

When will this going to be implemented, any info or anything
@akien-mga Is it still the milestone for 3.1?

[jahd2602]

According to this blogpost from godotengine.org/news from Nov 23, 2018:

Godot 3.1 will be a release that covers a much wider surface area of what our users expect, so we are not that much in need of new features any longer. The only really big feature planned after 3.1 is porting the rendering to Vulkan...

Well the feature of the current issue seems to have gathered enough attention to be considered for 3.2. Is there any reason on why it wasn't addresed in the blogpost?

[akien-mga]

We ran out of time for a big refactor of Android exporting, so this will be for 3.2.

[abhilash1910]

Since android offers a variety of third party plugins and databases (Firebase,Cloud,ML,),it seems like a good idea to have android sdk/jdk/ndk installed for sustained development .The reason being since android libraries change with versions there maybe some dependency libraries missing when exporting is done without having a proper version of the sdk installed.But this approach on the contrary requires the user or end developer to ship the entire product to android platform just for the sake of testing which poses an overhead of time.An alternative approach can be done like making an application for android which uses the android studio sdk and the jdk on the device for testing.The user can simply run the application connect with the computer with a cable port and then run the game in the godot editor itself and view the changes on the phone or mobile.The mobile uses the android sdk of the computer through its port and as far as Proposal1 is concerned there is little overhead of time and is also almost transparent,there is no hacking required.
Because there are certain libraries in the android kernel code which if not installed will lead to runtime crashes mostly with java.lang.io.exceptions .There is also an issue to be addressed that if Proposal 1 is given precedence then that implies hacking into separate versions of android each having more dependencies than the preovious ones.Proposal 2 can be made by simply installing android sdk on the computer and using an android app (to be built) by the developers to instantly check their games on the mobile; more like a remote.For full building of an app or exporting the generalised approach 2 can be taken.
As far as the matter of templates is concerned,a android8+ adaptive templates can be used.Plus inorder to shorten the time of building the application (with using the sdk ) engine optimisations as to enhancing garbage collectors and also optimisations like struct based approach can be used.
This is entirely my perspective ,if it is alright then I have some plans made regarding the application (android) which uses the sdk of the device for instant building of the application game.

[Dancovich]

Google Play now accepts apps uploaded in their new Android App Bundle (AAB) format.

This format allows Google Play to dynamically generate APKs for many device types and one of the features that directly affect games made in Godot is packing APKs containing only the native library made for that particular CPU architecture.

This means that the download size can be vastly improved by having the APK downloaded by a x86 device only contain the x86 library, the one downloaded by an arm64 only contain the arm64 library and so on and so forth.

I believe whatever approach Godot takes should support this new packaging format but I don't know how easy it is to "hack" an AAB package in contrast with an APK package which is basically a ZIP file.

[Calinou]

This means that the download size can be vastly improved by having the APK downloaded by a x86 device only contain the x86 library, the one downloaded by an arm64 only contain the arm64 library and so on and so forth.

Note that this is already possible, but it requires manual work by creating one APK per architecture then uploading it to Google Play using its multiple APK support.

[akien-mga]

This was fixed by #27781, which basically implements a mix of 1 and 2. The old system is preserved for one click deploy, but releases will now ship with the pre-compiled template source that can be recompiled in a new APK with custom modules/manifest changes/etc. using gradle.