Xamarin-android: Remove use of EmbeddedResources for res, assets etc.

[dellis1972]

In order to better support reference assemblies and faster incremental builds we need to stop using EmbeddedResource to handle moving res, assets etc between projects.

The problem with EmbeddedResource is that we cannot easily detect what has been updated in a library project. It might only be one file or just some C# code. In order to find that out we need to extract the entire zip and process it.

The new method will place the required resources/assets etc next to the final assembly. This format will be Nuget Friendly in that the normal Nuget packaging with "just work". We will also autogenerate a .targets file which can be used to make sure the required files and folders are included in the build. This .targets will need to be in a build directory in order for it to be picked up by nuget. Other files need to be in a content folder or some other folder which nuget will automatically pick up.

MyProject.dll
content\assets\*
content\res\*
content\lib\*
build\MyProject.targets

This layout will be updated as the idea is expanded upon.

[dellis1972]

@Redth it would be helpful to know the current folder layout used in the Support/Google Nuget packages (for Aar files) to make sure we are on the right track.

[Redth]

@dellis1972 nothing is released with these yet, so blank slate here.

I think we still want to use AndroidAarLibrary for some packages which require the downloading of them with Xamarin.Build.Download, but if we're bundling things inside the nupkg otherwise, I guess it doesn't hurt to have the .aar extracted already into folders which you are suggesting.

I think if we do this it should follow the exact same layout as an aar file itself, just under some directory. I'm thinking it might make sense to nest these under an aar directory just to be explicit at what this layout is?

Keep in mind we should support multitargeting in this scenario so we will want to have something like:

lib\MonoAndroid90\Some.dll
build\MonoAndroid90\Some.targets
content\MonoAndroid90\aar\assets\*
content\MonoAndroid90\aar\res\*
content\MonoAndroid90\aar\AndroidManifest.xml
content\MonoAndroid90\aar\annotations.zip
content\MonoAndroid90\aar\classes.jar
content\MonoAndroid90\aar\proguard.txt
content\MonoAndroid90\aar\public.txt
content\MonoAndroid90\aar\R.txt

[Redth]

Actually, another thought, perhaps AndroidAarLibrary could be updated to support a directory path of unzipped aar contents as well, and in doing so you could use both this nupkg layout as well as .aar files themselves...

[jonathanpeppers]

One note about NuGet is it will only import a single .props and .targets file per package that is the same name as the package.

My.Package.Name.nupkg:

My.Package.Name.dll
build/My.Package.Name.props
build/My.Package.Name.targets
buildTransitive/My.Package.Name.props
buildTransitive/My.Package.Name.targets

So Xamarin.Android can't generate our own .props and .targets file to include automatically in NuGet packages. That would prevent users from writing their own .props and .targets files?

I think we will have to make all behavior rely on looking for files and directories _next_ to assemblies?

So if we just looked at a typical bin\Release output folder:

# .NET assembly
My.Package.Name.dll

# from @(AndroidAsset)
assets\*

# from @(AndroidResource)
res\*

# from @(AndroidEnvironment)
env\*

# from my.java.package.aar
lib\my.java.package.aar\assets\*
lib\my.java.package.aar\res\*
lib\my.java.package.aar\AndroidManifest.xml
lib\my.java.package.aar\annotations.zip
lib\my.java.package.aar\classes.jar
lib\my.java.package.aar\proguard.txt
lib\my.java.package.aar\public.txt
lib\my.java.package.aar\R.txt

# from foo.jar, bar.jar
lib\foo.jar
lib\bar.jar

# from foo.so
lib\armeabi-v7a\foo.so
lib\arm64-v8a\foo.so
lib\x86\foo.so
lib\x86_64\foo.so

We would mirror this directory structure to the lib, lib\MonoAndroid10.0 or lib\net5.0-android directory. This way @(ProjectReference) could work the same as a @(PackageReference)? Both are just looking for files on disk and if they exist or not.

If a user wanted to do something manually by copying files from bin\Release, they could copy the whole directory and things would work.

[mattleibow]

You could have a known location in a NuGet package. Sort of how the runtimes work today. In fact, you could potentially leverage that for the same things. If there is a .aar or .jar file in the runtimes/native/android, then use that.

.NET aleady has a RID for android and ios.

https://github.com/dotnet/runtime/blob/master/src/libraries/pkg/Microsoft.NETCore.Platforms/runtime.json#L139
https://github.com/dotnet/runtime/blob/master/src/libraries/pkg/Microsoft.NETCore.Platforms/runtime.json#L960

This might work?

[dellis1972]

We could still support user defined targets/props via a wildcard Import like we do here.

<Import Project="$(MSBuildThisFileDirectory)\Xamarin.Android.Common\ImportBefore\*" 
            Condition="Exists('$(MSBuildThisFileDirectory)\Xamarin.Android.Common\ImportBefore')"/>

This will import all the .target and .props file in the ImportBefore directory.

The user could write their own file and perhaps mark it with an action, which would allow use to process it.

I'm not sure I like the idea of having the resources next to an assembly. What if there are two versions of an assembly for android? (for different API levels). Would we need to duplicate the resources? e.g

lib\MonoAndroid28\Some.dll
lib\MonoAndroid30\Some.dll

[jonathanpeppers]

We could still support user defined targets/props via a wildcard Import like we do here.

So for this to work, you would need:

<PackageReference Include="My.Package.Name" Version="1.0" GeneratePathProperty="true" />

Theoretically can do something like:

<Import Project="$(Pkg_My_Package_Name)\ImportBefore\*" 
            Condition="Exists('$(Pkg_My_Package_Name)\ImportBefore')"/>

$(Pkg_My_Package_Name) is set by the ResolvePackageAssets target.

If we tried to use @(ReferencePath), this is only set by the ResolveAssemblyReference target.

An <Import/> would need to be able to work inside a <Target/> for one of the above to work, I think.

If there is a .aar or .jar file in the runtimes/native/android, then use that.

I like @mattleibow's idea to follow how other .nupkg files are structured. Maybe a .nupkg could be:

# Multi-targeted .NET assemblies
lib\net6.0-android.29\My.Package.Name.dll
lib\net6.0-android.30\My.Package.Name.dll

# from @(AndroidAsset)
runtimes\android\native\assets\*

# from @(AndroidResource)
runtimes\android\native\res\*

# from @(AndroidEnvironment)
runtimes\android\native\env\*

# from my.java.package.aar
runtimes\android\native\lib\my.java.package.aar\assets\*
runtimes\android\native\lib\my.java.package.aar\res\*
runtimes\android\native\lib\my.java.package.aar\AndroidManifest.xml
runtimes\android\native\lib\my.java.package.aar\annotations.zip
runtimes\android\native\lib\my.java.package.aar\classes.jar
runtimes\android\native\lib\my.java.package.aar\proguard.txt
runtimes\android\native\lib\my.java.package.aar\public.txt
runtimes\android\native\lib\my.java.package.aar\R.txt

# from foo.jar, bar.jar
runtimes\android\native\lib\foo.jar
runtimes\android\native\lib\bar.jar

# from foo.so
runtimes\android-arm\native\foo.so
runtimes\android-arm64\native\foo.so
runtimes\android-x86\native\foo.so
runtimes\android-x64\native\foo.so

If you have multiple $(TFM) of the same assembly, we would only have a single runtimes/android folder. We would need to use $(RuntimeIdentifier)-style folders for native libraries.

In build output, bin\Release, the runtimes folder would be right next to the assembly, but a couple directories higher in a .nupkg file.

I couldn't find docs on the NuGet runtimes folder, but I found: https://stackoverflow.com/a/52454147

[jonathanpeppers]

Embedded Resources and .nupkg Files

Traditionally, a Xamarin.Android class library can contain many types
of Android-specific files:

• @(AndroidAsset) files in Assets\
• @(AndroidEnvironment) text files
• @(AndroidJavaLibrary) .jar files
• @(AndroidNativeLibrary) files in lib\[arch]\*.so
• @(AndroidResource) files in Resources\

These are packaged in different ways as EmbeddedResource files in
the output assembly:

• __AndroidEnvironment__[filename]
  
  
  
  • @(AndroidEnvironment) files as-is
  
  
• __AndroidLibraryProjects__.zip
  
  
  
  • assets - @(AndroidAsset)
  
  
  • res - @(AndroidResource)
  
  
• __AndroidNativeLibraries__.zip
  
  
  
  • lib\[arch]*.so - @(AndroidNativeLibrary)
  
  
• *.jar
  
  
  
  • *.jar files directly as EmbeddedResource
  
  

The problem with this approach, is we have to inspect every assembly
at build time and extract these files to a directory. Because we have
a custom format that Android does not understand, we can't leave the
files as-is. Android tooling like aapt2 or javac work with files
and directories on disk.

New Approach in .NET 6

We want to consider an approach that is NuGet-centric. When the above
implementation was written, NuGet was just getting started. Developers
still had the pattern of copying .NET assemblies from bin and
committing it to their source control of choice. The single file
approach worked well for this scenario.

If we look at the general structure of a NuGet package:

lib
    net6.0-android29\Foo.dll
    net6.0-android30\Foo.dll
# Optional reference assemblies
ref
    net6.0-android29\Foo.dll
    net6.0-android30\Foo.dll
# Optional native libraries
runtimes
    android-arm\native\libFoo.so
    android-arm64\native\libFoo.so
    android-x86\native\libFoo.so
    android-x64\native\libFoo.so

There is not a great place where all Android file types would fit
following this pattern.

In Android Studio, Android libraries are packaged as .aar
files. A Xamarin.Android library, Foo.csproj, could also generate an
.aar file in its $(OutputPath):

Foo.aar
    classes.jar
    res/
    assets/
    libs/*.jar
    jni/[arch]/*.so

@(AndroidEnvironment) files are specific to Xamarin.Android, so we
can make one addition to the file format:

Foo.aar
    env/

So the output of Foo.csproj would look like:

bin/Debug/[targetframework]/
    Foo.dll
    Foo.aar
bin/Release/[targetframework]/
    Foo.dll
    Foo.aar

If you ran the Pack target, you would get a .nupkg file with:

lib
    net6.0-android29\Foo.dll
    net6.0-android30\Foo.aar
    net6.0-android29\Foo.dll
    net6.0-android30\Foo.aar

When consuming the .nupkg files, Xamarin.Android will leave .aar
files on disk as-is and not extract them. If users want to copy around
lose files and consume them, there will only be 1 additional .aar
file they will need to copy.

.aar Dependencies

A Foo.csproj might include a Java bar.aar file that is a
Java/Kotlin dependency.

These should sit alongside the .NET assembly:

lib
    net6.0-android29\Foo.dll
    net6.0-android30\Foo.aar
    net6.0-android30\Bar.aar
    net6.0-android29\Foo.dll
    net6.0-android30\Bar.aar

A Baz.jar file would be included in the above Foo.aar file at:

Foo.aar
    classes.jar
    libs/Baz.jar

Native Libraries

Since both .aar and .nupkg files support native libraries,
Xamarin.Android should support consuming native libraries from both
locations. A Xamarin.Android class library, Foo.csproj will place
native libraries in the Foo.aar file by default.

Questions

1. Can javac work with .aar files directly?
2. Can r8 work with .aar files directly?
3. Since .aar files have a AndroidManifest.xml, do we need to add
   some support for this in Xamarin.Android class libraries?
4. If we add AndroidManifest.xml, should we go ahead and
   "pre-generate" all the C# attributes like
   [assembly:UsesPermission] into it? This could skip some work in
   <GenerateJavaStubs/> looking for attributes in .NET assemblies.
5. Do we need a marker in assemblies to note that an .aar file
   should be accompanied with it? This would also help identify "new"
   .NET 6 Android assemblies from legacy assemblies that use
   EmbeddedResource.

[mattleibow]

"pre-generate" all the C# attributes

You r/madlad!

This will basically remove almost _all_ custom code for handling .NET Android libraries. If you only ever have to deal with a single format (or that new one) - less code, less chance for bugs to hatch.

[dotMorten]

Finally! Nice to see native libs getting the same approach .net core has. I've been using a .targets file to do it myself and it's so ugly

[mattleibow]

@dotMorten How does .net core do it? I I know about all the bits in the runtimes folder... And UWP with the compiled resources in a sub folder...

Is that what you mean?

[dotMorten]

@mattleibow .net core and uwp both use the runtimes folder as well. I was only referring to that bit

[jonathanpeppers]

So the answer to both of these is "no":

1. Can javac work with .aar files directly?
2. Can r8 work with .aar files directly?

I also found that manifestmerger.jar needs to work with AndroidManifest.xml files on disk.

This means we have to extract .aar files during the build.

I'm going to turn https://github.com/xamarin/xamarin-android/issues/2441#issuecomment-689160795 into a PR that adds an .md file, so we can do inline comments.

[jonathanpeppers]

The new spec has moved here: https://github.com/xamarin/xamarin-android/pull/5099

Probably easier to read here:

https://github.com/jonathanpeppers/xamarin-android/blob/spec-embeddedresource/Documentation/guides/OneDotNetEmbeddedResources.md

[dellis1972]

@jonathanpeppers we can probably just extract the bits that need to be on disk rather than all of it. aapt2 compile can use res folders from with in zip file. I suspect that would be the largest folder anyway. So we can probably just extract assets manifest and native libs.