status-react 🚀 - Research process of publishing to F-Droid

The docs start here: https://f-droid.org/en/docs/
But the main things I care about are:

jakubgs on 1 Jul 2019

There's already a Nix package for fdroidserver tools:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/tools/fdroidserver/default.nix

"Server and tools for F-Droid, the Free Software repository system for Android";

jakubgs on 1 Jul 2019

Looks like the basic step is just running this in fdroiddata repo:

fdroid import --url=https://github.com/status-im/status-react --subdir=android/app

Which in turn generates a metadata/im.status.ethereum.yml file:

License: Unknown
SourceCode: https://github.com/status-im/status-react
IssueTracker: https://github.com/status-im/status-react/issues

RepoType: git
Repo: https://github.com/status-im/status-react

Builds:
  - versionName: Unknown
    versionCode: 0
    disable: Generated by import.py - check/set version fields and commit id
    commit: '?'
    subdir: android/app
    gradle:
      - yes
    buildjni:
      - yes

AutoUpdateMode: None
UpdateCheckMode: Tags

Which doesn't look very impressive. But it's a start.

jakubgs on 1 Jul 2019

We can see an example of this file for an existing similar project here:
https://gitlab.com/fdroid/fdroiddata/blob/master/metadata/org.walleth.yml

jakubgs on 1 Jul 2019

Christ, since the usually used repo for Java 8 has been discontinued:
https://launchpad.net/~webupd8team/+archive/ubuntu/java
I'm trying the AWS alternative - Corretto:
https://docs.aws.amazon.com/corretto/latest/corretto-8-ug/generic-linux-install.html

 $ java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment Corretto-8.212.04.2 (build 1.8.0_212-b04)
OpenJDK 64-Bit Server VM Corretto-8.212.04.2 (build 25.212-b04, mixed mode

Seems to be working fine with sdkmanager.

jakubgs on 1 Jul 2019

There is a command to attempt to build you app through fdroid but that would require at least one version to be defined in the metadata/im.status.ethereum.yml file:

fdroid build --latest --verbose --test im.status.ethereum

jakubgs on 1 Jul 2019

The fdroid import command sometimes fails with a ridiculously unrelated error:

 $ fdroid import --url=https://github.com/status-im/status-react --subdir=android/app --categories=Money --license=MPL-2.0
WARNING: No 'config.py' found, using defaults.
CRITICAL: 'bc1qu482c0tngkcvx3q7mrm8zmuldrh2f2lrh26ym0' is not a valid Bitcoin address in org.projectmaxs.main. Regex pattern: ^[a-zA-Z0-9]{27,34

Trace:

Traceback (most recent call last):
  File "/usr/bin/fdroid", line 170, in <module>
    main()
  File "/usr/bin/fdroid", line 146, in main
    mod.main()
  File "/usr/lib/python3/dist-packages/fdroidserver/import.py", line 219, in main
    apps = metadata.read_metadata()
  File "/usr/lib/python3/dist-packages/fdroidserver/metadata.py", line 814, in read_metadata
    check_metadata(app)
  File "/usr/lib/python3/dist-packages/fdroidserver/metadata.py", line 493, in check_metadata
    v.check(app[k], app.id)
  File "/usr/lib/python3/dist-packages/fdroidserver/metadata.py", line 434, in check
    .format(value=v, field=self.name, appid=appid, pattern=self.matching))
  File "/usr/lib/python3/dist-packages/fdroidserver/metadata.py", line 45, in warn_or_exception
    raise MetaDataException(value)
fdroidserver.exception.MetaDataException: 'bc1qu482c0tngkcvx3q7mrm8zmuldrh2f2lrh26ym0' is not a valid Bitcoin address in org.projectmaxs.main. Regex pattern: ^[a-zA-Z0-9]{27,34}$

jakubgs on 1 Jul 2019

😕1 😄1

Looks like I got that wrong. You are supposed to run this in the clone of fdroidclient repo.

~/soft/fdroidclient master ⇣
 lilim > fdroid import --url=https://github.com/status-im/status-react --subdir=android/app --categories=Money --license=MPL-2.0 --verbose
WARNING: No 'config.py' found, using defaults.
INFO: Getting source from git repo at https://github.com/status-im/status-react
DEBUG: > git -c core.askpass=/bin/true -c core.sshCommand=/bin/false -c url.https://.insteadOf=ssh:// -c url.https://u:[email protected]/[email protected]: -c url.https://u:[email protected]=git://bitbucket.org -c url.https://u:[email protected]=https://bitbucket.org -c url.https://u:[email protected]/[email protected]: -c url.https://u:[email protected]=git://github.com -c url.https://u:[email protected]=https://github.com -c url.https://u:[email protected]/[email protected]: -c url.https://u:[email protected]=git://gitlab.com -c url.https://u:[email protected]=https://gitlab.com clone -- https://github.com/status-im/status-react tmp/importer
Cloning into 'tmp/importer'...
DEBUG: Directory: tmp/importer
DEBUG: > git rev-parse --show-toplevel
DEBUG: Directory: tmp/importer
DEBUG: > git checkout -f origin/HEAD
DEBUG: Directory: tmp/importer
DEBUG: > git clean -dffx
DEBUG: Parsing manifest at 'tmp/importer/android/app/src/main/AndroidManifest.xml'
DEBUG: ..got package=im.status.ethereum, version=None, vercode=None
DEBUG: Parsing manifest at 'tmp/importer/android/app/build.gradle'
DEBUG: ..got package=im.status.ethereum, version=None, vercode=None
WARNING: Couldn't find latest version code
INFO: Wrote metadata/im.status.ethereum.yml

jakubgs on 9 Jul 2019

The next step is to define the build steps necessary to build our App on the F-Droid build farms:
https://f-droid.org/en/docs/Building_Applications
https://f-droid.org/en/docs/Build_Metadata_Reference

jakubgs on 9 Jul 2019

So the build steps are defined in the YAML meta file that is initially generated by fdroid import. Inside of that file various keys are set to define the build process.

There's something called prebuild:

Specifies a shell command (or commands - chain with &&) to run before the build takes place. Backslash can be used as an escape character to insert literal commas, or as the last character on a line to join that line with the next. It has no special meaning in other contexts; in particular, literal backslashes should not be escaped.
The command runs using bash.
Note that nothing should be built during this prebuild phase - scanning of the code and building of the source tarball, for example, take place after this. For custom actions that actually build things or produce binaries, use ’build’ instead.

And for build:

As for ’prebuild’, but runs during the actual build phase (but before the main Ant/Maven build). Use this only for actions that do actual building. Any prepartion of the source code should be done using ’init’ or ’prebuild’.
Any building that takes place before build will be ignored, as either Ant, mvn or gradle will be executed to clean the build environment right before build (or the final build) is run.

This seems to indicate that you can only build using Ant, Maven, or Gradle.

jakubgs on 9 Jul 2019

There is gradle key:

gradle: <flavour1>[,<flavour2>,...]
Build with Gradle instead of Ant, specifying what flavours to use. Flavours are case sensitive since the path to the output apk is as well.
If only one flavour is given and it is ’yes’, no flavour will be used. Note that for projects with flavours, you must specify at least one valid flavour since ’yes’ will build all of them separately.

Which seems to indicate not Gradle buildTypes, but productFlavors:
https://developer.android.com/studio/build/build-variants#product-flavors
Of which we don't seem to have any defined in our android/app/build.gradle.
Not sure which build type fdroid build would make if we have no flavors defined.

There is also:

gradleprops: <prop1>[,<prop2>,...]
List of Gradle properties to pass via the command line to Gradle. A property can be of the form foo or of the form key=value.
For example: gradleprops=enableFoo,someSetting=bar will result in gradle -PenableFoo -PsomeSetting=bar.

jakubgs on 9 Jul 2019

Not sure what this one affects in Gradle but it might be build type:

target: <target>
Specifies a particular SDK target for compilation, overriding the value defined in the code by upstream. This has different effects depending on what build system used — this flag currently affects Ant, Maven and Gradle projects only. Note that this does not change the target SDK in the AndroidManifest.xml, which determines the level of features that can be included in the build.

In the case of an Ant project, it modifies project.properties of the app and possibly sub-projects. This is likely to cause the whole build.xml to be rewritten, which is fine if it’s a ’standard’ android file or doesn’t already exist, but not a good idea if it’s heavily customised.

EDIT: Correction, it looks like target refers to targetSdkVersion in Gradle config.

jakubgs on 9 Jul 2019

I'm trying to build it, but I'm getting python errors now:

 $ fdroid build im.status.ethereum
hWARNING: No 'config.py' found, using defaults.
INFO: Using git version 2.20.1
INFO: Building version 0.13 (2019070916) of im.status.ethereum
INFO: Getting source for revision 0.13.0-mobile
INFO: Creating local.properties file at build/im.status.ethereum/local.properties
INFO: Creating local.properties file at build/im.status.ethereum/android/local.properties
INFO: Creating local.properties file at build/im.status.ethereum/android/app/local.properties
INFO: Cleaned build.gradle of keysigning configs at build/im.status.ethereum/android/app/build.gradle
INFO: Cleaning Gradle project...
ERROR: Could not build app im.status.ethereum: OSError while trying to execute /usr/lib/python3/dist-packages/gradlew-fdroid clean: [Errno 2] No such file or directory: '/usr/lib/python3/dist-packages/gradlew-fdroid': '/usr/lib/python3/dist-packages/gradlew-fdroid'
INFO: Build for app im.status.ethereum failed:
OSError while trying to execute /usr/lib/python3/dist-packages/gradlew-fdroid clean: [Errno 2] No such file or directory: '/usr/lib/python3/dist-packages/gradlew-fdroid': '/usr/lib/python3/dist-packages/gradlew-fdroid'
INFO: Finished
INFO: 1 build failed

jakubgs on 9 Jul 2019

It doesn't matter if I install fdroidserver with APT of with PIP, it fails with the same nonsense error.

According to the instructions the APT way is the way to go:
https://f-droid.org/en/docs/Installing_the_Server_and_Repo_Tools/

jakubgs on 9 Jul 2019

I tried running the build with:

prebuild: 'curl https://nixos.org/nix/install | sh'

And it did run, but then fails with:

...
installing 'nix-2.2.2'
building '/nix/store/5qjkri2jrxqsly68q9n2qh95xk5j06g1-user-environment.drv'...
created 141 symlinks in user environment
Segmentation fault
==== detail end ====
INFO: Finished
INFO: 1 build failed

Which makes no fucking sense, because the nix installation succeeded just fine.

But when I re-run it it works fine...

jakubgs on 9 Jul 2019

Looks like something like this works:

Builds:
  - versionName: '0.13'
    versionCode: 2019070916
    commit: '0.13.0-mobile'
    #subdir: android/app
    gradle:
      - yes
    buildjni: no
    init: |
      curl https://nixos.org/nix/install | sh
    prebuild: |
      . ~/.nix-profile/etc/profile.d/nix.sh && \
      nix-build --no-out-link '<nixpkgs>' -A glibcLocales && \
      nix-shell --run 'bundle install --gemfile=fastlane/Gemfile --quiet'

But it fails with the python errors when it gets to the build step.

jakubgs on 9 Jul 2019

For readability...

 init:
   - curl https://nixos.org/nix/install | sh
 prebuild:
   - . ~/.nix-profile/etc/profile.d/nix.sh
   - nix-build --no-out-link '<nixpkgs>' -A glibcLocales
   - nix-shell --run 'bundle install --gemfile=fastlane/Gemfile --quiet'

licaon-kter on 10 Jul 2019

Oh nice, I didn't know this was also a valid format, the manual stated that:

or commands - chain with &&

Thanks @licaon-kter!

jakubgs on 11 Jul 2019

So the list syntax does work, but I'm getting some Nix target errors which makes no sense to me:

Builds:
  - versionName: 'develop'
    versionCode: 9999
    commit: 'develop'
    #subdir: android/app
    buildjni: no
    gradle:
      - yes
    init:
      # Install Nix
      - curl https://nixos.org/nix/install | sh
      # Source Nix environment
      - . ~/.nix-profile/etc/profile.d/nix.sh
    prebuild:
      # Install required Gems
      - nix-shell -A targets.mobile.fastlane.shell --run 'bundle install --gemfile=fastlane/Gemfile --quiet' ./default.nix

And I'm running:

fdroid build im.status.ethereum:9999 -v -l

But I'm getting something weird:

+ nix-shell -A targets.mobile.fastlane.shell --run 'bundle install --gemfile=fastlane/Gemfile --quiet' ./default.nix
error: value is null while a set was expected, at /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/top-level/default.nix:63:5

Which appears to be this:

  # From a minimum of `system` or `config` (actually a target triple, *not*
  # nixpkgs configuration), infer the other one and platform as needed.
  localSystem = lib.systems.elaborate (
    # Allow setting the platform in the config file. This take precedence over
    # the inferred platform, but not over an explicitly passed-in one.
    builtins.intersectAttrs { platform = null; } config
    // args.localSystem);

Not sure what to make of this yet.

jakubgs on 17 Jul 2019

This seems to be related: https://github.com/NixOS/nixpkgs/issues/51858

Full trace:

+ nix-shell --show-trace -A targets.mobile.fastlane.shell --run 'bundle install --gemfile=fastlane/Gemfile --quiet' ./default.nix
error: while evaluating anonymous function at /home/sochan/soft/fdroidclient/build/im.status.ethereum/default.nix:2:1, called from undefined position:
while evaluating the attribute 'pkgs' at /home/sochan/soft/fdroidclient/build/im.status.ethereum/nix/nixpkgs-bootstrap.nix:12:10:
while evaluating anonymous function at /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/top-level/impure.nix:15:1, called from /home/sochan/soft/fdroidclient/build/im.status.ethereum/nix/nixpkgs-bootstrap.nix:3:10:
while evaluating anonymous function at /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/top-level/default.nix:20:1, called from /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/top-level/impure.nix:82:1:
while evaluating anonymous function at /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/stdenv/booter.nix:42:1, called from /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/top-level/default.nix:100:10:
while evaluating 'dfold' at /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/stdenv/booter.nix:60:27, called from /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/stdenv/booter.nix:117:4:
while evaluating 'go' at /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/stdenv/booter.nix:63:18, called from /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/stdenv/booter.nix:72:13:
while evaluating 'imap1' at /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/lib/lists.nix:103:14, called from /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/stdenv/booter.nix:78:30:
while evaluating 'reverseList' at /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/lib/lists.nix:380:17, called from /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/stdenv/booter.nix:85:6:
while evaluating anonymous function at /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/stdenv/default.nix:7:1, called from /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/top-level/default.nix:96:12:
while evaluating 'elaborate' at /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/lib/systems/default.nix:18:15, called from /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/top-level/default.nix:60:17:
value is null while a set was expected, at /nix/store/q9i5qy8xbp0jy3k34dh794pc8n766nnm-nixpkgs-source/pkgs/top-level/default.nix:63:5

It's almost as if config is passed to the nix-shell invocation as --argstr but it doesn't contain what it needs to contain.

jakubgs on 17 Jul 2019

The apparent fix is to default config to be an empty map:

config ? {}

https://github.com/NixOS/nixpkgs/commit/878965731f9e35580efc21b0d3e79022cfaeb0be

But we already set it to a map by default if not set in all Nix files.

jakubgs on 17 Jul 2019

Yeah, that was the reason I started using a default value for config recently.

pedropombeiro on 18 Jul 2019

This part of the instructions terrifies me:

# Set up Vagrant build box
cd ../fdroidserver
cp ./examples/makebuildserver.config.py ./
./makebuildserver
# Now wait several hours for this to finish
https://f-droid.org/en/docs/Submitting_to_F-Droid_Quick_Start_Guide/

Now wait several hours for this to finish

jakubgs on 25 Jul 2019

I might have to setup a dedicated server for building with fdroid build, I don't want to fill my laptop with garbage that makebuildserver pulls.

jakubgs on 25 Jul 2019

before setting up a whole server, maybe something like this can help? https://gitlab.com/fdroid/docker-executable-fdroidserver (though I am not up to date whether this is compatible with reproducible builds and similar stuff)

step21 on 25 Jul 2019

Eh, a Docker container? That does sound nice, but I thought that wouldn't work since it wants to use Vagrant. How does it use Vagrant within the container?

But yeah, I do like how isolated a Docker env would be, thanks for the suggestion @step21.

jakubgs on 25 Jul 2019

Btw, that error about error: value is null while a set was expected was due to me using an older version of status-react which didn't have the change that makes config in Nix never null.
Turns out fdroid build didn't fetch newest changes when i changed the commit to just be develop branch.

jakubgs on 25 Jul 2019

I'm still not entirely sure on how the server part ties into F-Droid. My current understanding is that I don't need the server to submit the app, the server is for F-Droid organization to build apps others submit. But I might be wrong.

Also, I'm still not entirely sure what flavour means in gradle: <flavour1>[,<flavour2>,...]. Considering I want to run gradle via Nix I'm hoping I can call a custom build command rather than depending on fdroid calling gradle the right way.

jakubgs on 25 Jul 2019

I'm still not entirely sure on how the server part ties into F-Droid. My current understanding is that I don't need the server to submit the app, the server is for F-Droid organization to build apps others submit. But I might be wrong.

Also, I'm still not entirely sure what flavour means in gradle: <flavour1>[,<flavour2>,...]. Considering I want to run gradle via Nix I'm hoping I can call a custom build command rather than depending on fdroid calling gradle the right way.

I just briefly checked the makebuildserver stuff and the docker link. I think you are right, you do not need the buildserver, and anyway it just sets up everything up in vagrant, so it is not necessarily better than a docker image. (though I did not verify if they are roughly the same, as I do not want to install vagrant again...) And the docker link I posted does not run makebuildserver and all that vagrant stuff but just installs froidserver afaict

step21 on 25 Jul 2019

Back at it again, after finishing some other projects.

I tried using the registry.gitlab.com/fdroid/docker-executable-fdroidserver:latest docker image suggested but what I got were some errors about Gradle download:

$  docker run --rm -u $(id -u):$(id -g) \
  -w /repo -v $(pwd):/repo -v $ANDROID_HOME:/android -e ANDROID_HOME=/android \
  registry.gitlab.com/fdroid/docker-executable-fdroidserver:latest \
  build im.status.ethereum:9999 -v -l

...
INFO: Cleaning Gradle project...
DEBUG: Directory: build/im.status.ethereum
DEBUG: > /fdroidserver/gradlew-fdroid clean
No suitable gradle version found - defaulting to 5.0
Downloading missing gradle version 5.0
mkdir: cannot create directory ‘/.cache’: Permission denied
curl: (23) Failed writing body (0 != 433)
sha256sum: /.cache/fdroidserver/gradle-5.0-bin.zip: No such file or directory
/.cache/fdroidserver/gradle-5.0-bin.zip: FAILED open or read
sha256sum: WARNING: 1 listed file could not be read
gradle download checksum mismatch! Exiting...
ERROR: Could not build app im.status.ethereum: Error cleaning im.status.ethereum:develop
==== detail begin ====
No suitable gradle version found - defaulting to 5.0
Downloading missing gradle version 5.0
mkdir: cannot create directory ‘/.cache’: Permission denied
curl: (23) Failed writing body (0 != 433)
sha256sum: /.cache/fdroidserver/gradle-5.0-bin.zip: No such file or directory
/.cache/fdroidserver/gradle-5.0-bin.zip: FAILED open or read
sha256sum: WARNING: 1 listed file could not be read
gradle download checksum mismatch! Exiting...
==== detail end ====
INFO: Finished
INFO: 1 build failed

jakubgs on 8 Aug 2019

But this is progress of a kind, because before without the docker image it was failing at:

...
INFO: Cleaning Gradle project...
DEBUG: Directory: build/im.status.ethereum
DEBUG: > /usr/local/lib/python3.7/dist-packages/gradlew-fdroid clean
ERROR: Could not build app im.status.ethereum: OSError while trying to execute /usr/local/lib/python3.7/dist-packages/gradlew-fdroid clean: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/dist-packages/gradlew-fdroid': '/usr/local/lib/python3.7/dist-packages/gradlew-fdroid'
INFO: Finished
INFO: 1 build failed

jakubgs on 8 Aug 2019

Based on the Dockerfile gradle is installed by apt:
https://gitlab.com/fdroid/docker-executable-fdroidserver/blob/master/Dockerfile#L15-19
So the issue must be with gradlew wrapper which tries to download a specific version ZIP:

The Wrapper is a script that invokes a declared version of Gradle, downloading it beforehand if necessary.
https://docs.gradle.org/current/userguide/gradle_wrapper.html

jakubgs on 8 Aug 2019

Ah, I see now:

mkdir: cannot create directory ‘/.cache’: Permission denied

Is due to me running the process under my own UID using -u $(id -u):$(id -g) but as a non-existent user which means its home becomes /(root) by default.

jakubgs on 8 Aug 2019

Also, it's downloading Gradle 5.0, which is different from what we use:
https://github.com/status-im/status-react/blob/c49e4933eeea8e02e2724b5eab27d046e8c4e549/nix/mobile/android/gradle.nix#L8
I don't see any setting in the metadata reference to change this:
https://f-droid.org/en/docs/Build_Metadata_Reference/
According to the Gradle docs:

The Gradle version used for executing the build. By default the wrapper task picks the exact same Gradle version that was used to generate the Wrapper files.
https://docs.gradle.org/current/userguide/gradle_wrapper.html

But that can be changed with:

gradle/wrapper/gradle-wrapper.properties
distributionUrl=https\://services.gradle.org/distributions/gradle-5.5.1-bin.zip
https://docs.gradle.org/current/userguide/gradle_wrapper.html

Not sure yet if we care enough to do that.

jakubgs on 8 Aug 2019

Ok, if I run the container with -u as root then it fails like this:

FAILURE: Build failed with an exception.

* What went wrong:
Task 'clean' not found in root project 'im.status.ethereum'.

* Try:
Run gradle tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Which essentially means that fdroid is trying to run a clean Gradle task that does not exist.

jakubgs on 8 Aug 2019

I'm not seeing any specific way of applying --stacktrace flag to Gradle.
I thought gradleprops would do that for a second but that's something different.

jakubgs on 8 Aug 2019

Our main issue is that we need to control the exact build command. Fdroid tries to run Gradle for us, but what we want to do is to run it via Nix, like we do here:
https://github.com/status-im/status-react/blob/c49e4933eeea8e02e2724b5eab27d046e8c4e549/Makefile#L113-L123
Which runs the Nix derivation from here:
https://github.com/status-im/status-react/blob/c49e4933eeea8e02e2724b5eab27d046e8c4e549/nix/mobile/android/targets/release-android.nix#L14-L27
If we don't run Gradle within Nix then the build will NOT be deterministic.

jakubgs on 8 Aug 2019

I could not find any issues about running a custom Gradle command with fdroid utility:
https://gitlab.com/fdroid/fdroidserver/issues?scope=all&utf8=%E2%9C%93&state=opened&search=gradle
Might have to open one. But before that I will review the metadata doc one more time.

jakubgs on 8 Aug 2019

Progress! By specifying the subdir to be android I've managed to get past the clean step.
Now I'm getting something different:

FAILURE: Build failed with an exception.

* Where:
Build file '/repo/build/im.status.ethereum/android/app/build.gradle' line: 3

* What went wrong:
A problem occurred evaluating project ':app'.
> Could not read script '/repo/build/im.status.ethereum/node_modules/react-native-config/android/dotenv.gradle' as it does not exist.

Which is this:
https://github.com/status-im/status-react/blob/c49e4933eeea8e02e2724b5eab27d046e8c4e549/android/app/build.gradle#L3
Which means that it can't find the node_modules which has the react-native-config package.
Normally Nix would have installed that, but that would be done when using make release-android.

jakubgs on 8 Aug 2019

It's annoying how I see no way of defining environment variables for the commands.

Also, I can't run the build as root, because then the Nix installation fails:

warning: installing Nix as root is not supported by this script!

jakubgs on 8 Aug 2019

So I have to create a user to run the build with. But to do that I have to either:

Modify the image with my own Dockerfile
Wrap fdroid command in a script that will first create user and then run it with that user

The first option sounds cleaner, but makes me worried about how this is executed on F-Droid servers.

jakubgs on 8 Aug 2019

Do you have the steps defined? Eg. Install these with sudo, then build these, then run gradle?

licaon-kter on 9 Aug 2019

Install these with sudo, then build these, then run gradle?
-- @licaon-kter

That's not how Nix works. Allow me to explain.

Nix is a functional package manager that stores all its packages under /nix/store and makes them available by running commands within a nix-shell environment, that modifies the PATH to include paths to all the necessary packages, for example, if I run make shell in our repo and examine PATH I will see:

~/work/status-react develop 17s
 lilim > make shell
Env is missing TARGET_OS, assuming no target platform. See nix/README.md for more details.
Configuring default Nix shell for target 'none'...

[nix-shell:~/work/status-react]$ echo $PATH | tr ':' '\n'
/home/sochan/work/status-react/node_modules/.bin
/nix/store/nqnxj4awgk8szyp5zbcrgxrrk08w9qw4-bash-interactive-4.4-p23/bin
/nix/store/qyp22r9zfhpiykkj57rj6jcd6iwgdccy-git-2.19.2/bin
/nix/store/qyymk6cf80hnng55l6qzmqdqc9rkcvqz-patchelf-0.9/bin
/nix/store/i82x3x0yiijkgyqkzh8ni87gspas0f48-bash-4.4-p23/bin
/nix/store/iyhgjyifzv329y8wq0m8n8zjj9xn54xi-curl-7.64.0-dev/bin
/nix/store/3sjrlbsgl3ngdxm62540nrn050il9fxd-nghttp2-1.35.1-bin/bin
/nix/store/994wjidck8w972a8x2p5lw45gnmbglbc-libkrb5-1.15.2-dev/bin
...(omitted)...

As you can see the environment within the shell is modified in such way to not use things installed on the system itself. In addition to that the shell(or the Nix build process) can be "pure", meaning nothing is pulled into the build process that wasn't defined in Nix expressions, no network access is allowed, and the env is modified as well to produce fully deterministic builds.

For that reason we build the Android app by running Gradle __within__ Nix build process. Which is why I'm trying to figure out how to customize exactly what build commands will be executed.

I have tried this:

Categories:
  - Money
License: MPL-2.0
SourceCode: https://github.com/status-im/status-react
IssueTracker: https://github.com/status-im/status-react/issues

RepoType: git
Repo: https://github.com/status-im/status-react

Builds:
  - versionName: 'develop'
    versionCode: 9999
    commit: 'develop'
    buildjni: no
    subdir: android
    gradle:
      - yes
    init:
      # Install Nix
      - ../scripts/setup
    build:
      # Prepare the JS bundle
      - make jsbundle-android
      # Assemble the release APK
      - make release-android
    output: result/app.apk

# Our version tags look like `0.13.0-mobile`
AutoUpdateMode: 'Version %v-mobile'
UpdateCheckMode: Tags

The command make release-android is what calls Nix which in turn calls Gradle andproduces the final result/app.apk.

But that config results in the failures on some clean step due to issues explained here:

DEBUG: Directory: build/im.status.ethereum/android
DEBUG: > /fdroidserver/gradlew-fdroid clean

...(omitted)...

FAILURE: Build failed with an exception.

* Where:
Build file '/repo/build/im.status.ethereum/android/app/build.gradle' line: 3

* What went wrong:
A problem occurred evaluating project ':app'.
> Could not read script '/repo/build/im.status.ethereum/node_modules/react-native-config/android/dotenv.gradle' as it does not exist.

Can I somehow disable this clean step?

jakubgs on 9 Aug 2019

Not sure, but you could just create a clean step that does nothing and outputs success.

licaon-kter on 9 Aug 2019

@licaon-kter I could? There is nothing like clean key described in the metadata doc:
https://f-droid.org/en/docs/Build_Metadata_Reference/
Is it just not documented?
EDIT: Yep:

fdroidserver.exception.MetaDataException: Unrecognised build flag 'clean' in 'metadata/im.status.ethereum.yml'

No such thing.

jakubgs on 9 Aug 2019

I didn't say it's a metadata key, it's a Gradle thing: https://docs.gradle.org/current/userguide/java_plugin.html#sec:clean

_Umm, disclaimer: I'm not a dev, so these are just bits that I see while building stuff for F-Droid._

/LE: The error says something about dotenv.gradle, that's ok?

licaon-kter on 9 Aug 2019

I managed to fix the dotenv.gradle and STATUS_GO_ANDROID_LIBDIR by adding a check on project.gradle.startParameter.taskNames here: 3a4963edfd0c7748c6136c386d1b8212913e40a4

But now I'm getting an error about the Gradle version:

FAILURE: Build failed with an exception.

* Where:
Build file '/repo/build/im.status.ethereum/android/app/build.gradle' line: 1

* What went wrong:
A problem occurred evaluating project ':app'.
> Failed to apply plugin [id 'com.android.application']
   > Minimum supported Gradle version is 5.1.1. Current version is 5.0. If using the gradle wrapper, try editing the distributionUrl in /repo/build/im.status.ethereum/android/gradle/wrapper/gradle-wrapper.properties to gradle-5.1.1-all.zip

jakubgs on 12 Aug 2019

I tried specifying the Gradle version in android/gradle/wrapper/gradle-wrapper.properties:

distributionUrl=https\://services.gradle.org/distributions/gradle-5.1.1-bin.zip
distributionSha256Sum=4953323605c5d7b89e97d0dc7779e275bccedefcdac090aec123375eae0cc798

But then I got back:

DEBUG: > /fdroidserver/gradlew-fdroid clean
Found 5.1.1 via distributionUrl
No hash for gradle version 5.1.1! Exiting...

Which is the result of this line:
https://github.com/f-droid/fdroidserver/blob/7f269be2f7f9326a08074ae39422401714c132ce/gradlew-fdroid#L25-L31
Which checks this map of hashes:
https://github.com/f-droid/fdroidserver/blob/7f269be2f7f9326a08074ae39422401714c132ce/gradlew-fdroid#L60-L125
Which clearly contains the 5.1.1 version. But as it turns out the Docker image has it's own outdated copy of the fdroidserver repo which does not contain the hash:

 lilim > d run --rm -it --entrypoint=bash fdroidbuild:latest
build@271a83073822:~$ grep -A60 get_sha /fdroidserver/gradlew-fdroid | tail -n 10
                    ["4.9"]="e66e69dce8173dd2004b39ba93586a184628bc6c28461bc771d6835f7f9b0d28" \
                    ["4.10"]="248cfd92104ce12c5431ddb8309cf713fe58de8e330c63176543320022f59f18" \
                    ["4.10.1"]="e53ce3a01cf016b5d294eef20977ad4e3c13e761ac1e475f1ffad4c6141a92bd" \
                    ["4.10.2"]="b49c6da1b2cb67a0caf6c7480630b51c70a11ca2016ff2f555eaeda863143a29" \
                    ["4.10.3"]="8626cbf206b4e201ade7b87779090690447054bc93f052954c78480fa6ed186e" \
                    ["5.0"]="6157ac9f3410bc63644625b3b3e9e96c963afd7910ae0697792db57813ee79a6" \
                    )
    [ ! ${gradle_hashes[$1]+abc} ] && exit 1
    echo "${gradle_hashes["$1"]}"
}

jakubgs on 12 Aug 2019

But passing my own version of fdroidserver repo to the container using -v results in:

Traceback (most recent call last):
  File "../fdroidserver/fdroid", line 170, in <module>
    main()
  File "../fdroidserver/fdroid", line 146, in main
    mod.main()
  File "/fdroidserver/fdroidserver/build.py", line 956, in main
    .format(path=config['sdk_path'], dirname=d))
fdroidserver.exception.FDroidException: Android SDK '/home/build/Android/Sdk' does not have 'build-tools' installed!

But that doesn't make sense because I clearly specify ANDROID_HOME to be /android, so why is it looking in /home/build/Android/Sdk?
Fixed it by mounting my Android SDK directly at /home/build/Android/Sdk, but it is weird how it's ignoring the env variable.

jakubgs on 12 Aug 2019

Now it's failing on yet another reference to node_modules:

* Where:
Build file '/repo/build/im.status.ethereum/android/app/build.gradle' line: 93

* What went wrong:
A problem occurred evaluating project ':app'.
> Could not read script '/repo/build/im.status.ethereum/node_modules/react-native/react.gradle' as it does not exist.

From here:
https://github.com/status-im/status-react/blob/acc6775a542ef112b4ed99d56406fa466981a303/android/app/build.gradle#L93
Ugh, it never fucking ends...
GOD DAMNIT

jakubgs on 12 Aug 2019

Fixed that by moving the apply to the if I added before: 1f6bdf8c80209d4e73cea4bea2617ed6f60c53c5

jakubgs on 12 Aug 2019

Okay, now I have the clean step succeeding, but it fails on due to implementation entries:

> Task :app:clean UP-TO-DATE
> Task :react-native-status:clean UP-TO-DATE

BUILD SUCCESSFUL in 5s
2 actionable tasks: 2 up-to-date
INFO: Scanning source for common problems...
ERROR: Found unknown maven repo '$rootDir/../node_modules/react-native/android' at android/build.gradle
ERROR: Found usual suspect 'firebase' at line 276 at android/app/build.gradle
ERROR: Found usual suspect 'google.*play.*services' at line 281 at android/app/build.gradle
ERROR: Found usual suspect 'firebase' at line 289 at android/app/build.gradle
ERROR: Found usual suspect 'firebase' at line 290 at android/app/build.gradle
WARNING: Found possible binary at android/app/src/debug/res/ic_launcher-web.png
ERROR: Found binary at ios/FirebaseCoreDiagnostics.framework/FirebaseCoreDiagnostics
ERROR: Found binary at ios/FirebaseInstanceID.framework/FirebaseInstanceID
ERROR: Found binary at ios/FirebaseMessaging.framework/FirebaseMessaging
ERROR: Found binary at ios/FirebaseNanoPB.framework/FirebaseNanoPB
ERROR: Found binary at ios/nanopb.framework/nanopb
ERROR: Found binary at ios/FirebaseCore.framework/FirebaseCore
ERROR: Could not build app im.status.ethereum: Can't build due to 11 errors while scanning
INFO: Finished
INFO: 1 build failed

In the dependencies section here:
https://github.com/status-im/status-react/blob/3b8d8d0013b4b2dd7cbe3b55ba44ea0f5792e29b/android/app/build.gradle#L260-L306
I'm beginning to think that I need to read some F-Droid code to figure out how exactly it calls Gradle.

It is possible that it is currently impossible to run Gradle within Nix shell in F-Droid.

jakubgs on 13 Aug 2019

you can scandelete: ios folder and in prebuild you'd sed /firebase/d and sed /play-services-base/,+2d the other stuff

licaon-kter on 13 Aug 2019

Hmmm...

scandelete: <path1>[,<path2>,...]
When running the scan process, any files that trigger errors - like binaries - will be removed. It acts just like scanignore, but instead of ignoring the files, it removes them.

Useful when a source code repository includes binaries or other unwanted files which are not needed for the build. Instead of removing them manually via rm, using scandelete is easier.

I'm not sure why I'd want to remove those things. Since I'm setting subdir: android why would the ios subdir at repo root cause issues? Makes no sense to me.

Also, if I remove firebase or anything else during prebuild that will just cause the eventual build to fail. Makes no sense either. I appreciate your help, but I don't get how that's supposed to work.

I think my issue is more fundamental than that. I need to figure out how I can customize exactly how fdroid calls gradle, because I want to run it within Nix, and I also don't want to use Gradle Wrapper.

jakubgs on 13 Aug 2019

The scanner runs first, it can't know what the final build will have or not, just that that ios stuff might be some unwanted binaries. It's a "hammer" not a "scalpel", it has no AI :), you just guide it, eg. If you don't need the ios folder for the Android build there's no harm in removing those, right? Or at least scanignore ;)

Since Firebase is not open-source, you are forced to remove it, either here blunt by sed, or maybe condition it using gradle logic and (if the scanner complains) just scanignore the whole build.gradle file, knowing that eventually it is not actually used.

licaon-kter on 13 Aug 2019

Oh, I wasn't aware that Firebase blocks use from being included in F-Droid. That's the first time I hear about that. I will have to discuss this with the team, thanks for the explanation @licaon-kter!

jakubgs on 13 Aug 2019

Yeah, F-Droid builds only open-source apps, with open-source dependencies.

Now, is Firebase a hard dep? Does the app work without it?

If not, you can always host your own repo and F-Droid users can add it and keep up to date. ;)

licaon-kter on 13 Aug 2019

I've created #8742 to see what our developers think about removing Firebase or making it optional.

jakubgs on 13 Aug 2019

👍2

Thanks to work by Eric in https://github.com/status-im/status-react/pull/9308 Firebase has been removed from status-react so I can return to working on this in the nearby future.

jakubgs on 29 Oct 2019

🎉5 👍2

I've gotten back into this and some notes to myself:

I need fdroidclient cloned and mounted
I need mounted Android SDK into the docker container to make it work
Need to have ANDROID_HOME environment variables set in the container
The DroidServer requries build-tools and platform-tools to be installed via sdkmanager
Needs app definition in metadata/im.status.ethereum.yml
Needs app version specified as first build argument

I still don't get why fdroidclient has to be mounted and not fdroiddata...

jakubgs on 19 Feb 2020

Client not Server? Sounds strange...

licaon-kter on 20 Feb 2020

According to the docs building from source is not the only way to get into F-Droid:

In the client, each package for an app is labelled either ‘source’ or ‘bin’. These correspond to two different messages you’ll see for the same package when browsing the repository on the web site, either “This version is built and signed by F-Droid, and guaranteed to correspond to the source tarball below” or “This version is built and signed by the original developer.”

Most packages are source builds. Those that are __binaries are usually that way for technical or historical reasons.__ Ideally everything in the F-Droid repo would be a source build.

Source builds are built and signed securely by F-Droid, and are accompanied on the web site by a complete and corresponding copy of the exact source code we built it with. It may include some prebuilt libraries and you may need to download some external libraries, but as long as those libraries are still on the internet, you should be able to build an exact copy.

__Binary packages are built and signed by the developer. We get them from public sources that the developer makes available directly, and never from third party sources.__

https://f-droid.org/en/docs/FAQ_-_General/#whats-the-difference-between-source-and-binary-builds

So if the F-Droid build process doesn't work for us there is still a possibility of publishing by just providing them with APKs.

See also:

__Will my app be built from source?__

Yes. In all but a very few cases (for technical or historical reasons), we build all applications directly from the source code. This guarantees that the source code is available for the version people have installed. While we’re not suggesting that your source code is incomplete, not published or out of date, this happens a lot.

https://f-droid.org/en/docs/FAQ_-_App_Developers/#will-my-app-be-built-from-source

jakubgs on 25 Feb 2020

There exists an F-Droid Forum that I could try posting to to figure out if building via Nix is even doable. It might take much more research and work to figure out that it isn't than just compiling a post explaining our issues and asking if this can be made to work.

jakubgs on 25 Feb 2020

Maybe that section needs some modifications.

We don't do that, unless the app builds reproducible. But... we need to be able to build it too, and verify of course.

licaon-kter on 25 Feb 2020

Well, my understanding is that F-Droid is very opinionated about how Gradle should be used to build the App, but in our case we run Gradle through Nix derivations, like so:
https://github.com/status-im/status-react/blob/e5f7a94ee29382236063b81ce9dccaa0a4f2f880/nix/mobile/android/targets/release-android.nix#L124

From my reading of docs my understanding was that F-Droid server wants to run Gradle directly, which makes our process incompatible. Is that incorrect?

jakubgs on 25 Feb 2020

When I try to run via docker inside of the fdroiddata folder like so:

 $ docker run --rm -u $(id -u):$(id -g) \                                                                    
  -w /repo -v $(pwd)/../fdroiddata:/repo -v $HOME/Android:/android -e ANDROID_HOME=/android -e HOME=/tmp \
  registry.gitlab.com/fdroid/docker-executable-fdroidserver:latest \
  build im.status.ethereum:2020021421 -v -l

I get back a nonsense error:

Traceback (most recent call last):
  File "../fdroidserver/fdroid", line 162, in <module>
    main()
  File "../fdroidserver/fdroid", line 138, in main
    mod.main()
  File "/fdroidserver/fdroidserver/build.py", line 991, in main
    allapps = metadata.read_metadata(not options.onserver, pkgs, options.refresh, sort_by_time=True)
  File "/fdroidserver/fdroidserver/metadata.py", line 814, in read_metadata
    check_metadata(app)
  File "/fdroidserver/fdroidserver/metadata.py", line 493, in check_metadata
    v.check(app[k], app.id)
  File "/fdroidserver/fdroidserver/metadata.py", line 434, in check
    .format(value=v, field=self.name, appid=appid, pattern=self.matching))
  File "/fdroidserver/fdroidserver/metadata.py", line 45, in warn_or_exception
    raise MetaDataException(value)
fdroidserver.exception.MetaDataException: 'bc1qu482c0tngkcvx3q7mrm8zmuldrh2f2lrh26ym0' is not a valid Bitcoin address in org.projectmaxs.transport.xmpp. Regex pattern: ^[a-zA-Z0-9]{27,34}

I don't get what a regex for a Bitcoin address has to do with me building an Ethereum app.
According to this guide:
https://f-droid.org/en/docs/Submitting_to_F-Droid_Quick_Start_Guide/

I'm supposed to run fdroid build in the fdroiddata folder:

cd ../fdroiddata
...
fdroid build --server -v -l net.artificialworlds.rabbitescape

But that clearly doesn't work with the container.

jakubgs on 25 Feb 2020

I can't even build org.fdroid.fdroid:16 as suggested in:
https://f-droid.org/en/docs/Building_Applications/

$ docker run --rm -u $(id -u):$(id -g) \
  -w /repo -v $(pwd)/../fdroiddata:/repo -v $HOME/Android:/android -e ANDROID_HOME=/android -e HOME=/tmp \
  registry.gitlab.com/fdroid/docker-executable-fdroidserver:latest \
  build org.fdroid.fdroid:16 -v -l          
WARNING: No 'config.py' found, using defaults.
Traceback (most recent call last):
  File "../fdroidserver/fdroid", line 162, in <module>
    main()
  File "../fdroidserver/fdroid", line 138, in main
    mod.main()
  File "/fdroidserver/fdroidserver/build.py", line 991, in main
    allapps = metadata.read_metadata(not options.onserver, pkgs, options.refresh, sort_by_time=True)
  File "/fdroidserver/fdroidserver/metadata.py", line 814, in read_metadata
    check_metadata(app)
  File "/fdroidserver/fdroidserver/metadata.py", line 493, in check_metadata
    v.check(app[k], app.id)
  File "/fdroidserver/fdroidserver/metadata.py", line 434, in check
    .format(value=v, field=self.name, appid=appid, pattern=self.matching))
  File "/fdroidserver/fdroidserver/metadata.py", line 45, in warn_or_exception
    raise MetaDataException(value)
fdroidserver.exception.MetaDataException: 'bc1qu482c0tngkcvx3q7mrm8zmuldrh2f2lrh26ym0' is not a valid Bitcoin address in org.projectmaxs.transport.xmpp. Regex pattern: ^[a-zA-Z0-9]{27,34}$

How is this supposed to work?

jakubgs on 25 Feb 2020

Update fdroidserver and fdroiddata from Gitlab directly, this address error has been long fixed.

You can bypass gradle, don't include this section

gradle:
  - yes

But run it in the build: section.

licaon-kter on 25 Feb 2020

Okay, so I can avoid F-Droid trying to run Gradle. That's great news, thanks!

But what about Nix? We require root permissions for sudo to install Nix so we can use it to build the App. I assume that wouldn't fly on F-Droid build servers. Or would it?

jakubgs on 25 Feb 2020

Okay, I pulled most recent changes for fdroiddata, but what about fdroidserver?

I'm looking at the readme for the docker container:
https://gitlab.com/fdroid/docker-executable-fdroidserver

And it seems like thae last commit was 1 year ago. And the instructions suggest that I should run:

$ docker run --rm \
  -u $(id -u):$(id -g) \
  -v $(pwd):/repo \
  registry.gitlab.com/fdroid/docker-executable-fdroidserver:latest

But it doesn't say what /repo is supposed to be, but I assume it's supposed to be fdroiddata, because how else would it get the yml file that defines how to build the app. How am I supposed to provide it with an up-to-date fdroidserver?

jakubgs on 25 Feb 2020

Oh, I see, it clones fdroidserver at root:
https://gitlab.com/fdroid/docker-executable-fdroidserver/-/blob/master/Dockerfile#L8
Which means I can try mounting my own clone at /fdroidserver.

jakubgs on 25 Feb 2020

Okay, by using -v $(pwd)/fdroidserver:/fdroidserver with the docker command I've made progress. Now it fails with:

INFO: Creating local.properties file at build/im.status.ethereum/local.properties
INFO: Creating local.properties file at build/im.status.ethereum/android/local.properties
DEBUG: Removing debuggable flags from build/im.status.ethereum/android
INFO: Cleaned build.gradle of keysigning configs at build/im.status.ethereum/android/app/build.gradle
INFO: Scanning source for common problems...
ERROR: Found binary at ios/nanopb.framework/nanopb
WARNING: Found possible binary at desktop/resources/icon_phone_white.png
WARNING: Found possible binary at desktop/resources/icon_money_white.png
WARNING: Found possible binary at desktop/resources/ic_stat_status_notification.png
WARNING: Found possible binary at resources/images/ui/new-chat-header.png
WARNING: Found possible binary at resources/images/ui/[email protected]
WARNING: Found possible binary at resources/images/ui/[email protected]
ERROR: Found unknown maven repo '(' at android/build.gradle
ERROR: Found unknown maven repo '(' at android/build.gradle
WARNING: Found possible binary at android/app/src/debug/res/ic_launcher-web.png
ERROR: Could not build app im.status.ethereum: Can't build due to 3 errors while scanning
INFO: Finished
INFO: 1 build failed
DEBUG: skip deploying full build logs: not enabled in config

jakubgs on 25 Feb 2020

It's not clear which line it doesn't like in android/build.gradle, but it might be this:
https://github.com/status-im/status-react/blob/9f2e2f9d3e0d7b1514ec6159bb08c532c72665b3/android/build.gradle#L63-L64

jakubgs on 25 Feb 2020

I wonder if there's a way to make fdroid build use a specific folder for its builds, instead of cloning every time. That way I could make modifications easier instead of committing them.

jakubgs on 25 Feb 2020

I can fix the:

ERROR: Found binary at ios/nanopb.framework/nanopb

By adding scanignore to the build metadata:

Builds:
  - versionName: '1.0.1'
    subdir: android
    scanignore:
      # repo holds iOS build files too
      - 'ios'

jakubgs on 25 Feb 2020

According to this issue:

That's the intended behavior. F-Droid cannot verify that artifacts in a maven repo are free software.
https://gitlab.com/fdroid/fdroidserver/issues/504

But it complaining specifically about '(' being a maven repo looks like an error with parsing of build.gradle.

jakubgs on 25 Feb 2020

No comments in .YML :)

If that's not for Android, scandelete: is better.

It's not clear which line it doesn't like

You can reach inside the docker and check, right?

But what about Nix? We require root permissions for sudo to install Nix so we can use it to build the App. I assume that wouldn't fly on F-Droid build servers. Or would it?

The build runs in a VM that gets created/destroyed as needed, there is sudo: see https://f-droid.org/en/docs/Build_Metadata_Reference/#Build

You can always grep fdroiddata and find examples. ;)

Eg. we usually do

sudo:
  - apt-get update || apt-get update
  - apt-get install <whatever>

...(the doubling is to bypass some glitches) you can install stuff from either stretch or if needed with -t stretch-backports

licaon-kter on 25 Feb 2020

But it complaining specifically about '(' being a maven repo looks like an error with parsing of

I guess you sed something and that breaks it. Do attach the recipe so far in a gist or something.

licaon-kter on 25 Feb 2020

Okay, I think what's more likely is that fdroid is getting confused by:
https://github.com/status-im/status-react/blob/9f2e2f9d3e0d7b1514ec6159bb08c532c72665b3/android/build.gradle#L54-L61
It would make sense since the url() specification of local paths for Maven modules would be unknown to fdroid and not verifiable.

jakubgs on 25 Feb 2020

@licaon-kter here's what I'm working with right now:
https://gist.github.com/jakubgs/af78e6baf01d0d21a2c9c18208ee4907

jakubgs on 25 Feb 2020

No comments in .YML :)

Not sure what you're talking about, YML has comments in the spec my dude:
https://yaml.org/spec/1.2/spec.html#id2780069

jakubgs on 25 Feb 2020

You can reach inside the docker and check, right?

How so? It's not telling me which line it complains about, why would looking inside the docker container help? I'm not seding anything at all.

The build runs in a VM that gets created/destroyed as needed, there is sudo

That's good to hear, thanks.

jakubgs on 25 Feb 2020

Also, I just noticed that we will need to build an unsigned version of the app:

output: glob/to/output.apk
Specify a glob __path where the resulting unsigned release apk from the build should be__. This can be used in combination with build methods like gradle=yes or maven=yes, but if no build method is specified, the build is manual. You should run your build commands, such as make, in build.
https://f-droid.org/en/docs/Build_Metadata_Reference/

jakubgs on 25 Feb 2020

Not sure what you're talking about, YML has comments in the spec my dude:

Ok, fight with the linter then fdroid lint -v im.status.ethereum ;)

How so? It's not telling me which line it complains about, why would looking inside the docker container help? I'm not seding anything at all.

Look inside at the file... see if it has a dangling ( if that's the error.

licaon-kter on 25 Feb 2020

It would make sense since the url() specification of local paths for Maven modules would be unknown to fdroid and not verifiable.

I see this here too: https://gitlab.com/staltz/manyverse/-/blob/master/android/build.gradle#L30 and builds fine with this recipe: https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/se.manyver.yml#L1193

licaon-kter on 25 Feb 2020

Also, I just noticed that we will need to build an unsigned version of the app:

Yes, so (I guess) remove https://github.com/status-im/status-react/blob/develop/android/app/build.gradle#L247 and line 250 via a sed in prebuild:

licaon-kter on 25 Feb 2020

Okay, I have verified using this:

    prebuild:
       - sed -i -e '54,61d' android/build.gradle

That the issue is indeed this section in android/build.gradle:
https://github.com/status-im/status-react/blob/9f2e2f9d3e0d7b1514ec6159bb08c532c72665b3/android/build.gradle#L54-L61
Which got me to the build stage, which failed due to lack of Nix:

INFO: Scanning source for common problems...
INFO: Removing binary at ios/nanopb.framework/nanopb
WARNING: Found possible binary at android/app/src/debug/res/ic_launcher-web.png
INFO: Creating source tarball...
INFO: Running 'build' commands in build/im.status.ethereum
DEBUG: Directory: build/im.status.ethereum
DEBUG: > bash -x -c make jsbundle-android && make release-android
+ make jsbundle-android
Nix not available, sourcing profile failed!

jakubgs on 25 Feb 2020

Ok, fight with the linter then fdroid lint -v im.status.ethereum ;)

I don't have to, there's no errors:

 $ docker run --rm -u $(id -u):$(id -g) \
  -w /repo -v $(pwd)/fdroiddata:/repo -v $(pwd)/fdroidserver:/fdroidserver -v $HOME/Android:/android -e ANDROID_HOME=/android -e HOME=/tmp \
  registry.gitlab.com/fdroid/docker-executable-fdroidserver:latest \
  lint -v im.status.ethereum
WARNING: No 'config.py' found, using defaults.

 $ echo $?
0

Look inside at the file... see if it has a dangling ( if that's the error.

There isn't, I was not seding the file, as I said. It's fine. As I said in the previous comment, the issue appears to be in:
https://github.com/status-im/status-react/blob/9f2e2f9d3e0d7b1514ec6159bb08c532c72665b3/android/build.gradle#L54-L61

builds fine with this recipe:

Well, if you look at the metadata it actually tells F-Droid to ignore that file:

    scanignore:
      - android/build.gradle

https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/se.manyver.yml#L487

jakubgs on 25 Feb 2020

Also, regarding comments, the docs state that:

scanignore: <path1>[,<path2>,...]
Enables one or more files/paths to be excluded from the scan process. This should only be used where there is a very good reason, and probably __accompanied by a comment explaining why it is necessary__.
https://f-droid.org/en/docs/Build_Metadata_Reference/

So it seems like the F-Droid documentation itself recommends adding comments to the metadata YML file.

jakubgs on 25 Feb 2020

Also:

Note that although the metadata files are designed to be easily read and writable by humans, they are also processed and written by various scripts. They can be automatically cleaned up when necessary. __The structure and comments will be preserved correctly, although the order of fields will be standardised.__
https://f-droid.org/en/docs/Build_Metadata_Reference/

jakubgs on 25 Feb 2020

Better use

 prebuild:
       - sed -i -e '/signingConfig\ signingConfigs/d' android/build.gradle

...cause lines tend to move.

I don't have to, there's no errors:
So it seems like the F-Droid documentation itself recommends adding comments to the metadata YML file.

I've seen issues with lint, but whatever.

Well, if you look at the metadata it actually tells F-Droid to ignore that file:

So that's the magic sauce then.

licaon-kter on 25 Feb 2020

Why not scandelete both ios and Desktop?

licaon-kter on 25 Feb 2020

Actually, setting:

    scanignore:
      - android/build.gradle

Results in:

ERROR: Unused scanignore path: android/build.gradle

So I don't get how that works for the app you linked

jakubgs on 25 Feb 2020

Also, I don't intend to use any sed magic. That's just awful. I'm just using some right not to debug issues so I don't have to commit every time I want to test something in the repo.

I want to create a specific target that will result in an unsigned APK. Using sed to make your app build correctly sounds like just asking for major trouble.

jakubgs on 25 Feb 2020

Still not sure what I'm supposed to do about:
https://github.com/status-im/status-react/blob/9f2e2f9d3e0d7b1514ec6159bb08c532c72665b3/android/build.gradle#L54-L61
Removing it is not an option, that's just going to break the build. And using scanignore makes it fail with the error above.

jakubgs on 25 Feb 2020

Looks like I can't install Nix if the user under which UID I'm running doesn't exist:

id: cannot find name for user ID 1000
/tmp/nix-binary-tarball-unpack.J3Z7kllsYP/unpack/nix-2.3.2-x86_64-linux/install: $USER is not set
Failed to install Nix package manager!

jakubgs on 25 Feb 2020

And I can't use sudo inside of the container because it doesn't exist:

DEBUG: > bash -x -c -- sudo useradd -u ${UID} builder && nix/scripts/setup.sh
+ sudo useradd -u 1000 builder
bash: sudo: command not found

jakubgs on 25 Feb 2020

And I can't use su because of lack of terminal?

DEBUG: > bash -x -c -- su root -c "useradd -u ${UID} builder" && nix/scripts/setup.sh
+ su root -c 'useradd -u 1000 builder'
su: must be run from a terminal

Which I assume it wants because UID 1000 doesn't have permissions to su and wants password.

jakubgs on 25 Feb 2020

Oh, but there's a special sudo metadata field:

sudo: xxxx
Specifies a script to be run using sudo bash -x -c "xxxx" in the buildserver VM guest. This script is run with full root privileges, but the state will be reset after each build. The vast majority of apps build using the standard Debian/stable base environment. This is useful for setting up the buildserver for complex builds that need very specific things that are not appropriate to install for all builds, or for things that would conflict with other builds.
https://f-droid.org/en/docs/Build_Metadata_Reference/

jakubgs on 25 Feb 2020

I want to create a specific target that will result in an unsigned APK. Using sed to make your app build correctly sounds like just asking for major trouble.

That's better yes 👍

So I don't get how that works for the app you linked

subdir: android means it starts in this folder so, change to:

    scanignore:
      - build.gradle

...right?

Oh, but there's a special sudo metadata field:

Already said that...yes...scroll up.

licaon-kter on 25 Feb 2020

I tried to fix the Nix installation error by using -e USER=builder and adding:

    sudo:
      # Required for Nix
      - useradd -u ${UID} builder
      - mkdir -m 0755 /nix
      - chown builder /nix

But that fails because:

WARNING: im.status.ethereum:1.0.1 runs this on the buildserver with sudo:
    useradd -u ${UID} builder && mkdir -m 0755 /nix && chown builder /nix
These commands were skipped because fdroid build is not running on a dedicated build server.

So using a Docker container might not work for this. What a nightmare.

subdir: android means it starts in this folder so, change to:

I already removed that, here's how it looks right now:
https://gist.github.com/jakubgs/673af2de1db0a4c9573bc5b13d4a4c99

Already said that...yes...scroll up.

You did indeed, I thought you meant I could use sudo not that there's a special key. One thing at a time though.

jakubgs on 25 Feb 2020

Besides, the example you linked uses subdir anyway:
https://gitlab.com/fdroid/fdroiddata/-/blob/master/metadata/se.manyver.yml#L502

jakubgs on 25 Feb 2020

These commands were skipped because fdroid build is not running on a dedicated build server

Then, use the Docker commands to setup whatever would need to be done in sudo and document it. We'll test on a proper server asap.

Besides, the example you linked uses subdir anyway:

I'm baffled by the folders used too sometimes.

licaon-kter on 25 Feb 2020

So using a Docker container might not work for this. What a nightmare.

Try with fdroid build --on-server ...

Ref: https://gitlab.com/fdroid/fdroidserver/-/blob/master/fdroidserver/build.py#L887

licaon-kter on 25 Feb 2020

Okay, so I made a new image using this Dockerfile:

FROM registry.gitlab.com/fdroid/docker-executable-fdroidserver:latest

RUN useradd -u 1000 builder
RUN mkdir -m 0755 /nix && chown builder /nix

And now Nix is installing properly:

+ nix/scripts/setup.sh
Setting up Nix package manager...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2399  100  2399    0     0  11003      0 --:--:-- --:--:-- --:--:-- 11055
downloading Nix 2.3.2 binary tarball for x86_64-linux from 'https://nixos.org/releases/nix/nix-2.3.2/nix-2.3.2-x86_64-linux.tar.xz' to '/tmp/nix-binary-tarball-unpack.lw37xEdB08'...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15.1M  100 15.1M    0     0  5720k      0  0:00:02  0:00:02 --:--:-- 5719k
performing a single-user installation of Nix...
copying Nix to /nix/store................................
installing 'nix-2.3.2'
building '/nix/store/mbxx0583rkga0xs7mml8yhzra080pw3b-user-environment.drv'...
created 6 symlinks in user environment
unpacking channels...
created 1 symlinks in user environment

Installation finished!  To ensure that the necessary environment
variables are set, please add the line

  . /tmp/.nix-profile/etc/profile.d/nix.sh

to your shell profile (e.g. ~/.profile).
The Nix package manager was successfully installed.
See STARTING_GUIDE.md if you're new here.

Progress.

jakubgs on 25 Feb 2020

Oh:

  --on-server           Specify that we're running on the build server

I guess that could work too. But using a modified image is fine too for now.

jakubgs on 25 Feb 2020

Okay, now it's failing on lack of space:

building '/nix/store/2z5mxw9ls0qrgams36jjxf02gwm4vx8n-status-react-patched-npm-gradle-modules.drv'...
unpacking sources
preallocating file of 22758792 bytes: No space left on device
error: build of '/nix/store/0gbcrcsv7mf7ixyqxk3vq5vhvkwbkis6-androidsdk.drv', '/nix/store/2z5mxw9ls0qrgams36jjxf02gwm4vx8n-status-react-patched-npm-gradle-modules.drv', '/nix/store/ahsk5k88yxis1ws4h2spp06zfl7xfli7-licensed-android-sdk.drv' failed
Makefile:168: recipe for target 'release-android' failed

I guess the Docker volume has some kind of size limitation.

jakubgs on 25 Feb 2020

I also opened this thread on the F-Droid forum:
https://forum.f-droid.org/t/can-i-build-an-app-for-f-droid-via-nix/8839
But you answered some of those questions already, so I'll post an update there.

jakubgs on 25 Feb 2020

Hello! What's the current status of this issue? Many of us are eager to see Status on F-Droid.

cyanlemons on 25 Apr 2020

👍1

Other priorities. I do intent to get back to it.

For now you can just install an APK from https://status.im/.

jakubgs on 26 Apr 2020

Congratulations to reproducible builds: https://our.status.im/reproducable-builds-with-nix/
-> Great work!

I wonder if this brings us a step closer to a submission to the f-droid store, does their build process support building with Nix?

mulles on 12 Jun 2020

It should. I just have to get back to dealing with F-Droid, and being alone in this kinda means I have many other priorities.

jakubgs on 15 Jun 2020

👍2

I managed get Nix to start building with the following Dockerfile which helps with my local permission issues:

FROM registry.gitlab.com/fdroid/docker-executable-fdroidserver:latest
ENV USER=builder
RUN useradd -u 1000 $USER
RUN mkdir -m 0755 /nix && chown $USER /nix

And it should not be necessary when this runs on F-Droid build servers.

And following metadata/im.status.ethereum.yml:

Categories:
  - Money
License: MPL-2.0
SourceCode: https://github.com/status-im/status-react
IssueTracker: https://github.com/status-im/status-react/issues

RepoType: git
Repo: https://github.com/status-im/status-react

Builds:
  - versionName: 'v1.4.1'
    versionCode: 9999
    commit: 'fdroid'
    buildjni: no
    subdir: android
    gradle: []
    scandelete:
      # iOS files are irrelevant to Android builds
      - ios
    scanignore:
      # We reference relative node_modules using ../
      - android/build.gradle
    sudo:
      # Required for Nix
      - mkdir -m 0755 /nix
      - chown builder /nix
    init:
      # Install Nix
      - ../nix/scripts/setup.sh
    build:
      - cd ../
      # Assemble the release APK
      - make release-android
    output: result/app.apk

# Our version tags look like `0.13.0-mobile`
AutoUpdateMode: 'Version %v-mobile'
UpdateCheckMode: Tags

But right now I'm getting:

curl: (23) Failed writing body (124 != 16375)
error: cannot download android-ndk-r21-linux-x86_64.zip from any mirror
error: writing to file: No space left on device
Makefile:165: recipe for target 'release-android' failed

Because the default container size is 10 GB:

 > docker info | grep 'Base Device Size'
  Base Device Size: 10.74GB

I should be able to adjust this using the dm.basesize setting.

Next step after that would be to prepare a Gradle/Nix target that makes an unsigned build.

jakubgs on 15 Jul 2020

By adding --storage-opt dm.basesize=30G to Docker daemon flags and --storage-opt size=30G to docker run flags I've managed to fix the No space on device issue:

 > docker run \
      --rm -u $(id -u):$(id -g) \
      --storage-opt size=30G \
      -w /repo -v $(pwd)/fdroiddata:/repo \
      -v $(pwd)/fdroidserver:/fdroidserver \
      -v $HOME/Android:/android \
      -e ANDROID_HOME=/android \
      -e HOME=/tmp -e USER=builder \
      statusteam/docker-executable-fdroidserver:latest \
      build -v im.status.ethereum

Seems to work up until we get to Gradle being called by Nix:

building '/nix/store/l22za2b07yrfmj8crfxv2y2h1hydn91a-status-react-build-release-android.drv'...
unpacking sources
secretsPhase
secretsCheckPhase
keystorePhase
building
/tmp/tmp-status-react-3171f1be9/nix-build-status-react-build-release-android.drv-0/android /tmp/tmp-status-react-3171f1be9/nix-build-status-react-build-release-android.drv-0

FAILURE: Build failed with an exception.

* What went wrong:
Failed to load native library 'libnative-platform.so' for Linux amd64.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org
builder for '/nix/store/l22za2b07yrfmj8crfxv2y2h1hydn91a-status-react-build-release-android.drv' failed with exit code 1

jakubgs on 15 Jul 2020

Threads like these suggests settings GRADLE_USER_HOME:

But it doesn't seem to help.
What baffles me is that this is a __pure__ Nix build, so the fact that it runs within some container should not affect it.

jakubgs on 16 Jul 2020

This comment clearly says that the issue is lack of permissions to access GRADLE_USER_HOME: https://github.com/gradle/gradle/issues/1493#issuecomment-344715852
I tried setting it to /tmp in Nix it works locally but fails in container.
This means the container has to have broken /tmp permissions. But when I check inside the container I see:

drwxrwxrwt   5 root    root      104 Jul 17 18:11 tmp

So with 777 as permissions this should not be happening.

jakubgs on 17 Jul 2020

I managed to fix the issue by setting:

export GRADLE_USER_HOME=$(mktemp -d)

But then I got this error:

Starting a Gradle Daemon (subsequent builds will be faster)

> Configure project :app
Unable to initialize metrics, ensure /home/builder/.android is writable, details: /home/builder/.android/analytics.settings (No such file or directory)

FAILURE: Build failed with an exception.

So I also had to set:

export ANDROID_SDK_HOME=$(mktemp -d)

jakubgs on 20 Jul 2020

Now I'm getting a new kind of error:

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/tmp-status-react-9950183a5/nix-build-status-react-build-release-android.drv-0/node_modules/react-native-reanimated/android/build.gradle' line: 89

* What went wrong:
A problem occurred configuring project ':react-native-reanimated'.
> Failed to find target with hash string 'android-29' in: /android

Which appears to be the first classpath appending line:

    task androidJavadoc(type: Javadoc) {
        source = android.sourceSets.main.java.srcDirs
        classpath += files(android.bootClasspath)
        classpath += files(project.getConfigurations().getByName('compile').asList())
        include '**/*.java'
    }

https://github.com/software-mansion/react-native-reanimated/blob/715f43234465c710d942d341a592998363d7a452/android/build.gradle#L89

jakubgs on 20 Jul 2020

I'm baffled by how the fdroid build environment can affect the Android Nix build considering it's a pure build:
https://github.com/status-im/status-react/blob/9950183a50efa1d3ff2291845ff6d0447df8ab0d/nix/scripts/build.sh#L49-L55

jakubgs on 20 Jul 2020

Interestingly enough, when I dropped the -e ANDROID_HOME=/android flag from the container the error changed:

> Failed to find target with hash string 'android-29' in: /opt/android-sdk

Which matches the default ANDROID_HOME set in the container:

 > docker run -it --entrypoint=/bin/bash statusteam/docker-executable-fdroidserver:latest                 
root@ae0a91bc3b9f:/repo# echo $ANDROID_HOME
/opt/android-sdk

Which indicated that somehow Nix pure build is inheriting the value of ANDROID_HOME. Which makes no sense.

jakubgs on 20 Jul 2020

The ANDROID_HOME must have been set using the ENV directing in the original Dockerfile:

 > docker inspect registry.gitlab.com/fdroid/docker-executable-fdroidserver:latest | jq '.[].Config.Env'        
[
  "PATH=/opt/android-sdk/tools:/opt/android-sdk/platform-tools:/opt/android-sdk/build-tools/25.0.2:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "LANG=C.UTF-8",
  "DEBIAN_FRONTEND=noninteractive",
  "ANDROID_HOME=/opt/android-sdk"
]

jakubgs on 20 Jul 2020

I added env | grep '^ANDROID to the Nix derivation and got this:

ANDROID_ABI_SPLIT=false
ANDROID_ABI_INCLUDE=armeabi-v7a;arm64-v8a;x86
ANDROID_SDK_ROOT=/nix/store/agjwnx35bvraps2a38pfjawxby77wv7y-licensed-android-sdk
ANDROID_NDK_ROOT=/nix/store/agjwnx35bvraps2a38pfjawxby77wv7y-licensed-android-sdk/ndk-bundl

So I'm sure ANDROID_HOME is not set I the build shell of Nix. Which means Gradle gets it from somewhere else.

jakubgs on 20 Jul 2020

I'm trying to strace -f --trace=%file the Gradle call to find what kind of files its reading.
I think I might be abel to find where it's getting the ANDROID_HOME=/opt/android-sdk setting from, if not from the environment.
The current issue is that the build stops after the gradle call fails, but does not exit. I need to check this.

jakubgs on 27 Jul 2020

I found some examples that looked promising but those files do not exist on the container and are created by Gradle:

[pid   933] openat(AT_FDCWD, "/home/builder/.android/analytics.settings", O_RDWR|O_CREAT, 0666) = 258
[pid   933] openat(AT_FDCWD, "/home/builder/.android/build-cache/3.5.3/.cache-use-marker", O_RDWR|O_CREAT|O_EXCL, 0666) = -1 EEXIST (File exists)

jakubgs on 27 Jul 2020

I added some prints for env variables in android/build.gradle to verify their values like so:

allprojects {
    beforeEvaluate {
      println("ANDROID_HOME: ${System.env.ANDROID_HOME}")
      println("ANDROID_SDK_ROOT: ${System.env.ANDROID_SDK_ROOT}")
      println("ANDROID_NDK_ROOT: ${System.env.ANDROID_NDK_ROOT}")

And I got back this:

> Configure project :react-native-reanimated
ANDROID_HOME: /nix/store/agjwnx35bvraps2a38pfjawxby77wv7y-licensed-android-sdk
ANDROID_SDK_ROOT: /nix/store/agjwnx35bvraps2a38pfjawxby77wv7y-licensed-android-sdk
ANDROID_NDK_ROOT: /nix/store/agjwnx35bvraps2a38pfjawxby77wv7y-licensed-android-sdk/ndk-bundle

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/tmp-status-react-efa95e5ce/nix-build-status-react-build-release-android.drv-0/node_modules/react-native-reanimated/android/build.gradle' line: 89

* What went wrong:
A problem occurred configuring project ':react-native-reanimated'.
> Failed to find target with hash string 'android-29' in: /opt/android-sdk

So ANDROID_HOME is clearly not set via environment variables.

jakubgs on 27 Jul 2020

It's worth noting that I didn't see it opening anything in /nix/store/agjwnx35bvraps2a38pfjawxby77wv7y-licensed-android-sdk.

jakubgs on 27 Jul 2020

What's interesting is that I can see it reading it much earlier than the error:

20:27:38.692 [INFO] [org.gradle.api.Project] Parsing /opt/android-sdk/build-tools/26.0.2/package.xml

jakubgs on 27 Jul 2020

I just realized that we don't disable Gradle daemon by default in Nix builds. I could not understand how it was getting the env variables of the container, but it was clearly via a connection to the deamon that was started outside of Nix.

I added -Dorg.gradle.daemon=false to the call and now it's building:

BUILD SUCCESSFUL in 6m 21s
867 actionable tasks: 867 executed
running tests
  8817416  00-00-1980 00:00   assets/index.android.bundle
installing

Extracting result: /nix/store/x5vxkfpdwppi3lx9accrdbms0dcwirx6-status-react-build-release-android
'/nix/store/x5vxkfpdwppi3lx9accrdbms0dcwirx6-status-react-build-release-android/app-release-unsigned.apk' -> './result/app-release-unsigned.apk'

SUCCESS
2020-07-27 21:09:54,137 INFO: Successfully built version v1.4.1 of im.status.ethereum
2020-07-27 21:09:54,142 ERROR: Could not build app im.status.ethereum: No apks match build/im.status.ethereum/result/app.apk
2020-07-27 21:09:54,145 INFO: Finished
2020-07-27 21:09:54,145 INFO: 1 build failed

But what I don't get is how did the Nix sandbox for pure builds not prevent this? Maybe because it's a local connection.

jakubgs on 27 Jul 2020

Looks like Nix build sandboxing isn't on by default for our builds: https://nixos.wiki/wiki/Nix#Sandboxing

jakubgs on 27 Jul 2020

Glorious success(almost):

2020-07-27 21:30:31,384 INFO: Successfully built version v1.4.1 of im.status.ethereum
2020-07-27 21:30:31,387 DEBUG: Using androguard from "/usr/local/lib/python3.5/dist-packages/androguard/__init__.py"
2020-07-27 21:30:31,634 DEBUG: Checking build/im.status.ethereum/result/app-release-unsigned.apk
2020-07-27 21:30:31,679 ERROR: Could not build app im.status.ethereum: Unexpected version/version code in output; APK: '1.4.1' / '2020072721',  Expected: 'v1.4.1' / '2020071121'
2020-07-27 21:30:31,679 INFO: Finished
2020-07-27 21:30:31,679 INFO: 1 build failed

Used config:

Categories:
  - Connectivity
  - Security
  - Money
License: MPL-2.0
SourceCode: https://github.com/status-im/status-react
IssueTracker: https://github.com/status-im/status-react/issues

RepoType: git
Repo: https://github.com/status-im/status-react.git

Builds:
  - versionName: 'v1.4.1'
    versionCode: 2020071121
    commit: '060cd3ff3a747d8f459d036736c6fb354268c00a'
    buildjni: no
    gradle: []
    scandelete:
      # iOS files are irrelevant to Android builds
      - ios
    scanignore:
      # We reference relative node_modules using ../
      - android/build.gradle
    sudo: 
      # Required for Nix
      - mkdir -m 0755 /nix
      - chown $USER /nix
    init:
      # Install Nix
      - ./nix/scripts/setup.sh
    build:
      # Assemble the release APK
      - make release-android
    output: ./result/app-release-unsigned.apk

# Our version tags look like `0.13.0-mobile`
AutoUpdateMode: 'Version %v-mobile'
UpdateCheckMode: Tags

jakubgs on 27 Jul 2020

By setting signingConfig null in android/app/build.gradle I managed to get it to succeed:

Extracting result: /nix/store/yh6si6sqfkanbc0pj5vniqq5s058fr75-status-react-build-release-android
'/nix/store/yh6si6sqfkanbc0pj5vniqq5s058fr75-status-react-build-release-android/app-release-unsigned.apk' -> './result/app-release-unsigned.apk'

SUCCESS
2020-07-28 08:46:09,944 INFO: Successfully built version 1.4.1 of im.status.ethereum
2020-07-28 08:46:10,275 INFO: success: im.status.ethereum
2020-07-28 08:46:10,275 INFO: Finished
2020-07-28 08:46:10,275 INFO: 1 build succeeded

Now the question is how I want to control this. Should I make a separate build type. Or disable signing with an env var.

jakubgs on 28 Jul 2020

I have a fully working unsigned build using fdroid branch:

2020-07-28 13:37:16,470 INFO: Using git version 2.11.0
2020-07-28 13:37:16,470 INFO: Building version 1.4.1 (2020071121) of im.status.ethereum
2020-07-28 13:37:16,471 WARNING: im.status.ethereum:1.4.1 runs this on the buildserver with sudo:
    mkdir -m 0755 /nix && chown $USER /nix
These commands were skipped because fdroid build is not running on a dedicated build server.
2020-07-28 13:37:16,471 INFO: Getting source for revision 1809c592a20916cef29057f3ace5a05a1b167c44
2020-07-28 13:37:21,472 INFO: Running 'init' commands in build/im.status.ethereum
2020-07-28 13:37:25,291 INFO: Creating local.properties file at build/im.status.ethereum/local.properties
2020-07-28 13:37:25,331 INFO: Cleaned build.gradle of keysigning configs at build/im.status.ethereum/android/app/build.gradle
2020-07-28 13:37:25,373 INFO: Scanning source for common problems...
2020-07-28 13:37:25,444 INFO: Ignoring unknown maven repo '$rootDir/../node_modules/react-native/android' at android/build.gradle
2020-07-28 13:37:25,444 INFO: Ignoring unknown maven repo '$rootDir/../node_modules/jsc-android/dist' at android/build.gradle
2020-07-28 13:37:25,459 INFO: Removing binary at ios/nanopb.framework/nanopb
2020-07-28 13:37:25,496 INFO: Creating source tarball...
2020-07-28 13:37:27,747 INFO: Running 'build' commands in build/im.status.ethereum
2020-07-28 13:45:29,924 INFO: Successfully built version 1.4.1 of im.status.ethereum
2020-07-28 13:45:30,253 INFO: success: im.status.ethereum
2020-07-28 13:45:30,253 INFO: Finished
2020-07-28 13:45:30,253 INFO: 1 build succeeded

Config:

Categories:
  - Connectivity
  - Security
  - Money

License: MPL-2.0
SourceCode: https://github.com/status-im/status-react
IssueTracker: https://github.com/status-im/status-react/issues
RepoType: git
Repo: https://github.com/status-im/status-react.git

Builds:
  - versionName: '1.4.1'
    versionCode: 2020071121
    commit: '1809c592a20916cef29057f3ace5a05a1b167c44'
    buildjni: no
    gradle: [] # Disable direct Gradle builds to use Nix
    scandelete: # iOS files are irrelevant to Android builds
      - ios
    scanignore: # We reference relative node_modules using ../
      - android/build.gradle
    sudo: # Required for Nix
      - mkdir -m 0755 /nix
      - chown $USER /nix
    init: # Install Nix
      - ./nix/scripts/setup.sh
    build: # Assemble the unsigned release APK
      - make release-android BUILD_TYPE=release BUILD_NUMBER=2020071121 ANDROID_APK_SIGNED=false
    output: ./result/app-release-unsigned.apk

Here's the PR for the changes necessary to make this work.

jakubgs on 28 Jul 2020

Congratulations @jakubgs . It's nice to see how hard you worked and how well you documented the way to the solution 😊

I am looking forward to have automatic updates of Status.im app through f-droid.
Status shows that it stand to its values 😊

mulles on 28 Jul 2020

👍2

First try (this a vagrant buildserver VM)...

20-07-28 14:05:46,943 DEBUG: buildserver > 2020-07-28 14:05:46,942 DEBUG: Directory: build/im.status.ethereum
2020-07-28 14:05:46,945 DEBUG: buildserver > 2020-07-28 14:05:46,943 DEBUG: > git remote set-head origin --auto
2020-07-28 14:05:47,455 DEBUG: buildserver > 2020-07-28 14:05:47,453 DEBUG: Directory: build/im.status.ethereum
2020-07-28 14:05:47,456 DEBUG: buildserver > 2020-07-28 14:05:47,454 DEBUG: > git checkout -f 1809c592a20916cef29057f3ace5a05a1b167c44
2020-07-28 14:05:48,077 DEBUG: buildserver > 2020-07-28 14:05:48,072 DEBUG: Directory: build/im.status.ethereum
2020-07-28 14:05:48,077 DEBUG: buildserver > 2020-07-28 14:05:48,073 DEBUG: > git clean -dffx
2020-07-28 14:05:48,181 DEBUG: buildserver > 2020-07-28 14:05:48,180 INFO: Running 'init' commands in build/im.status.ethereum
2020-07-28 14:05:48,182 DEBUG: buildserver > 2020-07-28 14:05:48,181 DEBUG: Directory: build/im.status.ethereum
2020-07-28 14:05:48,184 DEBUG: buildserver > 2020-07-28 14:05:48,181 DEBUG: > bash -x -c -- ./nix/scripts/setup.sh
2020-07-28 14:05:52,418 DEBUG: buildserver > 2020-07-28 14:05:52,416 ERROR: Could not build app im.status.ethereum: Error running init command for im.status.ethereum:1.4.1
2020-07-28 14:05:52,418 DEBUG: buildserver > ==== detail begin ====
2020-07-28 14:05:52,419 DEBUG: buildserver > + ./nix/scripts/setup.sh
2020-07-28 14:05:52,419 DEBUG: buildserver > Setting up Nix package manager...
2020-07-28 14:05:52,419 DEBUG: buildserver >   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-07-28 14:05:52,419 DEBUG: buildserver >                                  Dload  Upload   Total   Spent    Left  Speed
100  2490  100  2490    0     0   6543      0 --:--:-- --:--:-- --:--:--  6552
2020-07-28 14:05:52,420 DEBUG: buildserver > downloading Nix 2.3.6 binary tarball for x86_64-linux from 'https://releases.nixos.org/nix/nix-2.3.6/nix-2.3.6-x86_64-linux.tar.xz' to '/tmp/nix-binary-tarball-unpack.HX2GdFXSwi'...
2020-07-28 14:05:52,420 DEBUG: buildserver >   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2020-07-28 14:05:52,420 DEBUG: buildserver >                                  Dload  Upload   Total   Spent    Left  Speed
100 16.4M  100 16.4M    0     0  12.7M      0  0:00:01  0:00:01 --:--:-- 12.8M
2020-07-28 14:05:52,421 DEBUG: buildserver > Note: a multi-user installation is possible. See https://nixos.org/nix/manual/#sect-multi-user-installation
2020-07-28 14:05:52,421 DEBUG: buildserver > performing a single-user installation of Nix...
2020-07-28 14:05:52,421 DEBUG: buildserver > /tmp/nix-binary-tarball-unpack.HX2GdFXSwi/unpack/nix-2.3.6-x86_64-linux/install: directory /nix exists, but is not writable by you. This could indicate that another user has already performed a single-user installation of Nix on this system. If you wish to enable multi-user support see https://nixos.org/nix/manual/#ssec-multi-user. If you wish to continue with a single-user install for vagrant please run 'chown -R vagrant /nix' as root.
2020-07-28 14:05:52,424 DEBUG: buildserver > Failed to install Nix package manager!
2020-07-28 14:05:52,424 DEBUG: buildserver > Please see: https://nixos.org/nix/manual/#chap-installation
2020-07-28 14:05:52,424 DEBUG: buildserver > ==== detail end ====

licaon-kter on 28 Jul 2020

Thanks for testing @licaon-kter.

directory /nix exists, but is not writable by you

Right, because I assume that $USER is who will be running the build and installing Nix, which is why I do:

    sudo: # Required for Nix
      - mkdir -m 0755 /nix
      - chown $USER /nix

Is there a specific user I should be using there?

jakubgs on 28 Jul 2020

vagrant

licaon-kter on 28 Jul 2020

👍1

Okay, then that's what I'll use in the PR for fdroiddata.

jakubgs on 28 Jul 2020

Though if that might change at some point it would make sense to have an env variable that defines that.
Something like BUILD_USER or something. But I guess hardcoding vagrant is fine for now.
The issue with hardcoding it to vagrant is that it wont' work in the fdroid/docker-executable-fdroidserver:latest container.

jakubgs on 28 Jul 2020

I have created the PR for fdroiddata repo: https://gitlab.com/fdroid/fdroiddata/-/merge_requests/7179

jakubgs on 29 Jul 2020

Also added docs on submitting to F-Droid: https://github.com/status-im/status-react/pull/11008

jakubgs on 29 Jul 2020

Well this is bad. According to the build job on GitLab it fails on Nix installation:

warning: installing Nix as root is not supported by this script!

Because it apparently runs as root. So either I have to:

Make this somehow run as non-root user
Create nixbld group and users (doc)

jakubgs on 29 Jul 2020

@licaon-kter vagrant doesn't appear to be a valid user for GitLab Builds:

==== detail begin ====
+ mkdir -m 0755 /nix
+ chown vagrant /nix
chown: invalid user: ‘vagrant’
==== detail end ====

https://gitlab.com/jakubgs/fdroiddata/-/jobs/661345286

jakubgs on 29 Jul 2020

And if we look at a previous build where I used $USER it shows root:

+ mkdir -m 0755 /nix
+ chown root /nix

https://gitlab.com/jakubgs/fdroiddata/-/jobs/661321960

Which appears to mean that builds run as root.

jakubgs on 29 Jul 2020

If that is the case then according to this issue: https://github.com/NixOS/nix/issues/936

Installing single-user Nix under root is possible, you may find this build script useful, it sets up nix in a Debian image, and also properly initializes the nixbld* build users.

Which essentially does what this doc suggests:

ssgroupadd -r nixbld
for n in $(seq 1 10); do
  useradd -c "Nix build user $n" \
    -d /var/empty -g nixbld -G nixbld \
    -M -N -r -s "$(which nologin)" \
    nixbld$n;
done

jakubgs on 29 Jul 2020

Using this:

    sudo:
      - mkdir -m 0755 /nix
      - groupadd -r nixbld
      - for n in $(seq 1 10); do useradd -c "Nix build user $n" -d /var/empty -g nixbld
        -G nixbld -M -N -r -s "$(which nologin)" nixbld$n; done

Made the Nix installation work;

Installation finished!  To ensure that the necessary environment
variables are set, please add the line
  . /root/.nix-profile/etc/profile.d/nix.sh
to your shell profile (e.g. ~/.profile).
The Nix package manager was successfully installed.

But it fails later because $USER is not set:

+ make release-android BUILD_TYPE=release BUILD_NUMBER=2020070112 ANDROID_APK_SIGNED=false
Configuring Nix shell for target 'keytool'...
/root/.nix-profile/etc/profile.d/nix.sh: line 1: USER: unbound variable
./nix/scripts/shell.sh: line 78: exec: nix-shell: not found

jakubgs on 29 Jul 2020

@licaon-kter would it be possible to run the GitLab build as a normal user rather than root?

jakubgs on 29 Jul 2020

I'm kinda confused as what's the point of sudo field if everything is run as root...

jakubgs on 29 Jul 2020

Looks like we might have a problem because of react-native-camera that pulls stuff like play-services or firebase in:

+--- project :react-native-camera
|    +--- com.google.android.gms:play-services-vision:17.0.2
|    |    +--- com.google.android.gms:play-services-base:16.0.1
|    |    |    +--- com.google.android.gms:play-services-basement:16.0.1 -> 16.1.0
|    |    |    |    \--- androidx.legacy:legacy-support-v4:1.0.0
|    |    |    |         ...OMITTED...
|    |    |    \--- com.google.android.gms:play-services-tasks:16.0.1
|    |    |         \--- com.google.android.gms:play-services-basement:16.0.1 -> 16.1.0 (*)
|    |    +--- com.google.android.gms:play-services-basement:16.1.0 (*)
|    |    \--- com.google.android.gms:play-services-vision-common:[17.0.2] -> 17.0.2
|    |         +--- com.google.android.gms:play-services-base:16.0.1 (*)
|    |         +--- com.google.android.gms:play-services-basement:16.1.0 (*)
|    |         +--- com.google.android.gms:play-services-clearcut:16.0.0
|    |         |    +--- com.google.android.gms:play-services-base:16.0.1 (*)
|    |         |    +--- com.google.android.gms:play-services-basement:16.0.1 -> 16.1.0 (*)
|    |         |    +--- com.google.android.gms:play-services-phenotype:16.0.0
|    |         |    |    +--- com.google.android.gms:play-services-base:16.0.1 (*)
|    |         |    |    +--- com.google.android.gms:play-services-basement:16.0.1 -> 16.1.0 (*)
|    |         |    |    \--- com.google.android.gms:play-services-tasks:16.0.1 (*)
|    |         |    \--- com.google.android.gms:play-services-tasks:16.0.1 (*)
|    |         +--- com.google.android.gms:play-services-flags:16.0.1
|    |         |    +--- com.google.android.gms:play-services-base:16.0.1 (*)
|    |         |    \--- com.google.android.gms:play-services-basement:16.0.1 -> 16.1.0 (*)
|    |         \--- com.google.android.gms:play-services-phenotype:16.0.0 (*)

And it also appears to be pulling in Firebase:

+--- com.google.firebase:firebase-ml-vision:19.0.3
|    +--- com.google.firebase:firebase-common:16.0.3
|    \--- com.google.firebase:firebase-ml-common:[17.0.0] -> 17.0.0
|         +--- com.google.firebase:firebase-common:16.0.3 (*)
|         \--- com.google.firebase:firebase-iid:17.0.3
|              +--- com.google.firebase:firebase-common:16.0.3 (*)
|              \--- com.google.firebase:firebase-iid-interop:16.0.1

And it appears to be the case:
https://github.com/react-native-community/react-native-camera/blob/6e14c0b2/android/build.gradle#L78-L83

jakubgs on 30 Jul 2020

jakubgs on 30 Jul 2020

Andrey did some blazing fast work in #11016 to remove Play Services and Firebase. I updated the F-Droid PR.

Another issue is that #10620 also adds Firebase back for notifications, so we'll need to re-do that probably.

jakubgs on 30 Jul 2020

This work is currently blocked on 3 issues:

The #11016 PR needs to be merged to get rid of Firebase/Google Services, and it won't be included in 1.5.0
The #10620 PR adds Firebase back for Notifications so we need to redo that to not use it
Regardless of those two we still include installreferrer library that will not be accepted

We'll need to get rid of Firebase for the 3rd time and we'll need to be able to build a flavor of the app without installreferrer.

jakubgs on 6 Aug 2020

👍2

This work is currently blocked on 1 issue:

we still include installreferrer library that will not be accepted

flexsurfer on 11 Aug 2020

👍1

Since https://github.com/status-im/status-react/pull/11352 was merged I'll try updating our fdroiddata PR to exclude installreferrer and google stuff.

jakubgs on 13 Nov 2020

❤1 🎉1

I updated it with e38e6f16 which uses current develop state as 1.9.0-rc.1 version. We can update it later with proper 1.9.0.

jakubgs on 13 Nov 2020

I built 40e283979361878a72763a8b8c2f58fa0a012ba5 in fdroiddata#7179 and it looks like the APK still contains non-FOSS stuff:

https://plus.google.com/
https://www.googleapis.com/auth/appstate
https://www.googleapis.com/auth/datastoremobile
https://www.googleapis.com/auth/drive
https://www.googleapis.com/auth/drive.appdata
https://www.googleapis.com/auth/drive.apps
https://www.googleapis.com/auth/drive.file
https://www.googleapis.com/auth/fitness.activity.read
https://www.googleapis.com/auth/fitness.activity.write
https://www.googleapis.com/auth/fitness.blood_glucose.read
https://www.googleapis.com/auth/fitness.blood_glucose.write
https://www.googleapis.com/auth/fitness.blood_pressure.read
https://www.googleapis.com/auth/fitness.blood_pressure.write
https://www.googleapis.com/auth/fitness.body.read
https://www.googleapis.com/auth/fitness.body.write
https://www.googleapis.com/auth/fitness.body_temperature.read
https://www.googleapis.com/auth/fitness.body_temperature.write
https://www.googleapis.com/auth/fitness.location.read
https://www.googleapis.com/auth/fitness.location.write
https://www.googleapis.com/auth/fitness.nutrition.read
https://www.googleapis.com/auth/fitness.nutrition.write
https://www.googleapis.com/auth/fitness.oxygen_saturation.read
https://www.googleapis.com/auth/fitness.oxygen_saturation.write
https://www.googleapis.com/auth/fitness.reproductive_health.read
https://www.googleapis.com/auth/fitness.reproductive_health.write
https://www.googleapis.com/auth/games
https://www.googleapis.com/auth/games.firstparty
https://www.googleapis.com/auth/games_lite

https://gitlab.com/fdroid/fdroiddata/-/merge_requests/7179#note_448124916

Source: https://www.virustotal.com/gui/file/a078f9264c066cf4317473b73a211888e9011acc9dbf72f679a61c51d262d059/details

@Ferossgp any ideas?

jakubgs on 16 Nov 2020

Pretty sure it is:

diff --git a/android/app/build.gradle b/android/app/build.gradle
index 587de91a1..ea7f60a0d 100644
--- a/android/app/build.gradle
+++ b/android/app/build.gradle
@@ -127,6 +127,7 @@ def enableHermes = project.ext.react.get("enableHermes", false);
  * Exclude google library so we can publish to external android stores
  */
 def googleFree = project.env.get("GOOGLE_FREE", false)
+println('googleFree: ' + googleFree)

 def getVersionCode = { ->
     new ByteArrayOutputStream().withStream { stdOut ->

diff --git a/.env.release b/.env.release
index 5570e201f..af34048f5 100644
--- a/.env.release
+++ b/.env.release
@@ -17,3 +17,4 @@ RPC_NETWORKS_ONLY=1
 PARTITIONED_TOPIC=0
 ENABLE_ROOT_ALERT=1
 ENABLE_REFERRAL_INVITE=0
+GOOGLE_FREE=1



md5-ac273bb6243f6a6163c017794b7153b5



make release-android BUILD_TYPE=release



md5-31cab38ef7de91929272357ade98594f



> Configure project :app
Reading env from: .env
googleFree: 1

So as far as I can tell it works

jakubgs on 16 Nov 2020

Although I'm not actually sure what googleFree does in android/app/build.gradle as I don't see it used anywhere in that file:
https://github.com/status-im/status-react/blob/a91bdfb047558edf3f96a212a9bb1b0f815a7d71/android/app/build.gradle#L126-L129
So I'm confused...

jakubgs on 16 Nov 2020

Also, this has been added in #11352:
https://github.com/status-im/status-react/blob/a91bdfb047558edf3f96a212a9bb1b0f815a7d71/nix/mobile/android/release.nix#L103-L104
But I don't see .env being sourced anywhere, so this does nothing...

jakubgs on 16 Nov 2020

Also...you set it to 0 in the recipe, why? Sounds...ummm...backwards?

licaon-kter on 16 Nov 2020

Oh damn, I did. Maybe I changed it when testing and didn't change it back. Let me update that.

jakubgs on 16 Nov 2020

Doesn't seem like that fixed the issue:
https://www.virustotal.com/gui/file/12e37557f5c445ad998b7a78c4dc8f8c34866d4b2687c30160b21b07e6b92ad0/details

jakubgs on 16 Nov 2020

@jakubgs I fixed only the google dependency which was pulled by the referral program. Can you look at the dependency tree to see who pulls the rest?

Ferossgp on 17 Nov 2020

Sure, but can you confirm that:

def googleFree = project.env.get("GOOGLE_FREE", false)

Does nothing? Because it seems to do nothing.

jakubgs on 17 Nov 2020

Here is the result of:

GOOGLE_FREE=1 ./gradlew app:dependencies

https://gist.github.com/jakubgs/4f66650f3bb3dbddd48d60565ec3eca0

Some filtering:

 > grep com.google deps.log | sed 's/[|+ -\\]\+//' | sort -u
com.google.android.material:material:1.1.0
com.google.android.material:material:{strictly 1.1.0} -> 1.1.0 (c)
com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
com.google.code.findbugs:jsr305:3.0.1 -> 3.0.2
com.google.code.findbugs:jsr305:3.0.2
com.google.code.findbugs:jsr305:{strictly 3.0.2} -> 3.0.2 (c)
com.google.code.gson:gson:2.8.5
com.googlecode.json-simple:json-simple:1.1
com.google.errorprone:error_prone_annotations:2.2.0
com.google.guava:failureaccess:1.0.1
com.google.guava:guava:18.0 -> 27.0.1-jre (*)
com.google.guava:guava:23.0 -> 27.0.1-jre (*)
com.google.guava:guava:27.0.1-jre
com.google.guava:guava:27.0.1-jre (*)
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc:j2objc-annotations:1.1
com.google.jimfs:jimfs:1.1
com.google.protobuf:protobuf-java:3.4.0
com.google.zxing:core:3.3.0
com.google.zxing:core:{strictly 3.3.0} -> 3.3.0 (c)

jakubgs on 17 Nov 2020

The nice thing is, GOOGLE_FREE=1 is doing something, because if I use GOOGLE_FREE=0 I also get these:

com.google.android.gms:play-services-base:17.0.0
com.google.android.gms:play-services-basement:17.0.0
com.google.android.gms:play-services-basement:17.0.0 (*)
com.google.android.gms:play-services-basement:{strictly 17.0.0} -> 17.0.0 (c)
com.google.android.gms:play-services-base:{strictly 17.0.0} -> 17.0.0 (c)
com.google.android.gms:play-services-iid:17.0.0
com.google.android.gms:play-services-iid:{strictly 17.0.0} -> 17.0.0 (c)
com.google.android.gms:play-services-stats:17.0.0
com.google.android.gms:play-services-stats:{strictly 17.0.0} -> 17.0.0 (c)
com.google.android.gms:play-services-tasks:17.0.0
com.google.android.gms:play-services-tasks:17.0.0 (*)
com.google.android.gms:play-services-tasks:{strictly 17.0.0} -> 17.0.0 (c)

jakubgs on 17 Nov 2020

BTW, I removed the:

/**
 * Exclude google library so we can publish to external android stores
 */
def googleFree = project.env.get("GOOGLE_FREE", false)

Part from android/app/build.gradle and re-run GOOGLE_FREE=1 ./gradlew app:dependencies and I get the same result.
So it seems to be useless.

jakubgs on 17 Nov 2020

so it seems deps are good, so do we know where Interesting Strings come from ?

flexsurfer on 17 Nov 2020

i can't find these strings not in sources, not in dependencies

flexsurfer on 17 Nov 2020

This is the APK that licaon-kter built:
https://status-im-builds.ams3.digitaloceanspaces.com/im.status.ethereum_2020111309.apk
But that's before I fixed https://github.com/status-im/status-react/issues/8512#issuecomment-728092741.

jakubgs on 17 Nov 2020

I'm trying to build the recent 1.10.1 release via Fdroid docker image but the Gradle build in Nix fails with:

> Configure project :app
Unable to initialize metrics, ensure /home/builder/.android is writable, details: /home/builder/.android/analytics.settings (No such file or directory)

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/tmp-status-react-09f95dcdf/nix-build-status-react-build-release-android.drv-0/android/app/build.gradle' line: 1

* What went wrong:
A problem occurred evaluating project ':app'.
> Failed to apply plugin [id 'com.android.application']
   > com.android.prefs.AndroidLocation$AndroidLocationException: prop: null

I'm confused as to how it knows about /home/builder if the build is pure and doesn't inherit OS env variables.

jakubgs on 19 Jan 2021

Ah, I already had this issue: https://github.com/status-im/status-react/issues/8512#issuecomment-661011683

jakubgs on 19 Jan 2021

I've updated the F-Droid PR with the release 1.10.1 in a branch with a small android fix - fdroid-test-release - but according to @licaon-kter its still wrong, an he linked this:

https://www.virustotal.com/gui/file/25da6ed1a277e1245480768f0dfe4c05aa7095440764b50d4672364f8d0e901d/details

Last time he linked this result he said:

See Interesting strings:

But I have absolutely zero clue where from stuff like this appears in Interesting strings:

https://plus.google.com/
https://www.googleapis.com/auth/appstate
https://www.googleapis.com/auth/datastoremobile
https://www.googleapis.com/auth/drive
https://www.googleapis.com/auth/drive.appdata
...

As far as I can tell all Google dependencies are excluded, and searching the APK with strings utility finds nothing like this:

 > strings result/app-release.apk | grep googleapis | wc -l
0

jakubgs on 19 Jan 2021

I found out you can check for strings in an APK using aapt utility from Android SDK build tools:

 > aapt d --values strings result/app-release.apk | head 
String pool of 4765 unique UTF-8 non-sorted strings, 4765 entries and 0 styles using 252176 bytes:
String #0: %1$d / %2$d
String #1: %1$d%2$s
String #2: %1$s
String #3: %1$s – %2$s
String #4: %1$s – End date
String #5: %d new notification
String #6: %d new notifications
String #7: 0
String #8: 0.002

 > aapt d --values strings result/app-release.apk | wc -l
4769

But I can't find anything containing google or related:

 > aapt d --values strings result/app-release.apk | grep googleapis | wc -l
0

jakubgs on 19 Jan 2021

Oh, but I can find even more by unpacking the APK with apktool:

 > apktool d result/app-release.apk
 > cd apk-release
 > ag googleapis
smali/com/google/android/gms/common/FirstPartyScopes.smali
6:.field public static final GAMES_1P:Ljava/lang/String; = "https://www.googleapis.com/auth/games.firstparty"

smali/com/google/android/gms/common/Scopes.smali
6:.field public static final APP_STATE:Ljava/lang/String; = "https://www.googleapis.com/auth/appstate"
8:.field public static final CLOUD_SAVE:Ljava/lang/String; = "https://www.googleapis.com/auth/datastoremobile"
10:.field public static final DRIVE_APPFOLDER:Ljava/lang/String; = "https://www.googleapis.com/auth/drive.appdata"
12:.field public static final DRIVE_APPS:Ljava/lang/String; = "https://www.googleapis.com/auth/drive.apps"
14:.field public static final DRIVE_FILE:Ljava/lang/String; = "https://www.googleapis.com/auth/drive.file"
16:.field public static final DRIVE_FULL:Ljava/lang/String; = "https://www.googleapis.com/auth/drive"
20:.field public static final FITNESS_ACTIVITY_READ:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.activity.read"
22:.field public static final FITNESS_ACTIVITY_READ_WRITE:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.activity.write"
24:.field public static final FITNESS_BLOOD_GLUCOSE_READ:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.blood_glucose.read"
26:.field public static final FITNESS_BLOOD_GLUCOSE_READ_WRITE:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.blood_glucose.write"
28:.field public static final FITNESS_BLOOD_PRESSURE_READ:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.blood_pressure.read"
30:.field public static final FITNESS_BLOOD_PRESSURE_READ_WRITE:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.blood_pressure.write"
32:.field public static final FITNESS_BODY_READ:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.body.read"
34:.field public static final FITNESS_BODY_READ_WRITE:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.body.write"
36:.field public static final FITNESS_BODY_TEMPERATURE_READ:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.body_temperature.read"
38:.field public static final FITNESS_BODY_TEMPERATURE_READ_WRITE:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.body_temperature.write"
40:.field public static final FITNESS_LOCATION_READ:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.location.read"
42:.field public static final FITNESS_LOCATION_READ_WRITE:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.location.write"
44:.field public static final FITNESS_NUTRITION_READ:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.nutrition.read"
46:.field public static final FITNESS_NUTRITION_READ_WRITE:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.nutrition.write"
48:.field public static final FITNESS_OXYGEN_SATURATION_READ:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.oxygen_saturation.read"
50:.field public static final FITNESS_OXYGEN_SATURATION_READ_WRITE:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.oxygen_saturation.write"
52:.field public static final FITNESS_REPRODUCTIVE_HEALTH_READ:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.reproductive_health.read"
54:.field public static final FITNESS_REPRODUCTIVE_HEALTH_READ_WRITE:Ljava/lang/String; = "https://www.googleapis.com/auth/fitness.reproductive_health.write"
56:.field public static final GAMES:Ljava/lang/String; = "https://www.googleapis.com/auth/games"
58:.field public static final GAMES_LITE:Ljava/lang/String; = "https://www.googleapis.com/auth/games_lite"
62:.field public static final PLUS_LOGIN:Ljava/lang/String; = "https://www.googleapis.com/auth/plus.login"
67:.field public static final PLUS_ME:Ljava/lang/String; = "https://www.googleapis.com/auth/plus.me"

smali/com/google/android/gms/auth/api/signin/GoogleSignInOptions.smali
134:    const-string v1, "https://www.googleapis.com/auth/games_lite"
143:    const-string v1, "https://www.googleapis.com/auth/games"

jakubgs on 19 Jan 2021

😄1 👍1

Status-react: Research process of publishing to F-Droid

Most helpful comment

All 174 comments

Related issues