Moor: DynamicLibrary.open failed on some Android 6.0.1 devices

[knaeckeKami]

I thought this issue was fixed with the hack in #420.
Unfortunately this seems to be not enough, I recently received these crash reports again:

Non-fatal Exception: io.flutter.plugins.firebase.crashlytics.FlutterError: Invalid argument(s): Failed to load dynamic library (dlopen failed: library "/data/data/<appid>/lib/libsqlite3.so" not found)
       at ._defaultOpen(load_library.dart:46)
       at OpenDynamicLibrary.openSqlite(load_library.dart:98)
       at .sqlite3(sqlite3.dart:9)
       at .sqlite3(sqlite3.dart)
       at ._ensureSqlite3Initialized(db_open_helper.dart:27)
       ...

Affected Models:
Fairphone Model FP2
Android Version: 6.0.1

samsung Galaxy S5 Neo
Android Version: 6.0.1

I checked the APK file generated for these device, and the .so seems to be included correctly (lib/armeabi-v7a/sqlite3.so, next to libflutter.so).

[knaeckeKami]

It seems there is another bug with dynamic libraries on these devices. It only happens when using appbundle, when using apks it works fine (with the hack of using absolute paths for loading libraries).

The solution is to set

android.bundle.enableUncompressedNativeLibs=false in gradle.properties.

This also increases the disk usage of the installed app

I now think that that issue is caused by

• using app bundles
• using (some devices with?) android 6.0.1, which can only load dynamic libraries via an absolute path
• not using android.bundle.enableUncompressedNativeLibs = false, which causes the .so files to not get extracted, they stay inside their .apk files in /data/app/<appid>-<some-hash>/, so there is no absolute path to these .so files (see https://developer.android.com/studio/releases/gradle-plugin )
  
  
  
  • setting android.bundle.enableUncompressedNativeLibs=false causes the .so files to get extracted from their .apk file during the installation of the .apk, so one can load them via absolute path with the hack used in #420
  
  

[simolus3]

Thanks a lot for the report and your investigation! I'll warn about this in the README of sqlite3_flutter_libs and on the documentation website. Unfortunately that's probably all I can do here...

Although I wonder how those devices managed to load libflutter.so and libapp.so in the first place, which obviously must have worked as the crash came from Dart.

[knaeckeKami]

Although I wonder how those devices managed to load libflutter.so and libapp.so in the first place, which obviously must have worked as the crash came from Dart.

Yes, me too. I did not have time to instigate this, I think there must be a way to load the .so files directly from the APK. Maybe there's a difference between calling System.loadLibrary() from Java and dlopen via FFI on these devices?

However, I'm happy that I found a way to solve the issue, I already got feedback from affected users that it works now for them (at the cost of slightly increasing the installed app size for all users, though...).

I'll try to use the same approach as Flutter does, using context.applicationInfo.nativeLibraryDir, see https://chromium.googlesource.com/external/github.com/flutter/engine/+/3c9a22c778f89e826edfe0f1814346acedbefe59/shell/platform/android/io/flutter/view/FlutterMain.java#177

[knaeckeKami]

Ok I did another deep dive into this cursed issue.
I hope this helps someone who stumbles into the same problem.

I found that loading the library from Java with System.loadLibrary() works. After the library has been loaded from Java, it can also be loaded from Dart.

So my workaround is the following:

Adding an Android-Plugin:

package dev.littlebat.native_lib_dir;

import android.content.Context;
import android.content.pm.ApplicationInfo;

import androidx.annotation.NonNull;

import io.flutter.embedding.engine.plugins.FlutterPlugin;
import io.flutter.plugin.common.MethodCall;
import io.flutter.plugin.common.MethodChannel;
import io.flutter.plugin.common.MethodChannel.MethodCallHandler;
import io.flutter.plugin.common.MethodChannel.Result;
import io.flutter.plugin.common.PluginRegistry.Registrar;

/** NativeLibPlugin */
public class NativeLibDirPlugin implements FlutterPlugin, MethodCallHandler {
  /// The MethodChannel that will the communication between Flutter and native Android
  ///
  /// This local reference serves to register the plugin with the Flutter Engine and unregister it
  /// when the Flutter Engine is detached from the Activity
  private MethodChannel channel;
  private Context context;

  @Override
  public void onAttachedToEngine(@NonNull FlutterPluginBinding flutterPluginBinding) {
    context = flutterPluginBinding.getApplicationContext();
    channel = new MethodChannel(flutterPluginBinding.getBinaryMessenger(), "native_lib_dir");
    channel.setMethodCallHandler(this);
  }

  @Override
  public void onMethodCall(@NonNull MethodCall call, @NonNull Result result) {
    if (call.method.equals("getNativeLibraryDir")) {
      final ApplicationInfo info = context.getApplicationInfo();
      if(info != null) {
        result.success(info.nativeLibraryDir);
      }else{
        result.success(null);
      }
    } else if(call.method.equals("loadLibrary")){
      try {
        System.loadLibrary((String)(call.arguments));
        result.success(null);
      }catch(Throwable e){
        result.error("1", "could not load: " + e.toString(), null );
      }
    }else {
      result.notImplemented();
    }
  }

  @Override
  public void onDetachedFromEngine(@NonNull FlutterPluginBinding binding) {
    channel.setMethodCallHandler(null);
  }
}

Dart-side

class NativeLib {
  static const MethodChannel _channel =
      const MethodChannel('native_lib_dir');

  static Future<String> get nativeLibraryDir async {
    final String version = await _channel.invokeMethod('getNativeLibraryDir');
    return version;
  }

  static Future<dynamic> loadNativeLibraryInJava(String library) async {
    return _channel.invokeMethod('loadLibrary', library);
  }
}

overriding sqlite open:

///https://github.com/simolus3/moor/issues/895
DynamicLibrary _androidOpenSqlite(String nativeLibDir) {
  if (Platform.isAndroid) {
    try {
      debugPrint("load sqlite");
      return DynamicLibrary.open('libsqlite3.so');
    } catch (_) {
      debugPrint("fail, trying workarounds...");
      if (Platform.isAndroid) {
        try {
          final lib = DynamicLibrary.open("$nativeLibDir/libsqlite3.so");
          debugPrint(
              "successfully loaded sqlite3 with strange workaround at $nativeLibDir");
          return lib;
        } catch (_) {
          // On some (especially old) Android devices, we somehow can't dlopen
          // libraries shipped with the apk. We need to find the full path of the
          // library (/data/data/<id>/lib/libsqlite3.so) and open that one.
          // For details, see https://github.com/simolus3/moor/issues/420
          final appIdAsBytes = File('/proc/self/cmdline').readAsBytesSync();

          // app id ends with the first \0 character in here.
          final endOfAppId = max(appIdAsBytes.indexOf(0), 0);
          final appId =
              String.fromCharCodes(appIdAsBytes.sublist(0, endOfAppId));

          return DynamicLibrary.open('/data/data/$appId/lib/libsqlite3.so');
        }
      }

      rethrow;
    }
}

import 'package:sqlite3/open.dart';
import 'package:sqlite3/sqlite3.dart' as sqlite_lib;

// Do this once, before opening a database
// see https://github.com/simolus3/moor/issues/876
Future<void> _ensureSqlite3Initialized() async {
  try {
    if (_hasInitializedSqlite) {
      return;
    }
    _hasInitializedSqlite = true;
    if (Platform.isAndroid) {
      final nativeLibDir = await NativeLib.nativeLibraryDir;
      open.overrideFor(
          OperatingSystem.android, () => _androidOpenSqlite(nativeLibDir));
      //force sqlite open, throw exception on fail
      sqlite_lib.sqlite3.tempDirectory;
    }

  } catch (_) {
    try {
      //load sqlite from java
      //note the missing lib and .so, this needs to be like that from Java
      await NativeLib.loadNativeLibraryInJava("sqlite3");
      //open sqlite again
      sqlite_lib.reopen();
      debugPrint("loaded sqlite with weird workaround");

      final version = sqlite_lib.sqlite3.version;

      debugPrint("loaded sqlite ${version.libVersion} ${version.sourceId} ${version.versionNumber}");

    }catch(_){
      rethrow;
    }

  }
}

I needed to fork sqlite3 and add this method in sqlite3/lib/src/api/sqlite3.dart:

void reopen(){
  assert(sqlite3 == null);
  sqlite3 = Sqlite3._(open.openSqlite());
}

I have a device which prints:

1-17 20:39:12.010 2256-2307/? I/flutter: load sqlite
11-17 20:39:12.010 2256-2307/? I/flutter: fail, trying workarounds...
11-17 20:39:12.030 2256-2307/? I/flutter: load sqlite
11-17 20:39:12.030 2256-2307/? I/flutter: loaded sqlite with weird workaround
11-17 20:39:12.040 2256-2307/? I/flutter: loaded sqlite 3.32.3 2020-06-18 14:00:33 7ebdfa80be8e8e73324b8d66b3460222eb74c7e9dfd655b48d6ca7e1933cc8fd 3032003

when installing the app from an app bundle with that code, meaning only the last workaround that loads the library from java succeeds.

Hopefully, this helps someone who also stumbles into this

[simolus3]

Thank you again for looking into this and finding a solution. Do you think there'll be issues if we just load sqlite3 from Java in onAttachedToEngine and avoid all the method invocations? That would make things much simpler, I could add something like that to sqlite3_flutter_libs.

[knaeckeKami]

possibly. i'll ship that approach in the next release of my app, with the fallback to java as first workaround and the other ones later, and I'll log to crash reporting what worked and what not.

I can only test on so many devices, and there's a lot of weirdness going on with shared libraries and old android versions.
And testing that is very time consuming, since many failures only occur when building the app from an appbundle, which takes long to build, and then I have to use bundletool to install the appbundle.

I would also like to try what would happen if I changed the invocation of dlopen from RTLD_LAZY to RTLD_NOW here, but I have no experience how I would ship that code then.

[simolus3]

i'll ship that approach in the next release of my app, with the fallback to java as first workaround and the other ones later

Thanks a lot, this would be good to know. If it works I can add mechanisms to sqlite_flutter_libs and package:sqlite3 to hopefully make this easier for everyone.

I would also like to try what would happen if I changed the invocation of dlopen from RTLD_LAZY to RTLD_NOW here

FWIW the extensions functionality is unrelated to dart:ffi, you would probably have to change this one. But it might be easier to do something like

final self = DynamicLibrary.process();
final dlopen = self.lookupMethod<...>('dlopen');
dlopen(allocate('libsqlite3.so'), 2 /*RTLD_NOW*/);

Just to see what dlopen/dlerror returns in that case.

[knaeckeKami]

Thanks for your suggestion with the FFI invocation.
It doesn't make a difference if we use RTLD_NOW or RTLD_LAZY, both flags work on most devices, but on devices that cannot load the lib with RTLD_LAZY, RTLD_NOW also does not work.

Using Java first via

 System.loadLibrary("sqlite3");

seems to work more reliably, though.

[simolus3]

I added a similar mechanism to sqlite3_flutter_libs, but it's not yet published. If you add a git dependency:

dev_dependencies:
  sqlite3_flutter_libs:
    git:
      url: https://github.com/simolus3/sqlite3.dart.git
      path: sqlite3_flutter_libs

You can use the new applyWorkaroundToOpenSqlite3OnOldAndroidVersions() method. It basically calls System.loadLibrary("sqlite3") if we're on Android and DynamicLibrary.open('libsqlite3') fails.

Unfortunately I couldn't reproduce this (a standard emulator with Android 6 doesn't show this behavior). If it doesn't take too much time for you to reproduce this issue, trying that workaround would be much appreciated. I could also try to reproduce this in Firebase Device Lab if you know some devices where it fails.

[knaeckeKami]

With applyWorkaroundToOpenSqlite3OnOldAndroidVersions() it seems to work on my affected Android 6.0.1 device!

You can try it yourself with one of these devices, these are the ones that appeared in my crash reporting with this issue:

• Samsung Galaxy S5,
• Fairphone Model FP2
• Samsung Galaxy S5 Neo
• HUAWEI P8 Lite
• HUAWEI P9 lite
• HUAWEI P9
• HUAWEI Honor 7
• HUAWEI Honor 8
• OnePlus 2
• HUAWEI Y6 2017
• LG G4
• OPPO CPH1701
• Xiaomi Redmi 4 Pro
• PRIV by BlackBerry
• HTC ONE M8s
• Sony Xperia Z3 Compact
• Samsung Galaxy Note4
• Samsung Galaxy A3
• Samsung Galaxy S5 mini
• Moto G (3rd Gen)
• OnePlus X
• ZTE A2017G

All on Android 6.
I don't know which of them are available on DeviceLab, last time I checked they did not offer a lot of old devices...

If you try it out yourself, be sure to build an appbundle and then install just the device-specific apk (using bundletool) , as a fat apk might behave differently than an appbundle.

[simolus3]

I released the workaround in sqlite3_flutter_libs version 0.3.0. Again, thank you very much for your help Martin!!

[knaeckeKami]

Thanks for maintaining moor and dealing with the pain of being one of the pioneers with dart:ffi :)