Anki-android: "This isn't a valid apkg file" when importing files over 2GB

[david-allison-1]

User-reported. We're not providing error reporting on many paths which involve the string: R.string.import_log_no_apkg.

Many of these errors could be caused by errors other than a bad file import.

User had appropriate space for a large (2.4GB) file, and 4GB RAM. Stated that it imported correctly on PC. This implies we may have a bug here, but there's no opportunity to trace.

File: AnKing Step 1 V7 with Media.apkg: https://mega.nz/folder/UAwVRQbb#dA6sMOtPjRgqI83FYNvTug

[mikehardy]

great that we have the package, that'll help. Note that we may need as much as 3x the space temporarily but I thought you/we added a check for space :thinking: - I do know that direct import of large collections (by note/card count) basically blows up the app at the moment even if there is no media - turns into a GC spiral of death with all available memory consumed by something most likely not needed stored in most likely an inefficient temp data structure or something

[david-allison-1]

Space check was for Check Database - not the import, I was hoping to reduce the space needed for import via #6036.

In the meantime, I'll turn up the exception reports to see if we can get a problem. Still in contact with the user, and should be able to get a crash report from the alpha.

[DrTSN]

hello everyone i have the same issue with the same file ,my phone is android galaxy note 9 with free 20 gb and 6gb ram please find a fix soon and thx alot

[mikehardy]

@DrTSN I would not have any expectation there is a working fix soon. You should move immediately to the best workaround is to use Anki Desktop to import the package, then cloud sync it to AnkiWeb, and cloud sync it down to your device. Also note that even with 6GB available RAM in the phone, Android still only gives each program some 30-40MB of RAM to work with normally, which is why in general it is hard for us to import large decks

We're working to fix it obviously but do not block yourself on the fix - the Desktop/AnkiWeb/AnkiDroid pathway is the workaround

[david-allison-1]

Reproduced on an API 27 emulator.

E/AnkiPackageImporter: Failed to unzip apkg.
    java.util.zip.ZipException: error in opening zip file
        at java.util.zip.ZipFile.open(Native Method)
        at java.util.zip.ZipFile.<init>(ZipFile.java:235)
        at java.util.zip.ZipFile.<init>(ZipFile.java:149)
        at com.ichi2.libanki.importer.AnkiPackageImporter.run(AnkiPackageImporter.java:61)
        at com.ichi2.async.CollectionTask.doInBackgroundImportAdd(CollectionTask.java:1079)
        at com.ichi2.async.CollectionTask.doInBackground(CollectionTask.java:295)
        at com.ichi2.async.CollectionTask.doInBackground(CollectionTask.java:73)
        at android.os.AsyncTask$2.call(AsyncTask.java:333)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
        at java.lang.Thread.run(Thread.java:764)

From googling, this appears for many reasons. One of which was when trying to unzip a file larger than 2GB in older java versions.

https://stackoverflow.com/questions/53186835/error-in-opening-zip-file-opening-of-big-zip-files-2gb

[mikehardy]

Hmm - if it's a 2GB limit I wonder if Apache Compress could get around it? Would have an apparent cost of 634Kb which is pretty chunky https://repo1.maven.org/maven2/org/apache/commons/commons-compress/1.20/ but I bet most of that shrinks out with R8 based on a quick look at the contents

[david-allison-1]

@mikehardy I've got an inefficient ZipInputStream solution which gets past the initial hurdle. Would you prefer I got this working (small effort - would still be inefficient), or move to another library which will increase APK size?

[mikehardy]

"inefficient" with no metrics does not mean anything :-), I recall you thought something was inefficient prior but actually had no impact and the multidex wasn't much bloat in practice. In general working code is better than anything else so unless it's really inefficient I trust you :-)

[david-allison-1]

Had another read through the code and the media import won't be amenable to a stream-based approach. We really need to be able to index into the zip file, or we'll need a large rewrite of the importers

Will investigate commons-compress

P.S: There's no "before" to compare to currently, but I'll get performance figures before the PR goes out.

[mikehardy]

Sure - and with commons-compress I'm sure-pending-verification that R8 is going to delete out most of it resulting in not too large of an APK size bump. I'm willing to accept some kilobytes of increased size

[david-allison-1]

commons-compress 1.20 works

[mikehardy]

Have you ever done an assembleRelease? You'll have to make a java keystore so you know the password etc to it (because you won't have the real release one) but it should be possible, and you can check the result of minification and size added in APK Analyzer inside Android Studio. ./tools/make-parallel-release.sh for instance was usable for me (with a personal keystore) before I had the official AnkiDroid keystore as long as I put my custom store in the right location

[david-allison-1]

Did one for the visual editor, and done a few for self-improvement purposes.

It's a bit of a nightmare on Windows if done via the shell scripts (Perl breaks the build as it doesn't remove the .bak files, and the script deletes itself).

But... just running the release works. I'll get to it once I've finished my testing.

I'm not going to convert the syncing logic yet.
From a quick (lazy) glance, there's no alternative for ZipFile.OPEN_READ | ZipFile.OPEN_DELETE

[david-allison-1]

AnkiDroid-release.apk
Size: 13.5 MB (14,245,653 bytes)
Size on Disk: 13.5 MB (14,245,888 bytes)

[mikehardy]

2.11.2 is running 14135061 bytes so this is 110k more.
I still feel like 13.5MB (in general) is not that big, and we still have places we can cut - specifically I'm not doing ABI splitting (yet) so everyone gets 3.5MB of sqlite .so files when arm64 (majority case) is ony 900kb

TLDR, 110k to provably work (assuming it works) with a well supported lib, works for me

[mikehardy]

(although should put it behind compat if possible, and API gate to where we no longer need it?)

[david-allison-1]

Still needed as-of API 29 😕 (couldn't get 30 booting on my PC). I don't think there's much to gain with compat at this stage if there's no native replacement.

[mikehardy]

Mmm. Agreed, compat is to force maintainers to think of sunsetting, but with no native "sunrise", it's has no value no

[mikehardy]

@DrTSN we may have a working fix now in 2.12alpha10 which is processing in google play store now and should release momentarily, or you can try directly in parallel / next to your main install with this package https://github.com/ankidroid/Anki-Android/releases/download/v2.12alpha10/AnkiDroid-2.12alpha10.parallel.A.apk

[david-allison-1]

Already had a number of users happy that the alpha now unzips the new Anking decks. At this point I'm fairly confident we've fixed the bug with no regressions.

[mikehardy]

Great news! Just commenting to be clear to anyone else the unzip fix was for older devices and went with alpha7 I think - nice to have independent confirmation that simply unzipping was an issue and you nailed that.

Hopefully the OOSpace/OOMemory fixes I put in don't regress :crossed_fingers: - I will be listening very carefully to hear if any of the Anking users have issues now