Incubator-mxnet: Problem getting Amalgamation to work for Android Studio

[woodthom2]

I had a lot of problems getting the latest release to compile in Amalgamation. I’m using Ubuntu 14.

Some files are missing e.g. src/ndarray/unary_function.cc

I see that they were moved/renamed in this commit https://github.com/dmlc/mxnet/pull/1839/files in April. So commit 1839 broke the Amalgamation.

So I reverted to a commit from before April: Merge: d81adfa 45ee47c Author: Tianqi [email protected] Date: Wed Mar 23 09:04:12 2016 -0700

Now I am able to compile it and generate jni_libmxnet_predict.so. I renamed to libmxnet_predict.so to put it in the Android Studio Whatsthis app.

However then libmxnet_predict.so will not compile correctly in Android Studio.

                                                                Process: com.happen.it.make.whatsthis, PID: 27189
                                                                      java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "_ZN5mxnet21GraphStorageAllocatorC1EPNS_11StaticGraphERKSt6vectorIjSaIjEESt10shared_ptrINS_16GraphStoragePoolEE" referenced by "libmxnet_predict.so"...

I can see a lot of people had this problem (or similar):

http://stackoverflow.com/questions/34878870/ndk-cannot-locate-symbol-znwj

https://github.com/dmlc/mxnet/issues/1436 https://github.com/dmlc/mxnet/issues/1613

it is necessary to compile this because the libmxnet_predict.so supplied in the Android WhatsThis app by Leliana only works for the models supplied with Mxnet. When I use the Caffe->Mxnet converter to generate an Mxnet model file, the Android WhatsThis app crashes with an error in the native code.

This happens even if I run https://github.com/dmlc/mxnet/tree/master/tools/caffe_converter on VGG: the resultant model doesn’t work in Android Whatsthis. But no problems with iOS Whatsthis..

So I can suggest 2 alternatives:

1. can someone please provide a version of libmxnet_predict.so that works on Android for any Caffe model converted via convert_model.py? Then we need to update Whatsthis app.
2. does anybody know what is causing the java.lang.UnsatisfiedLinkError in the Android app when I compile libmxnet_predict.so?

[antinucleon]

Compiling is fixed in https://github.com/dmlc/mxnet/pull/2135

[woodthom2]

Thank you. I am now able to compile libmxnet OK. But still I get an error when running on Android Studio.

I have done the following

• built Android standalone toolchain (as in Amalgamation instructions https://github.com/dmlc/mxnet/tree/master/amalgamation) on Ubuntu 14 machine
• built OpenBLAS
• checked out the latest build of Mxnet, 81dd47a from 13 May 2016
• compiled Mxnet
• applied diff of https://github.com/dmlc/mxnet/pull/2135/files to mxnet_predict0.cc
• cd amalgamation; make ANDROID=1
• applied diff from https://github.com/dmlc/mxnet/tree/master/amalgamation to mxnext_predict-all.cc
• make ANDROID=1
• this has produced jni_libmxnet_predict.so, 6082116 bytes
• I checked out https://github.com/Leliana/WhatsThis and copied in jni_libmxnet_predict.so, renamed to libmxnet_predict.so
• I ran WhatsThis

On Ubuntu I get error
FATAL EXCEPTION: main
java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "_ZN5mxnet21GraphStorageAllocatorC1EPNS_11StaticGraphERKSt6vectorIjSaIjEESt10shared_ptrINS_16GraphStoragePoolEE" referenced by "libmxnet_predict.so"...

On Mac I get error:

java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "_ZN5mxnet2op16SimpleOpRegistry3GetEv" referenced by "libmxnet_predict.so"...

[antinucleon]

Thanks I will take a look tonight.

[woodthom2]

Thank you
On 13 May 2016 18:38, "Bing Xu" [email protected] wrote:

Thanks I will take a look tonight.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/dmlc/mxnet/issues/2133#issuecomment-219110142

[antinucleon]

@woodthom2 I have tested by using python: https://github.com/dmlc/mxnet/blob/master/tests/python/predict/mxnet_predict_example.py

Seems nothing is wrong with amalgamation. I am not quite sure what happened in Android side. Please clean everything and try again.

[SlipknotTN]

I managed to recompile prediction for Android using amalgamation almost one month ago.
The changes made were:

• mxnet_predict0.cc
  Added:

#include "src/operator/elementwise_unary_op.cc"
#include "src/operator/operator_util.cc"

Removed:

#include "src/ndarray/unary_function.cc"
#include "src/common/tblob_op_registry.cc"

• After mxnet_predict-all.cc first compilation fail, commented:

#include <bits/exception_ptr.h>
#include <bits/nested_exception.h>
#include <config.h>
#include <machine/ansi.h>
#include <packet/sse-inl.h>
#include <sys/_types.h>

and reran MAKE.

[woodthom2]

Hi

Thanks for your advice. In fact I had to change more things in mxnet_predict-all.cc. I am using 691f1d3 as a starting point.

My method:

When I compiled jni_libmxnet_predict.so there were some undefined references. I could identify them using:

ndk-which nm jni_libmxnet_predict.so | grep "\sU\s" | less

Then when there was an undefined method I could pipe to c++filt -n in order to identify the missing code.

Using this method I was able to identify that most of the contents of operator_util.cc were missing from mxnet_predict-all.cc.

I put the missing lines into mxnet_predict-all.cc. I am attaching the diff of what I changed, and my version of mxnet_predict-all.cc, as well as the compiled JNI binary. Hopefully this will help anybody who encountered the same problem.

changes-for-android-to-mxnet_predict-all.diff.txt
jni_libmxnet_predict.so.zip
mxnet_predict-all.cc.zip

[saustar]

@woodthom2 Hi I am facing similar issue. Error when trying to Run on android studio:

FATAL EXCEPTION: main
                                                                              Process: com.happen.it.make.whatsthis, PID: 10381
                                                                              java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "_ZN5mxnet2op16SimpleOpRegistry3GetEv" referenced by "libmxnet_predict.so"...
                                                                                  at java.lang.Runtime.loadLibrary(Runtime.java:371)

I didn't have to make changes as you have done. I have checked those changes and it were there in my .cc file.
2 month ago I was able to use it. Now I am facing issues.
mxnet_predict-all.cc.zip

[woodthom2]

Hi @saustar, what if you try the exact files I attached?

If not you can use the command c++filt -n to identify what missing method is _ZN5mxnet2op16SimpleOpRegistry3GetEv.

then you can check if it's missing in your mxnet_predict-all.cc and add it in. You probably have to do this several times to fill in all the missing methods

Also you can find all errors doing this:

make -j8 ANDROID=1 2> error.txt
grep error error.txt

to identify the missing methods, use this command:

ndk-which nm jni_libmxnet_predict.so | grep "\sU\s" | less

then you find all missing symbols without going to Android Studio. Probably you have a whole missing class which you will have to copy/paste into mxnet_predict-all.cc so that it builds and links correctly.

[saustar]

@woodthom2 I have tried with your mxnet_predict-all.cc file. Same issue :(
Also when I am not making for android and only use make as given in the documentation. mxnet_predict-all.cc is generated but process is not successful due to some errors. But I ignored them and then make ANDROID=1 . Could that be an issue?

I will work with your suggestion. Thanks for your response.

[woodthom2]

yes the first time it will give some errors, then you need to correct mxnet_predict-all.cc , and then make again.

[saustar]

Do you have any idea about how to reduce the size of jni_libmxnet_predict.so file?

[woodthom2]

No, sorry. Maybe @antinucleon can help?

[saustar]

@woodthom2 Can you see below the error file and mxnet_predict-all.cc file

c++filt -n _ZN5mxnet2op16SimpleOpRegistry3GetEv
mxnet::op::SimpleOpRegistry::Get()

When I do ctrl+F with SimpleOpRegistry in your file and my file I can see similar item. So whats missing?
error.txt.zip
mxnet_predict-all.cc.zip

[saustar]

Finally I am successful.

include "src/operator/operator_util.cc" was missing from mxnet_predict0.cc. That solve the issue but my so file size is more than 6.7 MB . Last time when I was able to make a successful SO file, size was 6.2 MB. I wonder why?

[phunterlau]

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!