Javacpp-presets: new preset for libpostal

Created on 14 Dec 2017  路  32Comments  路  Source: bytedeco/javacpp-presets

I have started working on a libpostal preset. libpostal is a C library for parsing/normalizing street addresses around the world using statistical NLP and open data. I have created a branch but i got stuck pretty fast on the compile part, see mailinglist. Libpostal also carries huge blobs of trainingdata that is in need of a javacpp strategy. The lib is functional on my msys win10 x64 default console setup. I am hoping for somebody to provide a similar example i can study, or even some support.

bug enhancement
Source
picture Maurice-Betzel

All 32 comments

Could you try to follow these instructions?
https://github.com/bytedeco/javacpp-presets/wiki/Build-Environments#windows-x86-and-x86_64

saudet picture saudet on 15 Dec 2017

Yes i did, but i am missing deeper knowlledge on what a autoreconfig does. I need help if the default build instructions from libpostal do not work within the preset setup. I see forgot to push my latest state to the branch yesterday.

Maurice-Betzel picture Maurice-Betzel on 15 Dec 2017

cppbuild.sh is just a bash script, it doesn't do anything special. We don't even need it. You could install libpostal on your system and provide those paths in the pom.xml file, as done for CUDA, for example:
https://github.com/bytedeco/javacpp-presets/blob/master/cuda/pom.xml

saudet picture saudet on 15 Dec 2017

If i use these methods in the JavaCPP generated class from my branch for libpostal the software locks up or a access violation gets thrown:

public static native @Cast("char**") @ByPtrPtr byte[] libpostal_expand_address(@Cast("char*") byte[] input, @ByVal libpostal_normalize_options_t options, @Cast("size_t*") SizeTPointer n); public static native libpostal_address_parser_response_t libpostal_parse_address(@Cast("char*") byte[] address, @ByVal libpostal_address_parser_options_t options);

Trying to convert these methods from byte[] to String type with the following Info:

.put(new Info("char").cast().valueTypes("byte").pointerTypes("BytePointer", "String"))

resulted in:

C:\msys64\home\betzm\javacpp-presets-libpostal\libpostal\target\classes\org\bytedeco\javacpp\jnilibpostal.cpp(2606): error C2440: '=': cannot convert from 'char **' to 'const char *'
C:\msys64\home\betzm\javacpp-presets-libpostal\libpostal\target\classes\org\bytedeco\javacpp\jnilibpostal.cpp(2606): note: Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast

Is it possible to do this?

Maurice-Betzel picture Maurice-Betzel on 20 Dec 2017

Thanks for reporting! I've fixed this issue in the commit above, so make sure to use JavaCPP 1.3.4-SNAPSHOT.

You also had an issue running Autoconf through JavaCPP on Windows, right? Could you give me something to reproduce that here? Thanks!

saudet picture saudet on 21 Dec 2017

Doing the libpostal Windows mingw64 shell c build as explaind here via the console, all goes well. As soon as i do the Maven preset build with the same build.sh instructions it blows up on executing the bootstrap.sh containing autoreconf -fi --warning=no-portability with the following result:
...aclocal-1.15: found macro AM_PROG_AR in /usr/share/aclocal-1.15/ar-lib.m4: 13
aclocal-1.15: found macro AM_AUTOMAKE_VERSION in /usr/share/aclocal-1.15/amversion.m4: 14
aclocal-1.15: found macro AM_SET_CURRENT_AUTOMAKE_VERSION in /usr/share/aclocal-1.15/amversion.m4: 33

aclocal-1.15: error: aclocal: file '/msys64/usr/share/aclocal/xsize.m4' does not exist
autoreconf: aclocal failed with exit status: 1
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1)
at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:764)
at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:711)
at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:289)

The file xsize.m4 is located at where the above path is pointing.
The same Maven build from msys shell works, but produces a POSIX dependend dll file. New msys and pacman installs on the msys shell or the mingw shell resulted in the same errornous behaviour. I am tipping on shell specific behaviour, but thats beyond my knowlledge.

Maurice-Betzel picture Maurice-Betzel on 21 Dec 2017

And would you have a cppbuild.sh script that fails in that way somewhere?

saudet picture saudet on 22 Dec 2017

Like this one : https://github.com/Maurice-Betzel/javacpp-presets/blob/3d9fa081b6d8949058b3d8cf3d0836249f67f66a/libpostal/cppbuild.sh

Maurice-Betzel picture Maurice-Betzel on 22 Dec 2017

This put me on the right track resulting in:

if [[ -z "$PLATFORM" ]]; then
    pushd ..
    bash cppbuild.sh "$@" libpostal
    popd
    exit
fi

echo "downloading libpostal"

LIBPOSTAL_VERSION=master
download https://github.com/openvenues/libpostal/archive/$LIBPOSTAL_VERSION.tar.gz libpostal-$LIBPOSTAL_VERSION.tar.gz

echo "unpacking libpostal"

mkdir -p $PLATFORM
cd $PLATFORM
tar -xzvf ../libpostal-$LIBPOSTAL_VERSION.tar.gz
cd libpostal-$LIBPOSTAL_VERSION


case $PLATFORM in
    windows-x86_64)
    echo "Starting windows-x86_64 build"
    cp -rf windows/* ./
    cwd=$(pwd)
    bash -lc "cd $cwd && autoreconf -fi --warning=no-portability"
    #./configure --datadir=/c
    ./configure --disable-data-download
    make -j4
    make install
        ;;
    *)
        echo "Error: Platform \"$PLATFORM\" is not supported"
;;

esac

cd ../..
Maurice-Betzel picture Maurice-Betzel on 22 Dec 2017

Thanks! Yeah, it's happening here as well. Not sure why, seems to be a bug in MSYS2. Running it with bash -lc works, so let's just do that.

saudet picture saudet on 23 Dec 2017

I got MacOS 10 running on my vmware machine today, maybee i can provide and test libpostal mac builds as well.

Maurice-Betzel picture Maurice-Betzel on 23 Dec 2017
👍1

BTW, when you're ready to send a pull request, please rebase your fork on the master branch (currently 1.3.4-SNAPSHOT), thanks!

saudet picture saudet on 27 Dec 2017

Of course, did that yesterday already, now i am having issues with the Centos 7 libpostal build. How do i deal with pkg-config and JavaCPP?

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

It should be installed by default. What is the problem?

saudet picture saudet on 27 Dec 2017

Package libpostal was not found in the pkg-config search path.
Perhaps you should add the directory containing `libpostal.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libpostal' found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:33.632s
[INFO] Finished at: Wed Dec 27 08:29:25 CET 2017
[INFO] Final Memory: 26M/430M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build (javacpp.cppbuild.install) on project libpostal: Execution javacpp.cppbuild.install of goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build failed: Process exited with an error: 1 -> [Help 1]
[ERROR]

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

Well, we're supposed to be building it from source in cppbuild.sh, so just do that...

saudet picture saudet on 27 Dec 2017

That is what i am trying to do:

case $PLATFORM in
linux-x86_64)
./bootstrap.sh
#./configure --datadir=[...some dir with a few GB of space...]
./configure --disable-data-download
make -j4
sudo make install
sudo ldconfig

gives me:

[INFO] g++ -I/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/ -I/usr/java/jdk1.8.0_121/include -I/usr/java/jdk1.8.0_121/include/linux /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjnilibpostal.so -llibpostal
[ERROR] Failed to execute JavaCPP Builder: Cannot run program "g++" (in directory "/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/linux-x86_64"): error=2, No such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:17.802s
[INFO] Finished at: Wed Dec 27 08:52:35 CET 2017
[INFO] Final Memory: 26M/360M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build (javacpp.compiler) on project libpostal: Failed to execute JavaCPP Builder: Cannot run program "g++" (in directory "/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/linux-x86_64"): error=2, No such file or directory -> [Help 1]

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

The default build from the console gives me working libpostal software. What does JavaCPP need to wrap this?

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

You'll need to install the C++ compiler of GCC as well:
https://github.com/bytedeco/javacpp-presets/wiki/Build-Environments#linux-x86-and-x86_64

saudet picture saudet on 27 Dec 2017

Oh boy, i did follow all of the instructions, but g++ was still missing somehow. A clear case of RTFE (read the f* exception), trying to run too fast. Next issue:

[INFO] g++ -I/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/ -I/usr/java/jdk1.8.0_121/include -I/usr/java/jdk1.8.0_121/include/linux /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjnilibpostal.so -llibpostal
/usr/bin/ld: cannot find -llibpostal
collect2: error: ld returned 1 exit status

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

Switched off ldconfig, results in

[INFO] Building for platform "linux-x86_64"
[INFO] Targeting /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java
[ERROR] Failed to execute JavaCPP Builder: Could not parse "libpostal.h": File does not exist
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:50.565s
[INFO] Finished at: Wed Dec 27 10:06:43 CET 2017
[INFO] Final Memory: 26M/430M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build (javacpp.parser) on project libpostal: Failed to execute JavaCPP Builder: Could not parse "libpostal.h": File does not exist -> [Help 1]

I am way out of my comfort zone here...

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

Switched back on include mkdir and libpostal.h copy to it, resulted in:

[INFO] --- javacpp:1.3.4-SNAPSHOT:build (javacpp.parser) @ libpostal ---
[INFO] Detected platform "linux-x86_64"
[INFO] Building for platform "linux-x86_64"
[INFO] Targeting /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java
[INFO] Parsing /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/libpostal.h
[INFO]
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ libpostal ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ libpostal ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes
[INFO] /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java: /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java uses unchecked or unsafe operations.
[INFO] /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- javacpp:1.3.4-SNAPSHOT:build (javacpp.compiler) @ libpostal ---
[INFO] Detected platform "linux-x86_64"
[INFO] Building for platform "linux-x86_64"
[INFO] Generating /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp
[INFO] Generating /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp
[INFO] Compiling /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/linux-x86_64/libjnilibpostal.so
[INFO] g++ -I/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/ -I/usr/java/jdk1.8.0_121/include -I/usr/java/jdk1.8.0_121/include/linux /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjnilibpostal.so -llibpostal
/usr/bin/ld: cannot find -llibpostal
collect2: error: ld returned 1 exit status
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:45.644s
[INFO] Finished at: Wed Dec 27 10:16:08 CET 2017
[INFO] Final Memory: 26M/360M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build (javacpp.compiler) on project libpostal: Execution javacpp.compiler of goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build failed: Process exited with an error: 1 -> [Help 1]

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

Am i missing the file for linking? And where do i get it from?

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

i copy the libpostal.a in the lib folder but with the same result. What am i missing?

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

Trying to link against the so file from the .libs folder....

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

I copied the whole generated .libs folder into the linux-x86_64/lib folder, with the same result:

/IdeaProjects/javacpp-presets
[INFO]
[INFO] --- javacpp:1.3.4-SNAPSHOT:build (javacpp.parser) @ libpostal ---
[INFO] Detected platform "linux-x86_64"
[INFO] Building for platform "linux-x86_64"
[INFO] Targeting /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java
[INFO] Parsing /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/libpostal.h
[INFO]
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ libpostal ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ libpostal ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes
[INFO] /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java: /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java uses unchecked or unsafe operations.
[INFO] /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/src/main/java/org/bytedeco/javacpp/libpostal.java: Recompile with -Xlint:unchecked for details.
[INFO]
[INFO] --- javacpp:1.3.4-SNAPSHOT:build (javacpp.compiler) @ libpostal ---
[INFO] Detected platform "linux-x86_64"
[INFO] Building for platform "linux-x86_64"
[INFO] Generating /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp
[INFO] Generating /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp
[INFO] Compiling /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/linux-x86_64/libjnilibpostal.so
[INFO] g++ -I/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/ -I/usr/java/jdk1.8.0_121/include -I/usr/java/jdk1.8.0_121/include/linux /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjnilibpostal.so -L/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/lib/ -Wl,-rpath,/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/lib/ -llibpostal
/usr/bin/ld: cannot find -llibpostal
collect2: error: ld returned 1 exit status
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:02.658s
[INFO] Finished at: Wed Dec 27 12:25:28 CET 2017
[INFO] Final Memory: 26M/360M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build (javacpp.compiler) on project libpostal: Execution javacpp.compiler of goal org.bytedeco:javacpp:1.3.4-SNAPSHOT:build failed: Process exited with an error: 1 -> [Help 1]

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

If i manually compile like this i get my libjnilibpostal.so:

g++ -I/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/include/ -I/usr/java/jdk1.8.0_121/include -I/usr/java/jdk1.8.0_121/include/linux /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnilibpostal.cpp /home/mbetzel/IdeaProjects/javacpp-presets/libpostal/target/classes/org/bytedeco/javacpp/jnijavacpp.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjnilibpostal.so -L/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/lib/ -Wl,-rpath,/home/mbetzel/IdeaProjects/javacpp-presets/libpostal/cppbuild/linux-x86_64/lib/ -lpostal

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

lovin this one:
cp src/.libs/libpostal.so ../lib/liblibpostal.so

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

Usually "make install" will do what we need. I designed it like that. Check the other presets, for example:
https://github.com/bytedeco/javacpp-presets/blob/master/libdc1394/cppbuild.sh

saudet picture saudet on 27 Dec 2017

I have studied the default setup, but it was not working. A working and tested build is now up. Hoping the mac build will be nearly the same :)

Maurice-Betzel picture Maurice-Betzel on 27 Dec 2017

I have added and tested the MacOS build and i am ready for a pull request. Could you take a look at the setup and report anything missing?

Maurice-Betzel picture Maurice-Betzel on 28 Dec 2017
👍1

Released!
http://search.maven.org/#search%7Cga%7C1%7Clibpostal
Thanks again for the contribution.

saudet picture saudet on 17 Jan 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

archenroot picture archenroot  路  56Comments
Neiko2002 picture Neiko2002  路  32Comments
oneengineer picture oneengineer  路  34Comments
blueberry picture blueberry  路  32Comments
nanguantong picture nanguantong  路  25Comments