Javacpp-presets: HDF5 support for Raspberry Pi

[timothyjward]

I've noticed that there is no version of the HDF5 preset for armhf. The preset mentions missing support for cross-compilation, but this was changed several years ago, and there are now Debian packages for HDF5 on armhf which means that it should be fine to run on Raspbian. I'm therefore wondering if it would be possible to start producing the preset for this platform?

[saudet]

Yes, we just need to get these files from their code generator running on ARM devices:
https://github.com/bytedeco/javacpp-presets/blob/master/hdf5/hdf5-linux-ppc64le.patch
I think we can use QEMU for that instead of having the hardware. I just haven't gotten around to it yet.

[peardox]

Just get Raspberry to send you kit - easy enough

[peardox]

Balenea https://www.balena.io/blog/balena-releases-first-fully-functional-64-bit-os-for-the-raspberry-pi-4/ have just released a Yocto based 'runs on all pi, 32 + 64' release. This is based on Yocto - https://www.yoctoproject.org.

The point here is that it's all container based so may make an interesting option with less compatibility issues.

[saudet]

@peardox We still need to run and maintain the hardware somehow. I don't have the resources to do that all by myself. Although, for this particular issue, would you be able to get us the generated files that we need to cross compile HDF5? That would help to get CI going.

[peardox]

Which ones you after?

ARM64 + ARMHF I guess. No problem. Tell me exactly what you need and that's what I'll create.

ARMHF will take a few days - the best one's out on loan, got another though that need some small fixes I may be able to get running sooner.

I notice you've got a hardware request section on bytedeco.org. I can look into getting you some hardware and cloud servers if you like. Sourcing (for free) was part of my job 20 years ago. Raspberry should be easy as should NVidia Rock etc (All ARM). Cloud server would be worth trying for PPC64le + Power8/9 (may be able to get physical kit for power 8)

[saudet]

Thanks! We need the 2 files shown in the patch file: H5lib_settings.c and H5Tinit.c, for both ARMHF and ARM64 yes.

Having hardware and/or cloud servers is good, thanks a lot, but we'd still need someone to setup Jenkins or something. It might be possible to use Travis CI on those machines, but I wouldn't count on it, it's probably easier to just use QEMU. Anyway, someone would need to try it out:
https://enterprise.travis-ci.com/

[peardox]

I'll look into it

On Sat, Oct 5, 2019 at 1:17 PM Samuel Audet [e-mail]
wrote:

Thanks! We need the 2 files shown in the patch file: H5lib_settings.c and
H5Tinit.c, for both ARMHF and ARM64 yes.

Having hardware and/or cloud servers is good, thanks a lot, but we'd still
need someone to setup Jenkins or something. It might be possible to use
Travis CI on those machines, but I wouldn't count on it, it's probably
easier to just use QEMU. Anyway, someone would need to try it out:
https://enterprise.travis-ci.com/

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bytedeco/javacpp-presets/issues/794?email_source=notifications&email_token=AAL2HIYQ4OC3GB62LH4L25TQNCAXPA5CNFSM4I4ISS22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEANRDSA#issuecomment-538644936,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAL2HI3N6SJ3I473G6KMHS3QNCAXPANCNFSM4I4ISS2Q
.

[peardox]

HDF5 ARM64 builds fine on Jetson Nano - see debug

The files H5lib_settings.c and H5Tinit.c are nowhere on the system - looks like nano's pre-patched

Note libhdf5-dev not installed for this run - see apt list

Going to try unofficial Pi4 build on ARM64 with known issues pre-fixed (flycapture, libssl0dev + jbig) as this is likely fresher (beta) than the Nano OS.

After that I have a 60G SD ready to test stock Pi4. Something occurred to me while nodding off to sleep - Pi default to a stupid sway size (100M) so gonna give it stupid swap in other direction (8g, both 32 + 64 bit versions of OS)

=================== apt list ==================
hdf5-helpers/bionic,now 1.10.0-patch1+docs-4 arm64 [installed,auto-removable]
hdf5-tools/bionic,now 1.10.0-patch1+docs-4 arm64 [installed]
libhdf5-100/bionic,now 1.10.0-patch1+docs-4 arm64 [installed,automatic]
libhdf5-cpp-100/bionic,now 1.10.0-patch1+docs-4 arm64 [installed,auto-removable]
libhdf5-dev/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-doc/bionic 1.10.0-patch1+docs-4 all
libhdf5-java/bionic 1.10.0-patch1+docs-4 all
libhdf5-jni/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-mpi-dev/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-mpich-100/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-mpich-dev/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-openmpi-100/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-openmpi-dev/bionic 1.10.0-patch1+docs-4 arm64
libhdf5-serial-dev/bionic 1.10.0-patch1+docs-4 all
libjhdf5-java/bionic 2.11.0+dfsg-3 arm64
libjhdf5-jni/bionic 2.11.0+dfsg-3 arm64
libpdl-io-hdf5-perl/bionic 1:0.73-3build2 arm64
python-hdf5storage/bionic 0.1.14-1 all
python-hdf5storage-doc/bionic 0.1.14-1 all
python3-hdf5storage/bionic 0.1.14-1 all
r-cran-hdf5/bionic 1.6.10-4.1 arm64
yorick-hdf5/bionic 0.8.0-8 arm64
=================== debug ==================
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] JavaCPP Presets for CPython 3.7.4-1.5.2-SNAPSHOT ... SUCCESS [15:51 min]
[INFO] JavaCPP Presets for NumPy 1.17.2-1.5.2-SNAPSHOT .... SUCCESS [10:38 min]
[INFO] JavaCPP Presets for HDF5 1.10.5-1.5.2-SNAPSHOT ..... SUCCESS [ 20.954 s]
[INFO] JavaCPP Presets for LLVM 9.0.0-1.5.2-SNAPSHOT ...... SUCCESS [05:06 min]
[INFO] JavaCPP Presets for Leptonica 1.78.0-1.5.2-SNAPSHOT SUCCESS [08:03 min]
[INFO] JavaCPP Presets for Tesseract 4.1.0-1.5.2-SNAPSHOT . SUCCESS [06:51 min]
[INFO] JavaCPP Presets for cpu_features 0.4.1-1.5.2-SNAPSHOT SUCCESS [ 14.842 s]
[INFO] JavaCPP Presets for Systems 1.5.2-SNAPSHOT ......... FAILURE [ 4.619 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 47:14 min
[INFO] Finished at: 2019-10-06T07:49:24+01:00

[saudet]

You'll need to build HDF5 from source to get those generated files.

[peardox]

OK - I'll give it a go - hdf5-1.10.5.tar.bz2 as per hdf5/cppbuild.sh I presume

Use any flags or just ...

./configure
make

Here's the output of ./configure

General Information:
-------------------
                   HDF5 Version: 1.10.5
                  Configured on: Fri Oct 11 01:28:46 BST 2019
                  Configured by: simon@nano
                    Host system: aarch64-unknown-linux-gnu
              Uname information: Linux nano 4.9.140-tegra #1 SMP PREEMPT Tue Sep 10 14:23:34 BST 2019 aarch64 aarch64 aarch64 GNU/Linux
                       Byte sex: little-endian
             Installation point: /home/simon/src/hdf5-1.10.5/hdf5

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables:
                        LDFLAGS:
                     H5_LDFLAGS:
                     AM_LDFLAGS:
                Extra libraries: -lz -ldl -lm
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: /usr/bin/gcc
                       CPPFLAGS:
                    H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L   -DNDEBUG -UH5_DEBUG_API
                    AM_CPPFLAGS:
                        C Flags:
                     H5 C Flags:  -std=c99  -pedantic -Wall -Wextra -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings -finline-functions -s -Wno-inline -Wno-aggregate-return -Wno-missing-format-attribute -Wno-missing-noreturn -O
                     AM C Flags:
               Shared C Library: yes
               Static C Library: yes


                        Fortran: no

                            C++: no

                           Java: no


Features:
---------
                   Parallel HDF5: no
Parallel Filtered Dataset Writes: no
              Large Parallel I/O: no
              High-level library: yes
                    Threadsafety: no
             Default API mapping: v110
  With deprecated public symbols: yes
          I/O filters (external): deflate(zlib)
                             MPE: no
                      Direct VFD: no
                         dmalloc: no
  Packages w/ extra debug output: none
                     API tracing: no
            Using memory checker: no
 Memory allocation sanity checks: no
          Function stack tracing: no
       Strict file format checks: no
    Optimization instrumentation: no

[saudet]

Anything should give the same thing, thanks!

[peardox]

OK - I'll do arm64 + 32 in the AM

[saudet]

Although, strangely enough, H5lib_settings.c contains information about the build configuration. Might be best to use the same flags as used for other architectures in cppbuild.sh.

[peardox]

Eek - just looked at cppbuild.sh

There's no entry for linux-arm64

I'll cobble something equivalent together from the linux-armhf version

On Fri, Oct 11, 2019 at 3:00 AM Samuel Audet [e-mail]
wrote:

Although, strangely enough, H5lib_settings.c contains information about
the build configuration. Might be best to use the same flags as used for
other architectures in cppbuild.sh.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bytedeco/javacpp-presets/issues/794?email_source=notifications&email_token=AAL2HIZ7KESMWRMWLTMRRNTQN7M35A5CNFSM4I4ISS22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA6PXNY#issuecomment-540867511,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAL2HIYBMH3RF2LUD25FBGTQN7M35ANCNFSM4I4ISS2Q
.

[peardox]

armhf = Raspberry Pi4 + Raspbian (Debian Buster)
arm64 = Jetson Nano + Ubuntu 18.04

See relevant hdf5--settings.log for build settings for configure switches in cppbuild.sh mode
Each arch has asis (straight configure) and cppbuild dir (estimated cppbuild configure for arm64)

hdf5.zip

[saudet]

Ok, it's done! Let me know if you encounter any issues with the snapshots: http://bytedeco.org/builds/

[saudet]

And binaries have now been released with version 1.5.2! Enjoy and thanks for the contributions