One: How can I release with external libraries?

[jinevening]

I'm writing this issue to ask helps for "release". I'm new to release, so any suggestion can be very helpful.
(@seanshpark Can you tell me the difference between "release" and https://github.com/Samsung/ONE/releases ?)

Problem

I made a module (record-minmax) using external libraries (hdf5 and boost). How can I release the project with external libraries?

boost

I found the cmake config file for boost in infra/cmake/packages/BoostConfig.cmake. The file downloads the boost library from the external download server written in BoostSourceConfig.cmake and builds the library.

I used the boost in record-minmax like this. nnas_find_package(Boost QUITE COMPONENTS program_options). Should I do anything more for the release with boost?

hdf5

I found the cmake config file for hdf5 in infra/nnfw/cmake/packages/HDF5Config.cmake. The file finds hdf5 in two ways.

1. Specifying the location of hdf5 library (H5Cpp.h, libhdf5.a, libhdf5_cpp.a)
2. Finding hdf5 pre-installed in the system (find_package(HDF5 COMPONENTS CXX QUIET))

My program (record-minmax) is placed under the compiler directory, so I cannot use nnfw_find_package. Instead, I just used find_package, which is the same as the second way of the HDF5Config.cmake.

So, our customers should install hdf5 in advance. If the customers use Ubuntu, this may be done easily (sudo apt-get install libhdf5-dev), but it would be not simple in other platforms.

It seems that hdf5 can be downloaded and built with cmake (like this). But for this, we need CMake minimum version 3.12 (3.15 is recommended) link. I have no idea about the cmake version of our release (Can anyone tell me?).

cc @mhs4670go @cgbahk

[seanshpark]

Can you tell me the difference between "release" and https://github.com/Samsung/ONE/releases

https://github.com/Samsung/ONE/releases is the URL where we get the "Releases" files or artifacts

[seanshpark]

What I think is

• run record-minmax without any problem is the purpose
• we may not have to build boost or hdf5 from source

To experience the problem of the end-user

• prepare a clean linux machine; virtual machine can be a solution
• download the software; https://github.com/Samsung/ONE/releases/download/1.6.0/nncc-1.6.0.tar.gz
• copy(or install) our release software to the clean linux box
• run record-minmax

[jinevening]

https://github.com/Samsung/ONE/releases is the URL where we get the "Releases" files or artifacts

It was a foolish question. I didn't see nncc-1.6.0.tar.gz before. So, release is making a package like nncc-1.6.0.tar.gz. Now I understand what is release. Thanks.

[jinevening]

The release file (ex: nncc-1.6.0.tar.gz) is made by the below command.

 ./nnas create-package --preset 20200630 --prefix install

It seems that ./nnas create-package calls infra/packaging/build. It calls preset_configure, builds modules, and calls preset_install.

preset_configure and preset_install are described in the preset saved in infra/packaging/preset. For example, 20200630 is the preset of the release for 2020/06/30. The preset describes which modules will be included in the package and how to compile those modules.

[jinevening]

I've downloaded nncc-1.6.0.tar.gz and tried to run record-minmax in the package. The host environment was Ubuntu 18.04 installed with hdf5 and boost.

I've faced error messages three times, all caused by link failures.

1. libhdf5_cpp.so.11 link failure

`tmp/release/nncc-package/bin/record-minmax: error while loading shared libraries: libhdf5_cpp.so.11: cannot open shared object file: No such file or directory`

I could remove the error message by making a symbolic link for the library, as follows.

sudo ln -s /usr/lib/x86_64-linux-gnu/libhdf5_cpp.so /usr/lib/x86_64-linux-gnu/libhdf5_cpp.so.11

1. libhdf5_serial.so.10 (solved similarly)
2. libboost_program_options.so.1.58.0 (solved similarly)

So, to run record-minmax, customers need to install hdf5 and boost, and set the libraries to proper versions (the versions used when record-minmax was compiled).

I can think of three solutions.

1. Make users install hdf5 and boost

In this case, we can make an "install script" to install the packages (if the customers use Ubuntu) and set the libraries to proper versions. The version of boost is fixed as 1.58.0 (infra/cmake/packages/BoostConfig.cmake), so we can easily make an install script for boost. But the version of hdf5 is not fixed for now, so it is difficult to make such an install script.

2. Release with hdf5/boost libraries

It would be possible to add hdf5/boost library files to the package and let record-minmax use the attached hdf5/boost libraries.

I've tried to include hdf5/boost libraries in lib directory of the release package. To properly link libraries, it is necessary to manually set the $RPATH of record-minmax, libhdf5, and libraries used by libhdf5, which is extremely complicated. I'm not familiar to the modification of elf binary, so I left this item as a future work.

3. Compile with static libraries?

I've run cmake with HDF5_USE_STATIC_LIBRARIES ON according to this link, but it did not work (don't know why).

[mhs4670go]

@jinevening

I've run cmake with HDF5_USE_STATIC_LIBRARIES ON according to this link, but it did not work (don't know why).

This is because of the cache. It would be better to use clean env.

$ ./nnas create-package --preset 20200630 --prefix install
$ ldd record-minmax
    linux-vdso.so.1 =>  (0x00007fffc77ed000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f52eb396000)
    libsz.so.2 => /usr/lib/x86_64-linux-gnu/libsz.so.2 (0x00007f52eb193000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f52eaf79000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f52ead75000)
    libboost_program_options.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0 (0x00007f52eaaf7000)
    libluci_import.so => /home/seongwoo/ONE/install/bin/./../lib/libluci_import.so (0x00007f52ea7da000)
    libluci_export.so => /home/seongwoo/ONE/install/bin/./../lib/libluci_export.so (0x00007f52ea57d000)
    libluci_interpreter.so => /home/seongwoo/ONE/install/bin/./../lib/libluci_interpreter.so (0x00007f52ea2e0000)
    libluci_lang.so => /home/seongwoo/ONE/install/bin/./../lib/libluci_lang.so (0x00007f52ea0ab000)
    libloco.so => /home/seongwoo/ONE/install/bin/./../lib/libloco.so (0x00007f52e9e7c000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f52e9a99000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f52e9790000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f52e9578000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f52e91ae000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f52eb5b3000)
    libaec.so.0 => /usr/lib/x86_64-linux-gnu/libaec.so.0 (0x00007f52e8fa6000)
    libluci_logex.so => /home/seongwoo/ONE/install/bin/./../lib/../lib/libluci_logex.so (0x00007f52e8d45000)
    libluci_log.so => /home/seongwoo/ONE/install/bin/./../lib/../lib/libluci_log.so (0x00007f52e8b3d000)
    libluci_env.so => /home/seongwoo/ONE/install/bin/./../lib/../lib/libluci_env.so (0x00007f52e893a000)
    libluci_pass.so => /home/seongwoo/ONE/install/bin/./../lib/../lib/libluci_pass.so (0x00007f52e86dd000)
    libluci_service.so => /home/seongwoo/ONE/install/bin/./../lib/../lib/libluci_service.so (0x00007f52e8491000)

[jinevening]

@mhs4670go Great! libhdf5 is removed. Can you share your cmake code for record-minmax here?

[mhs4670go]

@jinevening libboost_program_options can be static with Boost_USE_STATIC_LIBS option. I've added below line to preset.

cmake \
    -DCMAKE_INSTALL_PREFIX="${NNCC_INSTALL_PREFIX}" \
    -DHDF5_USE_STATIC_LIBRARIES=ON \ // NOTE here
    -DBoost_USE_STATIC_LIBS=ON \ // NOTE here
    -DCMAKE_BUILD_TYPE=release \
    -DBUILD_WHITELIST=$(join_by ";" "${REQUIRED_UNITS[@]}") \
    ${EXTRA_OPTIONS[@]} \
    "${NNAS_PROJECT_PATH}/infra/nncc"

And this is an applied verison;static hdf5, boost

$ ldd record-minmax                                       
    linux-vdso.so.1 =>  (0x00007fff70197000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3defc5f000)
    libsz.so.2 => /usr/lib/x86_64-linux-gnu/libsz.so.2 (0x00007f3defa5c000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f3def842000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3def63e000)
    libluci_import.so => /home/seongwoo/ONE/install/bin/./../lib/libluci_import.so (0x00007f3def321000)
    libluci_export.so => /home/seongwoo/ONE/install/bin/./../lib/libluci_export.so (0x00007f3def0c4000)
    libluci_interpreter.so => /home/seongwoo/ONE/install/bin/./../lib/libluci_interpreter.so (0x00007f3deee27000)
    libluci_lang.so => /home/seongwoo/ONE/install/bin/./../lib/libluci_lang.so (0x00007f3deebf2000)
    libloco.so => /home/seongwoo/ONE/install/bin/./../lib/libloco.so (0x00007f3dee9c3000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f3dee5e0000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3dee2d7000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f3dee0bf000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3dedcf5000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f3defe7c000)
    libaec.so.0 => /usr/lib/x86_64-linux-gnu/libaec.so.0 (0x00007f3dedaed000)
    libluci_logex.so => /home/seongwoo/ONE/install/bin/./../lib/../lib/libluci_logex.so (0x00007f3ded88c000)
    libluci_log.so => /home/seongwoo/ONE/install/bin/./../lib/../lib/libluci_log.so (0x00007f3ded684000)
    libluci_env.so => /home/seongwoo/ONE/install/bin/./../lib/../lib/libluci_env.so (0x00007f3ded481000)
    libluci_pass.so => /home/seongwoo/ONE/install/bin/./../lib/../lib/libluci_pass.so (0x00007f3ded224000)
    libluci_service.so => /home/seongwoo/ONE/install/bin/./../lib/../lib/libluci_service.so (0x00007f3decfd8000)

[mhs4670go]

FYI, These are thought to be solutions from my best knowledge.

1. Add -DHDF5_USE_STATIC_LIBRARIES=ON command to preset. (possible)

Because there is only record-minmax that uses hdf5 among release modules, it's okay to build and link hdf5 by static. But if so, there will be a difference between release-module and develop-module. There is a problem that the consistency of the test is broken, and debugging becomes difficult when an error occurs.

1. set HDF5_USE_STATIC_LIBRARIES ON globally. (impossible now)

If I build hdf5 with static, an error occurs because there is a few shared library modules. To link with these modules, hdf5 must compile with -fPIC, which seems to be hard as of now.

1. build hdf5 with static and dynamic at the same time. (impossible now)

Originally, it is possible to build a library as static and dynamic respectively. Then, I can link "dynamic" to shared library and "static" to static library. But since I have to use find_package function, I can't control this.

[mhs4670go]

I've tried to make above impossible things possible and it's worked out.

set HDF5_USE_STATIC_LIBRARIES ON (#2965)

• Instead of setting HDF5_USE_STATIC_LIBRARIES to ON globally, I set it locally.

But it's not enough. In FindHDF5.cmake, some variable(HDF5_CXX_LIBRARY_hdf5,HDF5_CXX_LIBRARY_hdf5_cpp) is set with CACHE option, which makes local setting ineffective.

• So, I had to unset these with CACHE option.

Then, libhdf5 links to target statically. But, I had to solve one more thing.

• Make external libraries static by replacing .so to .a

HDF5 uses external libraries like libz and libsz. But, HDF5 cmake script doesn't support the function that links external lib statically to HDF5. Since these libraries coexist with shared and static, I can replace .so to .a.

Of course, if you don't want to use these, you don't have to do this but remove these from list variable like HDF5_CXX_LIBRARIES.

[jinevening]

Of course, if you don't want to use these, you don't have to do this but remove these from list variable like HDF5_CXX_LIBRARIES.

I really want to use static compilation :) It would be the simplest way for users (just download and run).

I tested if #2965 works well in my environment, but it failed. Here's what I did (My machine uses Ubuntu 18.04).

1. Make a docker image

I made a Ubuntu 16.04 docker image with ./nnas build-docker-image. The command failed first time, but it somehow successfully created a docker image nnas:latest when I ran the command again.

1. Run a docker container based on the image nnas:latest

I created a docker container mounted with our project directory (ONE).

1. Create a package

I ran the command ./nnas create-package --preset 20200630 --prefix install in the docker, but it failed with below messages

[ 98%] Built target luci_interpreter_kernels_test
[ 99%] Built target circle2circle_test
[ 99%] Built target circle2circle
[ 99%] Linking CXX executable record-minmax
/usr/lib/x86_64-linux-gnu/hdf5/serial/lib/libhdf5.a(H5PL.o): In function `H5PL__open':
/build/hdf5-3JSo9x/hdf5-1.8.16+docs/debian/build/src/../../../src/H5PL.c:596: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used
 for linking
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libpthread.a(pthread_create.o): In function `allocate_stack':
/build/glibc-e6zv40/glibc-2.23/nptl/allocatestack.c:475: undefined reference to `_dl_stack_flags'
/build/glibc-e6zv40/glibc-2.23/nptl/allocatestack.c:613: undefined reference to `_dl_stack_flags'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libpthread.a(elision-lock.o): In function `elision_init':
/build/glibc-e6zv40/glibc-2.23/nptl/../sysdeps/unix/sysv/linux/x86/elision-conf.c:65: undefined reference to `_dl_x86_cpu_features'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libpthread.a(nptl-init.o): In function `__pthread_initialize_minimal_internal':
/build/glibc-e6zv40/glibc-2.23/nptl/nptl-init.c:305: undefined reference to `__libc_setup_tls'
/build/glibc-e6zv40/glibc-2.23/nptl/nptl-init.c:322: undefined reference to `_dl_cpuclock_offset'
/build/glibc-e6zv40/glibc-2.23/nptl/nptl-init.c:457: undefined reference to `_dl_pagesize'
/build/glibc-e6zv40/glibc-2.23/nptl/nptl-init.c:466: undefined reference to `_dl_pagesize'
/build/glibc-e6zv40/glibc-2.23/nptl/nptl-init.c:486: undefined reference to `_dl_init_static_tls'
/build/glibc-e6zv40/glibc-2.23/nptl/nptl-init.c:488: undefined reference to `_dl_wait_lookup_done'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libpthread.a(nptl-init.o): In function `__pthread_get_minstack':
/build/glibc-e6zv40/glibc-2.23/nptl/nptl-init.c:509: undefined reference to `_dl_pagesize'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libdl.a(dlopen.o): In function `dlopen':
(.text+0x5): undefined reference to `__dlopen'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libdl.a(dlclose.o): In function `dlclose':
(.text+0x1): undefined reference to `__dlclose'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libdl.a(dlsym.o): In function `dlsym':
(.text+0x5): undefined reference to `__dlsym'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libdl.a(dlerror.o): In function `dlerror':
(.text+0x1): undefined reference to `__dlerror'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libpthread.a(ptw-write.o): In function `__write_nocancel':
/build/glibc-e6zv40/glibc-2.23/nptl/../sysdeps/unix/syscall-template.S:84: undefined reference to `__syscall_error'
/build/glibc-e6zv40/glibc-2.23/nptl/../sysdeps/unix/syscall-template.S:84: undefined reference to `__syscall_error'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libpthread.a(ptw-read.o): In function `__read_nocancel':
/build/glibc-e6zv40/glibc-2.23/nptl/../sysdeps/unix/syscall-template.S:84: undefined reference to `__syscall_error'
/build/glibc-e6zv40/glibc-2.23/nptl/../sysdeps/unix/syscall-template.S:84: undefined reference to `__syscall_error'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libpthread.a(ptw-close.o): In function `__close_nocancel':
/build/glibc-e6zv40/glibc-2.23/nptl/../sysdeps/unix/syscall-template.S:84: undefined reference to `__syscall_error'
/usr/lib/gcc/x86_64-linux-gnu/6/../../../x86_64-linux-gnu/libpthread.a(ptw-close.o):/build/glibc-e6zv40/glibc-2.23/nptl/../sysdeps/unix/syscall-template.S:84: more undefined references to `__syscall_error
' follow
collect2: error: ld returned 1 exit status
compiler/record-minmax/CMakeFiles/record-minmax.dir/build.make:255: recipe for target 'compiler/record-minmax/record-minmax' failed
make[2]: *** [compiler/record-minmax/record-minmax] Error 1
CMakeFiles/Makefile2:5173: recipe for target 'compiler/record-minmax/CMakeFiles/record-minmax.dir/all' failed
make[1]: *** [compiler/record-minmax/CMakeFiles/record-minmax.dir/all] Error 2
Makefile:138: recipe for target 'all' failed
make: *** [all] Error 2
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt-get install python3-venv 

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/home/ONE.2/install/bin/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']

/usr/bin/python: No module named pip
/usr/bin/python: No module named pip

It seems that glibc is not linked well. (I'm not sure the virtual environment things are relevant, so I just copied and pasted the full message)

@mhs4670go Did you face the same problem? or did I something wrong when making the nncc package?

[mhs4670go]

@jinevening Well, this is because it linked libpthread and libdl statically, which shouldn't be happened. My draft just make only libz, and libsz static. It's weird. Print HDF5_CXX_LIBRARIES once in cmake configuration. There should be libpthread.so and libdl.so.

[mhs4670go]

build HDF5 in cmake configuration (#2989)

• Instead of use HDF5 located in machine, use what I built myself in cmake configure step

This is a very convenient way as I can control the target. When building directly, the option to not build zlib and szlib is also supported. In the above draft, it built HDF5 both dynamically and statically, and linked it where necessary.

TIPS. If you want to speed up configure time, give -DEXTERNALS_BUILD_THREADS={the number of your core * 2} option.

[jinevening]

I've tested #2989 works well both in clean Ubuntu 16.04 docker and Ubuntu 18.04 docker. We will take the static-compilation approach. Thanks @mhs4670go !

boost::program_option can be replaced with arser, after #3002 is landed. Again, thanks @mhs4670go