Dali: [QUESTION] undefined symbol (which is marked DLL_PUBLIC) when loading custom operator

[kindoblue]

I've hacked in place a custom operator, a FileReader variant, as a separate project (out of tree component)

The error is the following:

/home/ice/.virtualenvs/deep/bin/python: symbol lookup error: ./cmake-build-debug/libCustomOp.so: undefined symbol: _ZN4dali11start_indexEmmm

If you unmangled that string, the missing function is: dali::start_index(unsigned long, unsigned long, unsigned long)

In the source code, it is defined like that:

namespace dali {

DLL_PUBLIC size_t start_index(const size_t shard_id,
                              const size_t shard_num,
                              const size_t size);
.
.

But the symbol is not defined indeed. The operator seems to be compiled correctly, and among the dependencies you have the DALI libraries:

cmake-build-debug ldd libCustomOp.so
    linux-vdso.so.1 (0x00007ffd5e7c6000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8fd52fc000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8fd52f1000)
    libdali.so => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali/libdali.so (0x00007f8fd43f4000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8fd43d1000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8fd41f0000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8fd41d5000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8fd3fe1000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f8fd54ba000)
    libdali_kernels.so => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali/libdali_kernels.so (0x00007f8fc18b3000)
    libjpeg-644e261f.so.62 => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali//.libs/libjpeg-644e261f.so.62 (0x00007f8fc161b000)
    libtiff-05b9aade.so.5 => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali//.libs/libtiff-05b9aade.so.5 (0x00007f8fc139d000)
    libsndfile-ac31f8d9.so.1 => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali//.libs/libsndfile-ac31f8d9.so.1 (0x00007f8fc1121000)
    libavformat-dcba000c.so.58 => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali//.libs/libavformat-dcba000c.so.58 (0x00007f8fc0ea7000)
    libavcodec-d8696a33.so.58 => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali//.libs/libavcodec-d8696a33.so.58 (0x00007f8fc0c3a000)
    libavfilter-1bda4593.so.7 => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali//.libs/libavfilter-1bda4593.so.7 (0x00007f8fc0a18000)
    libavutil-1fe3f55e.so.56 => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali//.libs/libavutil-1fe3f55e.so.56 (0x00007f8fc0706000)
    libdali_core.so => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali/libdali_core.so (0x00007f8fc0457000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8fc0306000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8fc02ea000)
    libFLAC-b2df50ea.so.8 => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali//.libs/libFLAC-b2df50ea.so.8 (0x00007f8fc0070000)
    libogg-8261e0c3.so.0 => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali//.libs/libogg-8261e0c3.so.0 (0x00007f8fbfe66000)
    libvorbis-2a78489a.so.0 => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali//.libs/libvorbis-2a78489a.so.0 (0x00007f8fbfc28000)
    libvorbisenc-ebfb9cea.so.2 => /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali//.libs/libvorbisenc-ebfb9cea.so.2 (0x00007f8fbf976000)

But in the defined symbols, no dali::start_index, as you can see:

nm /home/ice/.virtualenvs/deep/lib/python3.7/site-packages/nvidia/dali/libdali.so | grep _ZN4dali11   

                 U _ZN4dali11DeviceGuardC1Ei@@DALI_CORE_0.26.0dev
                 U _ZN4dali11DeviceGuardD1Ev@@DALI_CORE_0.26.0dev
00000000001b3120 t _ZN4dali11GetExecutorIJiiimiiiNS_10QueueSizesEEEESt10unique_ptrINS_12ExecutorBaseESt14default_deleteIS3_EEbbbDpT_
0000000000185760 t _ZN4dali11make_stringIJA11_cSsSsEEESsDpRKT_
00000000000e7460 t _ZN4dali11make_stringIJA15_cNS_12DALIDataTypeEEEESsDpRKT_
00000000001084f0 t _ZN4dali11make_stringIJA15_cSsA43_cSsA3_cEEESsDpRKT_
00000000001d4c90 t _ZN4dali11make_stringIJA17_c13device_type_tEEESsDpRKT_
0000000000192bd0 t _ZN4dali11make_stringIJA17_cmA3_cSsEEESsDpRKT_
0000000000104df0 t _ZN4dali11make_stringIJA22_cSsA11_cSsA15_cSsEEESsDpRKT_
0000000000104cc0 t _ZN4dali11make_stringIJA22_cSsA11_cSsA19_cSsA17_cSsEEESsDpRKT_
00000000001b6870 t _ZN4dali11make_stringIJA24_cSsEEESsDpRKT_
00000000001b6270 t _ZN4dali11make_stringIJA26_ciEEESsDpRKT_
00000000001aa110 t _ZN4dali11make_stringIJA3_cSsA2_ciEEESsDpRKT_
00000000001d8840 t _ZN4dali11make_stringIJA65_ciA16_cmA10_cEEESsDpRKT_
00000000000f6ab0 t _ZN4dali11make_stringIJA7_ciA24_cEEESsDpRKT_
00000000001d93d0 t _ZN4dali11make_stringIJA87_cA4_cA12_cS2_A11_cEEESsDpRKT_
00000000000c8f10 t _ZN4dali11make_stringIJA91_cNS_11TensorShapeILin1EEEA10_cS3_A15_cS3_EEESsDpRKT_
00000000001b6240 t _ZN4dali11outCHWImageIhEEiRKSt6vectorIT_SaIS2_EEiiiiiiff
00000000001b6210 t _ZN4dali11outHWCImageIhEEiRKSt6vectorIT_SaIS2_EEiiiiiiff
000000000019bde0 t _ZN4dali11print_delimINS_12no_delimiterESsJA3_cEEEvRSoRKT_RKT0_DpRKT1_.isra.1356
000000000019bde0 t _ZN4dali11print_delimINS_12no_delimiterESsJA51_cEEEvRSoRKT_RKT0_DpRKT1_.isra.764
000000000019bde0 t _ZN4dali11print_delimINS_12no_delimiterESsJA64_cEEEvRSoRKT_RKT0_DpRKT1_.isra.767
0000000000148580 t _ZN4dali11SmallVectorIiLm16ESaIiEE12emplace_backIJRKiEEEvDpOT_
0000000000173b00 t _ZN4dali11SmallVectorIlLm256ESaIlEE7reserveEm
0000000000106300 t _ZN4dali11SmallVectorIlLm6ESaIlEE11copy_assignILm6ES1_EEvRKNS0_IlXT_ET0_EE
0000000000161a50 t _ZN4dali11SmallVectorIlLm6ESaIlEE7reserveEm
00000000001c76b0 t _ZN4dali11SmallVectorIlLm6ESaIlEEaSILm6EEERS2_ONS0_IlXT_ES1_EE
00000000001738c0 t _ZN4dali11SmallVectorIPKvLm256ESaIS2_EE7reserveEm
00000000001739e0 t _ZN4dali11SmallVectorIPvLm256ESaIS1_EE7reserveEm
00000000000f3820 t _ZN4dali11TensorShapeILin1EEC1IJEvEElDpT_
00000000001c7690 t _ZN4dali11TensorShapeILin1EEC1IJiEvEElDpT_
00000000000f3820 t _ZN4dali11TensorShapeILin1EEC2IJEvEElDpT_
00000000001c7690 t _ZN4dali11TensorShapeILin1EEC2IJiEvEElDpT_

Any suggestions on where to look? The forest of CMakeFiles is thick and difficult to navigate.

Thanks

[kindoblue]

Perhaps the reason is about how I include the function. From my code I include the header dali/operators/reader/loader/loader.h and DLL_PUBLIC should be resolve to extern or something like that

[kindoblue]

My code for the custom operator is here: https://github.com/kindoblue/seg_file_reader

[JanuszL]

Hi,
This symbol is part of operators part of DALI. So to use it you need to link to libdali_operators.so.

nm -C /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali_operators.so  | grep start_index
00000000025bfeed T dali::start_index(unsigned long, unsigned long, unsigned long)

[kindoblue]

Thanks a lot. Indeed if I link that library it finds the symbol

diff --git a/CMakeLists.txt b/CMakeLists.txt
index a895ff0..104b7d5 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -24,5 +24,5 @@ set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${DALI_COMPILE_FLAGS}")
 link_directories(CustomOp "${DALI_LIB_DIR}")

 cuda_add_library(CustomOp SHARED library.cpp file_file_loader.cc)
-target_link_libraries(CustomOp dali)
+target_link_libraries(CustomOp dali dali_operators)

At the beginning, I thought: oh gosh I need to fiddle with "extern" and stuff like that :-)

Thanks for the help