emscripten ports ICU missing icudt

[ldubost]

When compiling with USE_ICU, I'm getting the following error:

no symbols
no symbols
no symbols
no symbols
wasm-ld: error: /emsdk/.data/cache/wasm-obj/libicuuc.a(udata_a5e23168.cpp.o): undefined symbol: icudt62_dat
wasm-ld: error: /emsdk/.data/cache/wasm-obj/libicuuc.a(udata_a5e23168.cpp.o): undefined symbol: icudt62_dat
shared:ERROR: '/emsdk/upstream/bin/wasm-ld -o /tmp/emscripten_temp__hLcSu/x2t.wasm --allow-undefined --lto-O0 --no-check-features ./build/bin/linux_32/x2t.bc -L/emsdk/upstream/emscripten/system/local/lib ./build/lib/linux_32/libCryptoPPLib.a -L/emsdk/upstream/emscripten/system/lib ./build/lib/linux_32/libXlsFormatLib.a -L/emsdk/.data/cache/wasm-obj ./build/lib/linux_32/libkernel.so ./build/lib/linux_32/libgraphics.so ./build/lib/linux_32/libUnicodeConverter.so ./Common/3dParty/boost/boost_1_58_0/stage/lib/libboost_regex.a /emsdk/.data/cache/wasm-obj/libicuuc.a /emsdk/.data/cache/wasm-obj/libc-mt.a /emsdk/.data/cache/wasm-obj/libcompiler_rt.a /emsdk/.data/cache/wasm-obj/libc-wasm.a /emsdk/.data/cache/wasm-obj/libc++-mt-noexcept.a /emsdk/.data/cache/wasm-obj/libc++abi-mt-noexcept.a /emsdk/.data/cache/wasm-obj/libdlmalloc-mt.a /emsdk/.data/cache/wasm-obj/libpthread-mt.a /emsdk/.data/cache/wasm-obj/libc_rt_wasm.a --import-memory --import-table --shared-memory -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr --strip-debug --export __wasm_call_ctors --export __data_end --export main --export malloc --export free --export setThrew --export __errno_location --export fflush --export emscripten_get_global_libc --export __pthread_tsd_run_dtors --export _register_pthread_ptr --export pthread_self --export __emscripten_pthread_data_constructor --export emscripten_futex_wake --export memalign --export usleep --export _ZSt18uncaught_exceptionv --export htonl --export htons --export ntohs --export realloc --export _get_tzname --export _get_daylight --export _get_timezone --export _get_environ --export emscripten_builtin_memalign --export emscripten_builtin_free -z stack-size=5242880 --initial-memory=149880832 --no-entry --max-memory=149880832 --global-base=1024' failed (1)

Here is the final build command:

gcc -Wl,--no-check-features -O0 -s USE_ICU=1 -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s USE_PTHREADS=1 -s EXTRA_EXPORTED_RUNTIME_METHODS=["ccall"] -s EXPORT_ALL=1 -s ERROR_ON_UNDEFINED_SYMBOLS=0 -s TOTAL_MEMORY=149880832 -s WASM=1 ./build/bin/linux_32/x2t
.bc ./build/lib/linux_32/libCryptoPPLib.a ./build/lib/linux_32/libXlsFormatLib.a ./build/lib/linux_32/libkernel.so ./build/lib/linux_32/libgraphics.so ./build/lib/linux_32/libUnicodeConverter.so ./Common/3dParty/boost/boost_1_58_0/stage/lib/libboo
st_regex.a -lnodefs.js -o x2t.js

The build works when not using USE_ICU=1 but I'm then missing the icu methos:

warning: undefined symbol: ucnv_close_62
warning: undefined symbol: ucnv_fromUnicode_62
warning: undefined symbol: ucnv_getCCSID_62
warning: undefined symbol: ucnv_getMaxCharSize_62
warning: undefined symbol: ucnv_getMinCharSize_62
warning: undefined symbol: ucnv_getName_62
warning: undefined symbol: ucnv_openCCSID_62
warning: undefined symbol: ucnv_open_62
warning: undefined symbol: ucnv_toUnicode_62

which makes some part of the code fail while other parts are working.

It seems the libicudata is normally a separate library but it does not seem to be build in the ICU port.

If more info is needed I'm happy to provide it.

[ldubost]

I've been able to workaround the problem by copying the file

/emsdk/.data/ports/icu/icu-release-62-1/icu4c/source/stubdata/stubdata.cpp

from stubdata to common into

/emsdk/.data/ports/icu/icu-release-62-1/icu4c/source/common/stubdata.cpp

and then force a rebuild by removing the cache directory "icu" to force a rebuild:

/emsdk/.data/cache/wasm-obj/ports-builds/icu

This provided the function needed. I'm not sure how this could be fixed fully.

[ldubost]

I was necessary to remove

/emsdk/.data//cache/wasm-obj/libicuuc.a

to force the rebuild of icu

Alternatively it's also possible to add the stubdata.cpp to your own project for making the build go through with USE_ICU=1

[kripken]

I think to fully fix this tools/ports/icu.py needs to be updated. I think when it was added in #7167 we just got basic stuff working, as a beginning.

[elliott-wen]

I was looking at this problem recently. One potential problem of using stubdata is that it may throw errors as described in the ICU document.

==================================================
Why am I seeing a small ( only a few K ) instead of a large ( several megabytes ) data shared library (icudt)?
Opening ICU services fails with U_MISSING_RESOURCE_ERROR and u_init() returns failure.

ICU libraries always must link with the ICU data library. However, so that ICU can bootstrap itself, it first builds a 'stub' data library, in icu\source\stubdata, so that the tools can function. You should only use this in production if you are NOT using DLL-mode data access, in which case you are accessing ICU data as individual files, as an archive (.dat) file, or some other means. Normally, you should be using the larger library built from icu\source\data.

However, I havent figured out a way to compile ICU data for WebAssembly platform. I am also not sure it is a good idea to bundle a 20MB file for WebAssembly apps

[ldubost]

I don't believe it's needed to compile the whole icudata into WebAssembly. If you add the stub function then the project will compile with USE_ICU. In the end I did this in my project by adding the stub function in my own project instead of the workaround to build the ICU lib with it. However it's not easy for users of USE_ICU=1 to understand this is what is currently needed to make it work.

Of course, there might be some functions in ICU that make use of that data in the icudata libe. It's unclear when it is used.

In the case of the project I used ICU for, it did not seem to cause any problems not to have the remaining of the icudata. Just the stub fixed the problem and the compiled program worked.

What I don't fully understand, is how the USE_ICU would have ever worked when it was first created. It could be that the emscripten compiler was nicer about the missing symbols. I tried "-s ERROR_ON_UNDEFINED_SYMBOLS=0" but it has not effect.

[elliott-wen]

@ldubost It really depends on what kind of applications you are building. For example, if you are porting a text typesetter and want to support Arabic/Japanese/Chinese, you may need the data file. :)

[yowl]

ucal.h and ucol.h are also missing. Would they also get fixed if https://github.com/emscripten-core/emscripten/issues/10129#issuecomment-569715048 was addressed?

[yowl]

I'm not familiar with this at all but looking at the icu.py perhaps it can be changed from

    ports.install_header_dir(os.path.join(dest_path, 'icu4c', 'source', 'common', 'unicode'))

to

    ports.install_header_dir(os.path.join(dest_path, 'icu4c', 'source', 'common', 'unicode'))
    ports.install_header_dir(os.path.join(dest_path, 'icu4c', 'source', 'i18n', 'unicode'))

As that seems to be where the other header files are.

[yowl]

In the end I did

  def create():
    logging.info('building port: icu')

    source_path = os.path.join(ports.get_dir(), 'icu', 'icu-' + TAG)
    dest_path = os.path.join(shared.Cache.get_path('ports-builds'), 'icu')

    shutil.rmtree(dest_path, ignore_errors=True)
    shutil.copytree(source_path, dest_path)

    final = os.path.join(dest_path, libname)
    ports.build_port(os.path.join(dest_path, 'icu4c', 'source', 'common'), final, [os.path.join(dest_path, 'icu4c', 'source', 'common'), os.path.join(dest_path, 'icu4c', 'source', 'i18n')], ['-DU_COMMON_IMPLEMENTATION=1'])

    ports.install_header_dir(os.path.join(dest_path, 'icu4c', 'source', 'common', 'unicode'), 'common/unicode')
    ports.install_header_dir(os.path.join(dest_path, 'icu4c', 'source', 'i18n', 'unicode'), 'i18n/unicode')
    return final

But its still not right as it doesn't add the include directories and I have to do that on the command line.