Riot: tests/pkg_tensorflow-lite: tests randomly failing on nrf52dk and esp32-wroom-32

Created on 15 Jan 2020  路  18Comments  路  Source: RIOT-OS/RIOT

Description

captured output (from murdock):

nrf52: https://gist.github.com/c35dd538a85c7a52ff987da7214d20dc
esp32-wroom-32: https://gist.github.com/0a3ac0d0362d2d8a52e289c54c2eee3b

Steps to reproduce the issue

Expected results

Actual results

Versions

Solution

  • [x] Blacklist ESP32 boards in tests/pgk_tensorflow_lite as a quick fix of this issue. (#13140)
  • [x] Fix the C linkage compilation problem with ESP32 toolchain v2. (#13141)
  • [ ] Roll out the updated ESP32 toolchain v2 in docker. (https://github.com/RIOT-OS/riotdocker/pull/97)
  • [ ] Remove the workaround implementation of __cxa_guard_* functions.
  • [ ] Remove ESP32 from the blacklist in tests/pgk_tensorflow_lite.
tests bug
Source
picture kaspar030

All 18 comments

espr32-wroom-32 nrf52dk seems more serious since its a hardfault.

fjmolinas picture fjmolinas on 15 Jan 2020

espr32-wroom-32 seems more serious since its a hardfault.

I guess you mean nrf52dk. But probably the cause is the same.

kaspar030 picture kaspar030 on 15 Jan 2020
👍1

Failed again on PR #12848, hardfaults on nrf52 as well:

esp32-wroom-32: https://gist.github.com/fjmolinas/77016d18c258cf8d815f10a72b11fddb
nrf52dk: https://gist.github.com/fjmolinas/fb8bcab2a9baddeaf6f5797715ce3a69

fjmolinas picture fjmolinas on 15 Jan 2020

Can't reproduce on nrf52dk:

TOOLACHAIN=llvm RIOT_CI_BUILD=1 BUILD_IN_DOCKER=1 TRIBE_CI=1 BOARD=nrf52dk make -C tests/pkg_tensorflow-lite/ flash term -j3

Welcome to pyterm!
Type '/exit' to exit.
2020-01-15 17:06:58,311 # READY
s
2020-01-15 17:06:58,366 # START
2020-01-15 17:06:58,370 # main(): This is RIOT! (Version: buildtest)
2020-01-15 17:06:58,392 # Digit prediction: 7

make: Leaving directory '/home/francisco/workspace/RIOT/tests/pkg_tensorflow-lite'
fjmolinas picture fjmolinas on 15 Jan 2020

I tested on esp32-wroom-32 and nrf52832-mdk (same CPU as nrf52dk)

on esp32, it indeed doesn't work: the firmware doesn't start at all.

BUILD_IN_DOCKER=1 make BOARD=esp32-wroom-32 -C tests/pkg_tensorflow-lite/ flash test
make: Entering directory '/work/riot/RIOT/tests/pkg_tensorflow-lite'
ESP32_SDK_DIR should be defined as /path/to/esp-idf directory
ESP32_SDK_DIR is set by default to /opt/esp/esp-idf
Launching build container using image "riot/riotbuild:latest".
docker run --rm -t -u "$(id -u)" \
    -v '/usr/share/zoneinfo/Europe/Paris:/etc/localtime:ro' -v '/work/riot/RIOT:/data/riotbuild/riotbase:delegated' -e 'RIOTBASE=/data/riotbuild/riotbase' -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' -e 'BUILD_DIR=/data/riotbuild/riotbase/build' -e 'RIOTPROJECT=/data/riotbuild/riotbase' -e 'RIOTCPU=/data/riotbuild/riotbase/cpu' -e 'RIOTBOARD=/data/riotbuild/riotbase/boards' -e 'RIOTMAKE=/data/riotbuild/riotbase/makefiles'      \
    -e 'BOARD=esp32-wroom-32' \
    -w '/data/riotbuild/riotbase/tests/pkg_tensorflow-lite/' \
    'riot/riotbuild:latest' make  'BOARD=esp32-wroom-32' 'EXTERNAL_MODULE_DIRS=/data/riotbuild/riotbase/tests/pkg_tensorflow-lite/mnist' 
ESP32_SDK_DIR should be defined as /path/to/esp-idf directory
ESP32_SDK_DIR is set by default to /opt/esp/esp-idf
Building application "tests_pkg_tensorflow-lite" for "esp32-wroom-32" with MCU "esp32".

make[1]: Nothing to be done for 'prepare'.
make[1]: Nothing to be done for 'prepare'.
make[1]: Nothing to be done for 'prepare'.
"make" -C /data/riotbuild/riotbase/pkg/flatbuffers
"make" -C /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/bin/pkg/esp32-wroom-32/flatbuffers/src -f /data/riotbuild/riotbase/pkg/flatbuffers/Makefile.flatbuffers
"make" -C /data/riotbuild/riotbase/pkg/gemmlowp
"make" -C /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/bin/pkg/esp32-wroom-32/gemmlowp -f /data/riotbuild/riotbase/pkg/gemmlowp/Makefile.gemmlowp
"make" -C /data/riotbuild/riotbase/pkg/tensorflow-lite
"make" -C /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/bin/pkg/esp32-wroom-32/tensorflow-lite/tensorflow/lite/c -f /data/riotbuild/riotbase/pkg/tensorflow-lite/Makefile.tensorflow-lite-c
"make" -C /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/bin/pkg/esp32-wroom-32/tensorflow-lite/tensorflow/lite/core/api -f /data/riotbuild/riotbase/pkg/tensorflow-lite/Makefile.tensorflow-lite-core
"make" -C /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/bin/pkg/esp32-wroom-32/tensorflow-lite/tensorflow/lite/kernels -f /data/riotbuild/riotbase/pkg/tensorflow-lite/Makefile.tensorflow-lite-kernels
"make" -C /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/bin/pkg/esp32-wroom-32/tensorflow-lite/tensorflow/lite/kernels/internal -f /data/riotbuild/riotbase/pkg/tensorflow-lite/Makefile.tensorflow-lite-kernels-internal
"make" -C /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/bin/pkg/esp32-wroom-32/tensorflow-lite/tensorflow/lite/micro/kernels -f /data/riotbuild/riotbase/pkg/tensorflow-lite/Makefile.tensorflow-lite-micro-kernels
"make" -C /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/bin/pkg/esp32-wroom-32/tensorflow-lite/tensorflow/lite/micro/memory_planner -f /data/riotbuild/riotbase/pkg/tensorflow-lite/Makefile.tensorflow-lite-memory
"make" -C /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/bin/pkg/esp32-wroom-32/tensorflow-lite/tensorflow/lite/micro -f /data/riotbuild/riotbase/pkg/tensorflow-lite/Makefile.tensorflow-lite
"make" -C /data/riotbuild/riotbase/boards/esp32-wroom-32
"make" -C /data/riotbuild/riotbase/boards/common/esp32
"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/cpu/esp32
"make" -C /data/riotbuild/riotbase/cpu/esp32/freertos
"make" -C /data/riotbuild/riotbase/cpu/esp32/periph
"make" -C /data/riotbuild/riotbase/cpu/esp32/vendor
"make" -C /data/riotbuild/riotbase/cpu/esp32/vendor/esp-idf
"make" -C /data/riotbuild/riotbase/cpu/esp32/vendor/esp-idf/driver
"make" -C /data/riotbuild/riotbase/cpu/esp32/vendor/esp-idf/esp32
"make" -C /data/riotbuild/riotbase/cpu/esp32/vendor/esp-idf/soc
"make" -C /data/riotbuild/riotbase/cpu/esp32/vendor/esp-idf/spi_flash
"make" -C /data/riotbuild/riotbase/cpu/esp_common
"make" -C /data/riotbuild/riotbase/cpu/esp_common/vendor
"make" -C /data/riotbuild/riotbase/cpu/esp_common/vendor/xtensa
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotbase/sys/cpp11-compat
"make" -C /data/riotbuild/riotbase/sys/div
"make" -C /data/riotbuild/riotbase/sys/isrpipe
"make" -C /data/riotbuild/riotbase/sys/log
"make" -C /data/riotbuild/riotbase/sys/luid
"make" -C /data/riotbuild/riotbase/sys/newlib_syscalls_default
"make" -C /data/riotbuild/riotbase/sys/posix/pthread
"make" -C /data/riotbuild/riotbase/sys/random
"make" -C /data/riotbuild/riotbase/sys/random/tinymt32
"make" -C /data/riotbuild/riotbase/sys/stdio_uart
"make" -C /data/riotbuild/riotbase/sys/test_utils/interactive_sync
"make" -C /data/riotbuild/riotbase/sys/timex
"make" -C /data/riotbuild/riotbase/sys/tsrb
"make" -C /data/riotbuild/riotbase/sys/xtimer
"make" -C /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/mnist
   text    data     bss     dec     hex filename
  92453   76520   19128  188101   2dec5 /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/tests_pkg_tensorflow-lite.elf
/work/riot/RIOT/dist/tools/esptool/esptool.py --chip esp32 elf2image -fm dout -fs 2MB -ff 40m    -o /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/tests_pkg_tensorflow-lite.elf.bin /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/tests_pkg_tensorflow-lite.elf; printf "\n" > /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/partitions.csv; printf "nvs, data, nvs, 0x9000, 0x6000\n" >> /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/partitions.csv; printf "phy_init, data, phy, 0xf000, 0x1000\n" >> /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/partitions.csv; printf "factory, app, factory, 0x10000, " >> /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/partitions.csv; ls -l /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/tests_pkg_tensorflow-lite.elf.bin | awk '{ print $5 }' >> /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/partitions.csv; python /work/riot/RIOT/cpu/esp32/gen_esp32part.py --disable-sha256sum --verify /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/partitions.csv /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/partitions.bin
esptool.py v2.4.0
Parsing CSV input...
/work/riot/RIOT/dist/tools/esptool/esptool.py --chip esp32 -p /dev/ttyUSB0 -b 460800 --before default_reset --after hard_reset write_flash -z -fm dout -fs detect -ff 40m    0x1000 /work/riot/RIOT/cpu/esp32/bin/bootloader.bin 0x8000 /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/partitions.bin 0x10000 /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/tests_pkg_tensorflow-lite.elf.bin
esptool.py v2.4.0
Connecting....
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse
MAC: 80:7d:3a:fd:d9:90
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash params set to 0x0320
Compressed 18432 bytes to 11321...
Wrote 18432 bytes (11321 compressed) at 0x00001000 in 0.3 seconds (effective 580.6 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 85...
Wrote 3072 bytes (85 compressed) at 0x00008000 in 0.0 seconds (effective 6053.2 kbit/s)...
Hash of data verified.
Compressed 200016 bytes to 114887...
Wrote 200016 bytes (114887 compressed) at 0x00010000 in 2.7 seconds (effective 589.3 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
r
/work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyUSB0" -b "115200" --noprefix --no-repeat-command-on-empty-line
Connect to serial port /dev/ttyUSB0
Welcome to pyterm!
Type '/exit' to exit.
READY
s
START
mets Jun  8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DOUT, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:4004
load:0x40078000,len:7304
load:0x40080000,len:7016
entry 0x40080344

Help: Press s to start test, r to print it is ready
s
^C

on the nrf52832-mdk, no crash, it works as expected:

make BOARD=nrf52832-mdk -C tests/pkg_tensorflow-lite/ flash test
make: Entering directory '/work/riot/RIOT/tests/pkg_tensorflow-lite'
Building application "tests_pkg_tensorflow-lite" for "nrf52832-mdk" with MCU "nrf52".

[INFO] cloning flatbuffers
Cloning into '/work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/flatbuffers'...
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 16123 (delta 1), reused 0 (delta 0), pack-reused 16109
Receiving objects: 100% (16123/16123), 9.48 MiB | 974.00 KiB/s, done.
Resolving deltas: 100% (11107/11107), done.
HEAD is now at 9e7e8cb Bumped version to 1.11.0
[INFO] updating flatbuffers /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/flatbuffers/.pkg-state.git-downloaded
echo v1.11.0 > /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/flatbuffers/.pkg-state.git-downloaded
[INFO] patch flatbuffers
[INFO] cloning gemmlowp
Cloning into '/work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/gemmlowp'...
remote: Enumerating objects: 16, done.
remote: Counting objects: 100% (16/16), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 3488 (delta 6), reused 9 (delta 4), pack-reused 3472
Receiving objects: 100% (3488/3488), 3.56 MiB | 1.05 MiB/s, done.
Resolving deltas: 100% (2396/2396), done.
HEAD is now at dc69acd fix avx header include
[INFO] updating gemmlowp /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/gemmlowp/.pkg-state.git-downloaded
echo dc69acdf61d7a64260ae0eb9c17421fef0488c02 > /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/gemmlowp/.pkg-state.git-downloaded
[INFO] patch gemmlowp
[INFO] cloning tensorflow-lite
Cloning into '/work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/tensorflow-lite'...
remote: Enumerating objects: 804240, done.
remote: Total 804240 (delta 0), reused 0 (delta 0), pack-reused 804240
Receiving objects: 100% (804240/804240), 459.24 MiB | 724.00 KiB/s, done.
Resolving deltas: 100% (650425/650425), done.
HEAD is now at 1768c8f2fa Merge pull request #34770 from ROCmSoftwarePlatform:google-upstream-eigen-patch
[INFO] updating tensorflow-lite /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/tensorflow-lite/.pkg-state.git-downloaded
echo 1768c8f2fa155d4c6406e8ff7addf374c83de7ad > /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/tensorflow-lite/.pkg-state.git-downloaded
[INFO] patch tensorflow-lite
"make" -C /work/riot/RIOT/pkg/flatbuffers
"make" -C /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/flatbuffers/src -f /work/riot/RIOT/pkg/flatbuffers/Makefile.flatbuffers
"make" -C /work/riot/RIOT/pkg/gemmlowp
"make" -C /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/gemmlowp -f /work/riot/RIOT/pkg/gemmlowp/Makefile.gemmlowp
"make" -C /work/riot/RIOT/pkg/tensorflow-lite
"make" -C /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/tensorflow-lite/tensorflow/lite/c -f /work/riot/RIOT/pkg/tensorflow-lite/Makefile.tensorflow-lite-c
"make" -C /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/tensorflow-lite/tensorflow/lite/core/api -f /work/riot/RIOT/pkg/tensorflow-lite/Makefile.tensorflow-lite-core
"make" -C /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/tensorflow-lite/tensorflow/lite/kernels -f /work/riot/RIOT/pkg/tensorflow-lite/Makefile.tensorflow-lite-kernels
"make" -C /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/tensorflow-lite/tensorflow/lite/kernels/internal -f /work/riot/RIOT/pkg/tensorflow-lite/Makefile.tensorflow-lite-kernels-internal
"make" -C /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/tensorflow-lite/tensorflow/lite/micro/kernels -f /work/riot/RIOT/pkg/tensorflow-lite/Makefile.tensorflow-lite-micro-kernels
"make" -C /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/tensorflow-lite/tensorflow/lite/micro/memory_planner -f /work/riot/RIOT/pkg/tensorflow-lite/Makefile.tensorflow-lite-memory
"make" -C /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/pkg/nrf52832-mdk/tensorflow-lite/tensorflow/lite/micro -f /work/riot/RIOT/pkg/tensorflow-lite/Makefile.tensorflow-lite
"make" -C /work/riot/RIOT/boards/nrf52832-mdk
"make" -C /work/riot/RIOT/core
"make" -C /work/riot/RIOT/cpu/nrf52
"make" -C /work/riot/RIOT/cpu/cortexm_common
"make" -C /work/riot/RIOT/cpu/cortexm_common/periph
"make" -C /work/riot/RIOT/cpu/nrf52/periph
"make" -C /work/riot/RIOT/cpu/nrf5x_common
"make" -C /work/riot/RIOT/cpu/nrf5x_common/periph
"make" -C /work/riot/RIOT/drivers
"make" -C /work/riot/RIOT/drivers/periph_common
"make" -C /work/riot/RIOT/sys
"make" -C /work/riot/RIOT/sys/auto_init
"make" -C /work/riot/RIOT/sys/cpp11-compat
"make" -C /work/riot/RIOT/sys/div
"make" -C /work/riot/RIOT/sys/isrpipe
"make" -C /work/riot/RIOT/sys/newlib_syscalls_default
"make" -C /work/riot/RIOT/sys/stdio_uart
"make" -C /work/riot/RIOT/sys/test_utils/interactive_sync
"make" -C /work/riot/RIOT/sys/timex
"make" -C /work/riot/RIOT/sys/tsrb
"make" -C /work/riot/RIOT/sys/xtimer
"make" -C /work/riot/RIOT/tests/pkg_tensorflow-lite/mnist
   text    data     bss     dec     hex filename
  98940     280   13124  112344   1b6d8 /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/nrf52832-mdk/tests_pkg_tensorflow-lite.elf
/work/riot/RIOT/dist/tools/pyocd/pyocd.sh flash /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/nrf52832-mdk/tests_pkg_tensorflow-lite.hex
### Flashing Target ###
[====================] 100%
0007455:INFO:loader:Erased 102400 bytes (25 sectors), programmed 102400 bytes (25 pages), skipped 0 bytes (0 pages) at 14.45 kB/s
Done flashing
r
/work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
Connect to serial port /dev/ttyACM0
Welcome to pyterm!
Type '/exit' to exit.
READY
s
START
main(): This is RIOT! (Version: 2020.01-devel-1843-g367250)

Digit prediction: 7

make: Leaving directory '/work/riot/RIOT/tests/pkg_tensorflow-lite'

Note that on esp32, the generated firmware has a very large data section, this doesn't seem normal:

   text    data     bss     dec     hex filename
  92453   76520   19128  188101   2dec5 /data/riotbuild/riotbase/tests/pkg_tensorflow-lite/bin/esp32-wroom-32/tests_pkg_tensorflow-lite.elf

Compared to nrf52, it's only 280:

   text    data     bss     dec     hex filename
  98940     280   13124  112344   1b6d8 /work/riot/RIOT/tests/pkg_tensorflow-lite/bin/nrf52832-mdk/tests_pkg_tensorflow-lite.elf
aabadie picture aabadie on 15 Jan 2020

Maybe @gschorcht could have an idea on what is wrong with esp32 ?

aabadie picture aabadie on 15 Jan 2020

For esp32, I suspect something when building cpp files. The generated firmware is always insanely bigger than with other ARM platforms. Example with tests/cpp11_thread:

  • nrf52:
   text    data     bss     dec     hex filename
  29776     152    2700   32628    7f74 /data/riotbuild/riotbase/tests/cpp11_thread/bin/nrf52832-mdk/tests_cpp11_thread.elf
  • esp32:
   text    data     bss     dec     hex filename
  81041   69260    6944  157245   2663d /data/riotbuild/riotbase/tests/cpp11_thread/bin/esp32-wroom-32/tests_cpp11_thread.elf
aabadie picture aabadie on 15 Jan 2020

For esp32, I suspect something when building cpp files. The generated firmware is always insanely bigger than with other ARM platforms.

ESP32 is not an ARM :thinking:

gschorcht picture gschorcht on 15 Jan 2020

I will check when I'm back at my desk.

gschorcht picture gschorcht on 15 Jan 2020

ESP32 is not an ARM

Sure, it was just to give some numbers to compare with. Maybe this doesn't make sense.

aabadie picture aabadie on 15 Jan 2020

ESP32 is not an ARM

Sure, it was just to give some numbers to compare with. Maybe this doesn't make sense.

I was just a bit confused because of the statement: bigger than with other ARM platforms :wink:

gschorcht picture gschorcht on 15 Jan 2020

I got tests/pkg_tensorflow-lite running on esp32-wroom-32 :smile:

--- Miniterm on /dev/ttyUSB0  115200,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
START
main(): This is RIOT! (Version: 2020.01-devel-1843-g36725)
Digit prediction: 7

but the problem is indeed serious and a more general problem of C++ on ESP32 :worried: We have to decide on how to deal with it.

The Problem

With tests/pkg_tensorflow-lite I got the following backtrace in GDB:

#0  abort () at /home/gs/esp/crosstool-NG/.build/src/newlib-2.2.0/newlib/libc/stdlib/abort.c:62
#1  0x400e0992 in __cxxabiv1::__terminate (handler=0x400d62c0 <abort>) at /home/gs/esp/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
#3  0x400e0d91 in __cxxabiv1::__cxa_get_globals () at /home/gs/esp/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/eh_globals.cc:133
#4  0x400e08fa in __cxxabiv1::__cxa_throw (obj=0x3ffb57c0, tinfo=0x3f4105f8 <typeinfo for __gnu_cxx::__concurrence_lock_error>, dest=0x400dfea0 <__gnu_cxx::__concurrence_lock_error::~__concurrence_lock_error()>) at /home/gs/esp/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/eh_throw.cc:65
#5  0x400dff71 in __gnu_cxx::__throw_concurrence_lock_error () at /home/gs/esp/crosstool-NG/.build/xtensa-esp32-elf/build/build-cc-gcc-final/xtensa-esp32-elf/libstdc++-v3/include/ext/concurrence.h:102
#6  __gnu_cxx::__recursive_mutex::lock (this=<optimized out>) at /home/gs/esp/crosstool-NG/.build/xtensa-esp32-elf/build/build-cc-gcc-final/xtensa-esp32-elf/libstdc++-v3/include/ext/concurrence.h:208
#7  (anonymous namespace)::mutex_wrapper::mutex_wrapper (this=0x3ffb2940 <main_stack+2788>) at /home/gs/esp/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/guard.cc:74
#8  0x400e0031 in __cxxabiv1::__cxa_guard_acquire (g=0x3ffb3c28 <guard variable for setup()::micro_error_reporter>) at /home/gs/esp/crosstool-NG/.build/src/gcc-5.2.0/libstdc++-v3/libsupc++/guard.cc:296
#9  0x400d98d9 in setup () at /home/gs/src/RIOT-Xtensa-ESP.working/tests/pkg_tensorflow-lite/mnist/main_functions.cc:50
#10 0x400d97d6 in main (argc=0, argv=0x0) at /home/gs/src/RIOT-Xtensa-ESP.working/tests/pkg_tensorflow-lite/main.cpp:28

That is, the system lock was simply an abortdue to an exception in __cxa_guard_acquire which is called on static object initialization in the setup function. This led me to the problem.

When I was porting RIOT to ESP32, I started with the existing ESP32 tool chain from the vendor. Unfortunately, this toolchain was generated with crosstool-NG and thread support using option --enable-threads=posix. That in turn led to the problem that the __cxa_guard_* functions used the pthread_once function from module pthread where the parameter once was of incompatible type.

When I was trying to enable C++ on ESP32, I therefore replaced the __cxa_guard_* functions in libstdc++.a with an own version, see PR #12060. This workaround was working for all our C++ applications. Obviously, it doesn't work with tests/pgk_tensorflow_lite.

How to solve

The problem can be solved with a recompiled tool chain version in which the thread support is disabled. I have already such a version on my local system.

That is, we have to roll out an updated toolchain. Afterwards, we can remove the workaround introduced with PR #12060.

What are the options

I suggest the following approach:

  1. Blacklist ESP32 boards for the moment in `tests/pgk_tensorflow_lite as a quick fix of this issue.
  2. Roll out an updated toolchain in docker
  3. Remove the workaround implementation of __cxa_guard_* functions.

I could provide everything as different PRs within one day.

What do you think?

gschorcht picture gschorcht on 15 Jan 2020
👍1

Thanks for the detailed and clear analysis @gschorcht.

What do you think?

Your plan sounds good.

aabadie picture aabadie on 15 Jan 2020

Besides RIOT_VERSION anything that I might be doing different that might be happening in murdock compared to TOOLACHAIN=llvm RIOT_CI_BUILD=1 BUILD_IN_DOCKER=1 BOARD=nrf52dk make -C tests/pkg_tensorflow-lite/ flash term -j3?

fjmolinas picture fjmolinas on 15 Jan 2020

@gschorcht @fjmolinas, I opened #13140 in order to blacklist ESP32 architecture for tensorflow-lite until the toolchain is updated in Docker.

aabadie picture aabadie on 16 Jan 2020

Regarding the nrf52dk, I can reproduce the hard fault with LLVM and not only on nrf52dk, it affects all enabled Cortex-M cpus (M3, M4 and M7). I'm suspecting something with the FPU but I'm unsure.

Maybe I suggest that we disable the LLVM toolchain on Cortex-M ?

aabadie picture aabadie on 16 Jan 2020

@fjmolinas, there's a typo in your command (TOOLACHAIN instead of TOOLCHAIN) and I'm wondering if the TOOLCHAIN variable is passed to Docker when it's used like this.

aabadie picture aabadie on 16 Jan 2020

@fjmolinas, I opened https://github.com/tensorflow/tensorflow/issues/35939 on TensorFlow ;) because I don't really understand what the problem is (it happens in TensorFlow itself).
To not block other PRs with unrelated failures, I suggest to blacklist the LLVM toolchain for Cortex-M. Do you agree ? Note that I think this PR would also need a backport.

aabadie picture aabadie on 16 Jan 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

chrysn picture chrysn  路  5Comments
silkeh picture silkeh  路  5Comments
jue89 picture jue89  路  5Comments
kaspar030 picture kaspar030  路  6Comments
OlegHahm picture OlegHahm  路  4Comments