One: [onert] Illegal instruction with Tizen/RPi4

Created on 9 Sep 2020 · 16Comments · Source: Samsung/ONE

While working on #4154 , I got Illegal instruction and killed in last section of the document when execute without BACKENDS="cpu"

Product/out/bin/nnpackage_run \
--nnpackage /opt/usr/home/owner/media/models/mobilenet_v2_1.4_224

Result

Can't load libOpenCL.so: libOpenCL.so: cannot open shared object file: No such file or directory
Can't load libGLES_mali.so: libGLES_mali.so: cannot open shared object file: No such file or directory
Can't load libmali.so: libmali.so: cannot open shared object file: No such file or directory
Couldn't find any OpenCL library.
Illegal instruction (core dumped)

gdb back trace

[New LWP 77540]
[New LWP 77541]
[New LWP 77542]
Can't load libOpenCL.so: libOpenCL.so: cannot open shared object file: No such file or directory
Can't load libGLES_mali.so: libGLES_mali.so: cannot open shared object file: No such file or directory
Can't load libmali.so: libmali.so: cannot open shared object file: No such file or directory
Couldn't find any OpenCL library.
[New LWP 77543]
[New LWP 77544]
[New LWP 77545]
[New LWP 77546]

Thread 1 "nnpackage_run" received signal SIGILL, Illegal instruction.
0x0000007ff60679b8 in arm_compute::MemoryManagerOnDemand::~MemoryManagerOnDemand() () from /lib64/libarm_compute.so
(gdb) bt
# 0  0x0000007ff60679b8 in arm_compute::MemoryManagerOnDemand::~MemoryManagerOnDemand() () from /lib64/libarm_compute.so
# 1  0x0000007ff6c4c354 in onert::backend::acl_common::AclTensorManager<onert::backend::acl_neon::operand::INETensor, onert::backend::acl_neon::operand::NETensor, onert::backend::acl_neon::operand::NESubTensor>::~AclTensorManager() () from /opt/usr/nnfw-test/Product/out/bin/../lib/../lib/../lib/libbackend_acl_neon.so
# 2  0x0000007ff73d3974 in onert::exec::ExecutorBase::~ExecutorBase() () from /opt/usr/nnfw-test/Product/out/bin/../lib/../lib/libonert_core.so
# 3  0x0000007ff7432640 in onert::exec::LinearExecutor::~LinearExecutor() () from /opt/usr/nnfw-test/Product/out/bin/../lib/../lib/libonert_core.so
# 4  0x0000007ff73c88dc in std::_Sp_counted_ptr_inplace<std::unordered_map<onert::util::Index<unsigned int, onert::ir::SubgraphIndexTag>, std::unique_ptr<onert::exec::IExecutor, std::default_delete<onert::exec::IExecutor> >, std::hash<onert::util::Index<unsigned int, onert::ir::SubgraphIndexTag> >, std::equal_to<onert::util::Index<unsigned int, onert::ir::SubgraphIndexTag> >, std::allocator<std::pair<onert::util::Index<unsigned int, onert::ir::SubgraphIndexTag> const, std::unique_ptr<onert::exec::IExecutor, std::default_delete<onert::exec::IExecutor> > > > >, std::allocator<std::unordered_map<onert::util::Index<unsigned int, onert::ir::SubgraphIndexTag>, std::unique_ptr<onert::exec::IExecutor, std::default_delete<onert::exec::IExecutor> >, std::hash<onert::util::Index<unsigned int, onert::ir::SubgraphIndexTag> >, std::equal_to<onert::util::Index<unsigned int, onert::ir::SubgraphIndexTag> >, std::allocator<std::pair<onert::util::Index<unsigned int, onert::ir::SubgraphIndexTag> const, std::unique_ptr<onert::exec::IExecutor, std::default_delete<onert::exec::IExecutor> > > > > >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() ()
   from /opt/usr/nnfw-test/Product/out/bin/../lib/../lib/libonert_core.so
# 5  0x0000007ff7aa4f98 in std::_Sp_counted_ptr_inplace<onert::exec::Execution, std::allocator<onert::exec::Execution>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() ()
   from /opt/usr/nnfw-test/Product/out/bin/../lib/libnnfw-dev.so
# 6  0x0000007ff7a9d8a0 in nnfw_session::~nnfw_session() () from /opt/usr/nnfw-test/Product/out/bin/../lib/libnnfw-dev.so
# 7  0x0000007ff7a9ce18 in nnfw_close_session () from /opt/usr/nnfw-test/Product/out/bin/../lib/libnnfw-dev.so
# 8  0x00000000004158cc in ?? ()
# 9  0x0000007ff7683ff4 in __libc_start_main (main=0x4152f0, argc=3, argv=0x7ffffffcc8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>)
    at ../csu/libc-start.c:308
# 10 0x0000000000416ad8 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind further

Source

seanshpark

All 16 comments

Similar issue #3846 on QEMU

seanshpark on 9 Sep 2020

Hi @seanshpark

GPU DDK - MESA DDK for VC4 - for RPi4 doesn't support OpenCL so you cannot test inference on RPI4 with GPU.

daeinki on 9 Sep 2020

RPi4 doesn't support OpenCL so you cannot test inference on RPI4 with GPU.

Thanks! I know this fact but can we gracefully end the program without Illegal instruction (core dumped) ?
Or, is it like, I'm running in console test program...and user wont see this in usual Tizen Apps so don't need to fix anything?

seanshpark on 9 Sep 2020

Just leaving other informations... show assembly

   │0x7ff60679a4 <_ZN11arm_compute21MemoryManagerOnDemandD2Ev+44>   adrp   x21, 0x7ff6275000 <_ZTVN11arm_compute12NEMeanStdDevE+32>                                                                  │
   │0x7ff60679a8 <_ZN11arm_compute21MemoryManagerOnDemandD2Ev+48>   ldr    x0, [x21, #3992]                                                                                                          │
   │0x7ff60679ac <_ZN11arm_compute21MemoryManagerOnDemandD2Ev+52>   cbz    x0, 0x7ff6067a00 <_ZN11arm_compute21MemoryManagerOnDemandD2Ev+136>                                                        │
   │0x7ff60679b0 <_ZN11arm_compute21MemoryManagerOnDemandD2Ev+56>   add    x1, x20, #0x8                                                                                                             │
   │0x7ff60679b4 <_ZN11arm_compute21MemoryManagerOnDemandD2Ev+60>   mov    w0, #0xffffffff                 // #-1                                                                                    │
  >│0x7ff60679b8 <_ZN11arm_compute21MemoryManagerOnDemandD2Ev+64>   ldaddal        w0, w0, [x1]                                                                                                      │
   │0x7ff60679bc <_ZN11arm_compute21MemoryManagerOnDemandD2Ev+68>   cmp    w0, #0x1                                                                                                                  │
   │0x7ff60679c0 <_ZN11arm_compute21MemoryManagerOnDemandD2Ev+72>   b.eq   0x7ff6067a14 <_ZN11arm_compute21MemoryManagerOnDemandD2Ev+156>  // b.none

Cursor at ldaddal w0, w0, [x1] line.

(gdb) info register w0
w0             0xffffffff          4294967295
(gdb) info register x1
x1             0x60e1f8            6349304

seanshpark on 9 Sep 2020

I think ldaddal doesn't work in RPi4. I had a small test with this code and got same Illegal instruction (core dumped)

void asm02(void)
{
  int buffer = 1;
  int *pbuffer = &buffer;
  int result = 0x5678;
  int *presult = &result;

  asm("mov w0, #0xffffffff \n");
  asm("ldr x1, [%0] \n" : : "r"(pbuffer));
  asm("ldaddal w0, w0, [x1] \n");
  asm("str x1, [%0] \n" : : "r"(presult));

  std::cout << "result = " << result << std::endl;
}

and flag

set(CMAKE_CXX_FLAGS "-march=armv8.1-a")

But, with -march=armv8-a, compiler dropped this error

Error: selected processor does not support `ldaddal w0,w0,[x1]'

@daeinki , could you please provide a guide how to build ARMCL package for Tizen so that I could try with other options so that ldaddal don't generate?

seanshpark on 10 Sep 2020

👀1

With copied file gbs.conf from this repo,

gbs -c gbs.conf build -A aarch64 --ccache --include-all --keep-packs --incremental

this command seems to build :)

source from https://review.tizen.org/gerrit/#/admin/projects/platform/upstream/armcl with tizen branch.

I've applied this change

diff --git a/packaging/libarmcl.spec b/packaging/libarmcl.spec
index bf251ce92..86257402c 100644
--- a/packaging/libarmcl.spec
+++ b/packaging/libarmcl.spec
@@ -56,7 +56,7 @@ scons -j8             \
        os=tizen        \
        build=native    \
 %ifarch aarch64
-       arch=arm64-v8.2-a       \
+       arch=arm64-v8a  \
 %else
        arch=armv7a     \
 %endif

seanshpark on 10 Sep 2020

👍1

[ 9603s] error: /home/abuild/rpmbuild/BUILD/libarmcl-v20.05/libarmcl.manifest: Security manifest file read failed.

seanshpark on 10 Sep 2020

Copied libarmcl.manifest to ~/GBS-ROOT/local/BUILD-ROOTS/scratch.aarch64.0/home/abuild/rpmbuild/BUILD/libarmcl-v20.05 folder. And try again...
--> still same error

seanshpark on 10 Sep 2020

Copied libarmcl.manifest to ~/GBS-ROOT/local/BUILD-ROOTS/scratch.aarch64.0/home/abuild/rpmbuild/BUILD/libarmcl-v20.05 folder. And try again...
--> still same error

I will check it out.

daeinki on 10 Sep 2020

👍1

Could you try to build again with below patch?
In my case, libarmcl arm64 packaging has been completed.

diff --git a/packaging/libarmcl.spec b/packaging/libarmcl.spec
index bf251ce..7a812db 100644
--- a/packaging/libarmcl.spec
+++ b/packaging/libarmcl.spec
@@ -36,9 +36,10 @@ Summary:     Sample application and benchmark binaries to test ARM Compute Library

 %prep
 %setup -q
-cp %{SOURCE1001} .

 %build
+cp %{SOURCE1001} .
+

daeinki on 10 Sep 2020

👍1

Could you try to build again with below patch?

This patch worked! Thank you!

seanshpark on 10 Sep 2020

Could you try to build again with below patch?

This patch worked! Thank you!

I will apply this patch. Thanks.

daeinki on 10 Sep 2020

Now execution is OK with RPi4.

sh-3.2#  Product/out/bin/nnpackage_run \
--nnpackage /opt/usr/home/owner/media/models/mobilenet_v2_1.4_224

Package Filename /opt/usr/home/owner/media/models/mobilenet_v2_1.4_224
Can't load libOpenCL.so: libOpenCL.so: cannot open shared object file: No such file or directory
Can't load libGLES_mali.so: libGLES_mali.so: cannot open shared object file: No such file or directory
Can't load libmali.so: libmali.so: cannot open shared object file: No such file or directory
Couldn't find any OpenCL library.
===================================
MODEL_LOAD   takes 69.547 ms
PREPARE      takes 255.074 ms
EXECUTE      takes 356.188 ms
- MEAN     :  356.188 ms
- MAX      :  356.188 ms
- MIN      :  356.188 ms
- GEOMEAN  :  356.188 ms
===================================

seanshpark on 10 Sep 2020

🎉1

My local build had problem with installation

sh-3.2# rpm -i --force libarmcl-v20.05-1.aarch64.rpm
error: Failed dependencies:
        rpmlib(ShortCircuited) <= 4.9.0-1 is needed by libarmcl-v20.05-1.aarch64

So I've done this with

rpm -i --force --nodeps libarmcl-v20.05-1.aarch64.rpm

seanshpark on 10 Sep 2020

I'll close this issue after verification with new rpm package

seanshpark on 11 Sep 2020

Close as review.tizen.org change has landed

seanshpark on 14 Sep 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

[one-cmds] one-build failed with error message

YongseopKim · 3Comments

[Wiki] Update ONE-DCO Signed-off-by signature

kishcs · 3Comments

Channel-wise INT16 quantization support

jinevening · 3Comments

Compiler FE: Enable ReLU6 Op

binarman · 3Comments

Compiler FE: conversion of pytorch to TF with Ops

seanshpark · 3Comments