Edgetpu: RuntimeError: Encountered unresolved custom op: edgetpu-custom-op.Node number 0 (edgetpu-custom-op) failed to prepare.

[BotScutters]

My setup:

Using a Coral M.2 Accelerator A+E key in a device running Ubuntu 16.04
The model file is a MobileNet v2 SSD object detection model compiled for edgetpu, and I have successfully run this model file without issue on other similar devices.

Running:

import tflite_runtime.interpreter as tflite

model = tflite.Interpreter(
    "/src/object_detection/model/model_edgetpu.tflite", 
    experimental_delegates=[tflite.load_delegate('libedgetpu.so.1')]
)
model.allocate_tensors()

The model.allocate_tensors() call results in the following RuntimeError:

RuntimeError: Encountered unresolved custom op: edgetpu-custom-op.Node number 0 (edgetpu-custom-op) failed to prepare.

Given that I've seen this code + model work on other devices, I'm guessing that the issue here is either a hardware connection or perhaps a corrupt/missing installation. Under what conditions should I expect to see this error? Do you know what's likely causing it now?

Thanks!

[Namburger]

Hi Scott :)

can you share the outputs of these commands?

dpkg -l | grep libedgetpu
python3 -c 'print(__import__("tflite_runtime").__version__)'

[BotScutters]

I should be able to check on that soon. Unfortunately it's a remote device that I can only access sporadically, so I'm running off of logs.

On a theoretically identical device that follows the same installation procedures, this is what I get:

➜ dpkg -l | grep libedgetpu
ii  libedgetpu1-std:amd64                      14.0                                            amd64        Support library for Edge TPU
➜ python3 -c 'print(__import__("tflite_runtime").__version__)'
2.1.0.post1

[Namburger]

Hi @scottamain
Yes, that is what I expected, the error:

RuntimeError: Encountered unresolved custom op: edgetpu-custom-op.Node number 0 (edgetpu-custom-op) failed to prepare.

means that the libedgetpu1-std package and the tflite_runtime package were built on different tensorflow commits. Those 2 packages 14.0 and 2.1.0.post1 are definitely compatible. So I would definitely check that first.

If it is still failing, could you try stracing the process when it fails and send me the log?

[BotScutters]

Thanks @Namburger , thanks for helping out on this! I should be able to get this info tomorrow. The full trace I have from the last time it failed is below:

020-06-29 12:13:39 GMT CRITICAL [Launcher.py:71] Error: Unable to load models. Encountered unresolved custom op: edgetpu-custom-op.Node number 0 (edgetpu-custom-op) failed to prepare.
Traceback (most recent call last):
  File "Launcher.py", line 68, in __load_models
    return Scene.load_models(config['model_definitions'], models_to_load)
  File "/opt/ObjectDetection/Scene.py", line 181, in load_models
    model_version)
  File "/opt/ObjectDetection/Object_Detection.py", line 44, in init
    self.model.allocate_tensors()
  File "/home/user/Envs/object_detection/lib/python3.5/site-packages/tflite_runtime/interpreter.py", line 242, in allocate_tensors
    return self._interpreter.AllocateTensors()
  File "/home/user/Envs/object_detection/lib/python3.5/site-packages/tflite_runtime/interpreter_wrapper.py", line 110, in AllocateTensors
    return _interpreter_wrapper.InterpreterWrapper_AllocateTensors(self)
RuntimeError: Encountered unresolved custom op: edgetpu-custom-op.Node number 0 (edgetpu-custom-op) failed to prepare.
Stack (most recent call last):
  File "Launcher.py", line 197, in <module>
    ObjectDetection(config=config).run()
  File "Launcher.py", line 32, in init
    models = self.__load_models(cameras_dict)
  File "Launcher.py", line 71, in __load_models
    logger.critical('Error: Unable to load models. {}'.format(e), exc_info=True, stack_info=True) 

Will you need more than that?

[Namburger]

Hey Scott,
yeah, that still not enough info to go from, unfortunately. Basically, with strace I'd be able to pinpoint the exact path of the share library you've loaded and get an md5sum of that, just to make sure that the correct library are installed; basically, to verify this: https://github.com/google-coral/edgetpu/issues/159#issuecomment-652060231

I guess one thing to clarify is that with shared libraries (.so) in linux, you can have multiple version installed and it loaded during runtime. That means that even if you have the correct library installed but somehow also have an older version, your program could load the wrong version and cause this type of behavior.

[BotScutters]

Ah, gotcha. I'll post back when I've got it, thanks!

[BotScutters]

Actually, I've just managed to get my local device to produce the same error.

To reproduce, I had an active Python thread that had run the following:

import tflite_runtime.interpreter as tflite

model = tflite.Interpreter(
    "/src/object_detection/model/model_edgetpu.tflite", 
    experimental_delegates=[tflite.load_delegate('libedgetpu.so.1')]
)
model.allocate_tensors()

With that thread still active, I ran the same thing in _a separate thread_. The new thread spat out the same RuntimeError: Encountered unresolved custom op: edgetpu-custom-op.Node number 0 (edgetpu-custom-op) failed to prepare. error.

I've sent you the associated strace via e-mail.

This makes me think that the device showing errors may have multiple processes trying to claim the TPU, with this error resulting as a result of a conflict there. Does that makes sense to you?

[Namburger]

@BotScutters Ohh, well that totally make sense :)
Yeah, the TPU resources is not threadsafe, you can't have more than one processes accessing the same device resource. The kernel doesn't allow this, and even if it's allowed, there are much inputs data that needs to be loaded on the TPU's RAM so 2 processes uploading this data on it and try to call invoke will surely cause some undefined behavior.

Now on the system that has the original issue, I supposed that one never works even for single thread? I saw the strace that you sent to my email, but I don't see libedgetpu.so being loaded at all. A normal strace with our stack should have a sucessfull call to load the dynamic library like this:

openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libedgetpu.so.1.0", O_RDONLY|O_CLOEXEC) = 3

That is the general idea, but I'll send you mode details via our email :)

[BotScutters]

Good morning, @Namburger,
I just got some data regarding this issue.

The response from the two commands you asked to run:

➜ dpkg -l | grep libedgetpu
ii libedgetpu1-std:amd64 14.0 amd64 Support library for Edge TPU

➜ python3 -c 'print(_import("tflite_runtime").version_)'
2.1.0.post1

And here's an excerpt from the strace log from when it was trying to load libedgetpu.so.1, and I've send you the complete log separately. It looks like it's searching in the wrong directories, and actually in some directories associated with OpenVINO (we've run previous experiments with an Intel Neural Compute Stick on this device), so that seems interesting. What determines the path(s) it checks while trying to load libedgetpu?

open("/opt/intel//computer_vision_sdk_2018.5.455/deployment_tools/inference_engine/external/hddl/lib/libedgetpu.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/opt/intel//computer_vision_sdk_2018.5.455/deployment_tools/inference_engine/external/gna/lib/libedgetpu.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/opt/intel//computer_vision_sdk_2018.5.455/deployment_tools/inference_engine/external/mkltiny_lnx/lib/libedgetpu.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/opt/intel//computer_vision_sdk_2018.5.455/deployment_tools/inference_engine/external/omp/lib/libedgetpu.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/opt/intel//computer_vision_sdk_2018.5.455/deployment_tools/inference_engine/lib/ubuntu_16.04/intel64/libedgetpu.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("tls/x86_64/libedgetpu.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("tls/libedgetpu.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("x86_64/libedgetpu.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("libedgetpu.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 8
fstat(8, {st_mode=S_IFREG|0644, st_size=130228, ...}) = 0
mmap(NULL, 130228, PROT_READ, MAP_PRIVATE, 8, 0) = 0x7f751b6fd000
close(8)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libedgetpu.so.1", O_RDONLY|O_CLOEXEC) = 8
read(8, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
fstat(8, {st_mode=S_IFREG|0644, st_size=942584, ...}) = 0
mmap(NULL, 953984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 8, 0) = 0x7f7515bc8000

Thanks again for all of your help on this!

[Namburger]

Hi @BotScutters
To my knowledge, loading the share libraries is a linux kernel's mechanism during run time of applications that has dependices. For instance, if you look in /usr/lib/x86_64-linux-gnu, there will be tons of shared libraries (including our libedgetpu.so, "so" = "shared objects"). Nearly all of your standard linux programs will be dependent on some share objects, for example, even the ls command needs some shared libraries:

$ ldd /bin/ls
        linux-vdso.so.1 (0x00007ffcc0feb000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x0000799154ea9000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000799154ce8000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x0000799154c74000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x0000799154c6f000)
        /lib64/ld-linux-x86-64.so.2 (0x0000799155113000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x0000799154c4e000)

Now the problem with linux, is that there is nothing preventing you from installing multiple version of the same .so file, and that is why I'm suspecting that your .so for libedgetpu that your program loads may not match what is required by tflite_runtime 2.1.0.post1.

If you set your LD_LIBRARY_PATH, that will tell the kernel to load from those paths first, for instance:

export LD_LIBARRY_PATH=/path/to/shared_objects:LD_LIBRARY_PATH
./run_your_program

will tells the kernel to looks for share libraries first in /path/to/shared_objects before looking in the rest of what was in LD_LIBRARY_PATH before.

From the logs, it looks as if your kernel looks for libedgetpu in these directories first: /opt/intel//computer_vision_sdk_2018.5.455/deployment_tools/inference_engine/* but all failed with return code -1.
The one that succeeded was /usr/lib/x86_64-linux-gnu/libedgetpu.so.1, so could you share with me the outputs of:

md5sum /usr/lib/x86_64-linux-gnu/libedgetpu.so.1

Thanks!

[BotScutters]

Hi @Namburger

Thanks for getting back to me and I hope you had a nice holiday weekend.

Sure enough, LD_LIBRARY_PATH had changed from the default, thanks for pointing that out.

The md5sum for /usr/lib/x86_64-linux-gnu/libedgetpu.so.1 was 980ef8d9db4585893d2280c1e2ab61cf.

[Namburger]

Nice! From the md5sum, it is indeed loading libedgetpu.so version 14 standard, does this fixes your issue now?

[BotScutters]

Doesn't seem to fix the issue, unfortunately. The device has been taken offline, so I won't be able to investigate this for a bit, but once it's back up I should have improved access to it and will revisit this issue. Thanks Nam!

[mrusme]

Had the same issue for a Python script I've been working on, solved it by replacing the tflite.Interpreter(... call from @BotScutters (up there in the issue description) with a call to this function:

def make_interpreter(model_file):
  model_file, *device = model_file.split('@')
  return tflite.Interpreter(
      model_path=model_file,
      experimental_delegates=[
          tflite.load_delegate(EDGETPU_SHARED_LIB,
                               {'device': device[0]} if device else {})
      ])

Also added the following to the script head:

EDGETPU_SHARED_LIB = {
  'Linux': 'libedgetpu.so.1',
  'Darwin': 'libedgetpu.1.dylib',
  'Windows': 'edgetpu.dll'
}[platform.system()]

That solved the issue, at least in my case.

[Namburger]

@mrusme oh, that's most likely because EDGETPU_SHARED_LIB didn't mapped to the correct platform