Edgetpu: unable to run tflite_runtime with libedgetpu on raspberry pi zero wh

[rhadnum]

Hi,

Having some issues getting the edgetpu runtime to work with the tflite_runtime package on my raspberry pi zero (with the USB accelerator) - I'm new to all this and I've tried everything I can think of, can anyone help? Findings are below, many many thanks in advance.

dpkg -l | grep edgetpu
ii  libedgetpu1-std:armhf                 14.1                                   armhf        Support library for Edge TPU
ii  python3-edgetpu                       14.1                                   armhf        Edge TPU Python API

Attempting using my own compiled tflite_runtime package following the steps specified here: https://github.com/google-coral/edgetpu/issues/198 results in: Illegal instruction:

python3 -c 'print(__import__("tflite_runtime").__version__)'
2.1.0
LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0 python3 index.python
Illegal instruction

and then I've tried using some prebuilt wheels that other people have posted on the various issues reported, the one below is tflite_runtime 2.2.0 (not sure if the rc3 makes any significant differences here)

python3 -c 'print(__import__("tflite_runtime").__version__)'
2.2.0rc3
LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0 python3 index.python
Traceback (most recent call last):
  File "index.python", line 94, in <module>
    interpreter.allocate_tensors()
  File "/home/pi/Documents/Python 3/Object Detection/tflite1-env/lib/python3.7/site-packages/tflite_runtime/interpreter.py", line 242, in allocate_tensors
    return self._interpreter.AllocateTensors()
  File "/home/pi/Documents/Python 3/Object Detection/tflite1-env/lib/python3.7/site-packages/tflite_runtime/interpreter_wrapper.py", line 115, in AllocateTensors
    return _interpreter_wrapper.InterpreterWrapper_AllocateTensors(self)
RuntimeError: Internal: Unsupported data type in custom op handler: 0Node number 2 (EdgeTpuDelegateForCustomOp) failed to prepare.

Then the last one is tflite_runtime 2.3.1:

python3 -c 'print(__import__("tflite_runtime").__version__)'
2.3.1
LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0 python3 index.python
Traceback (most recent call last):
  File "index.python", line 94, in <module>
    interpreter.allocate_tensors()
  File "/home/pi/Documents/Python 3/Object Detection/tflite1-env/lib/python3.7/site-packages/tflite_runtime/interpreter.py", line 243, in allocate_tensors
    return self._interpreter.AllocateTensors()
RuntimeError: Internal: Unsupported data type in custom op handler: 0Node number 2 (EdgeTpuDelegateForCustomOp) failed to prepare.

From what I can tell the last 2 issues (tflite_runtime_2.2.0rc3 & tflite_runtimme_2.3.1) are closer to success than the first, the issues reported before mine indicate some sort of mismatch between the commit of tensorflow that the libedgetpu runtime has been built with and the version used by the tflite_runtime, however my compiled version of tflite_runtime_2.1.0 should have fixed this shouldn't it?

(All of these have been ran making sure I have the correct libedgetpu runtime for the architecture of the RPI0 (armv6) by doing the below commands from the official edgetpu git repository (which I believe is 14.1?):

git clone https://github.com/google-coral/edgetpu.git
cd edgetpu/libedgetpu/throttled/armv6
sudo cp libedgetpu.so.1 /usr/lib/arm-linux-gnueabihf

but the above is just me guessing that this needs to be done really.

Once again, thanks in advance!

[Namburger]

@rhadnum hello, could you share the outputs of this:

$ uname -a

[rhadnum]

hi @Namburger, here you go:

uname -a
Linux raspberrypi 5.4.51+ #1333 Mon Aug 10 16:38:02 BST 2020 armv6l GNU/Linux

[Namburger]

The issue is that we no longer release binaries for armv6 as per requirements (although we still do for the libedgetpu.so). So when trying to use the tflite_runtime package, it is throwing some illegal instruction because it wasn't built for that.
Just to ensure that you've built the right commit, could you also share this outputs?

python3 -c 'print(__import__("tflite_runtime").__git_version__)'

[Edit] Also, this:

LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0 python3 index.python
Illegal instruction

Seems pretty bizzard, not something I've seen before, but I believe it must be part of what tflite_runtime was trying to load.

[rhadnum]

Ah right, that makes sense.

Weirdly it returns the below:

python3 -c 'print(__import__("tflite_runtime").__git_version__)'
0.6.0-76902-gd855adfc5a

although I definitely remember running git checkout d855adfc5a0195788bf5f92c3c7352e638aa1109 in the tensorflow repo root, am I doing something wrong here?

Regarding the LD_PRELOAD - if I don't specify that then it returns the below so not sure if I've compiled something wrong there:

python3 index.python
Traceback (most recent call last):
  File "/home/pi/Documents/Python 3/Object Detection/tflite1-env/lib/python3.7/site-packages/tflite_runtime/interpreter_wrapper.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /home/pi/Documents/Python 3/Object Detection/tflite1-env/lib/python3.7/site-packages/tflite_runtime/_interpreter_wrapper.so: undefined symbol: __atomic_compare_exchange_8

[Namburger]

Hope I'm not spamming or throwing too much info, but here is my suggestion:

1) As above, get your tflite_runtime package commit version (use one that doesn't cause your Illegal instruction issue, since that's an external dependencies that we can't control, I suggest your latest package 2.3.1 as it is more updated):

python3 -c 'print(__import__("tflite_runtime").__git_version__)'

2) Build your own libedgetpu.so from this repo: https://github.com/google-coral/libedgetpu/
Make sure to change the commit here to be the same commit with your tflite_runtime package and then build using (you'll need docker installed and do this on an x86_64 machine):

DOCKER_CPUS="armv6" DOCKER_TARGETS=libedgetpu make docker-build

Then you can copy the resulting libedgetpu.so to your rpi0 and mv it to /usr/lib/arm-linux-gnueabihf.

I do want to make a note that we stopped supporting armv6 and don't have any tests coverage for this architect, but I'd love to see some success stories! Please let me know if this works,
Cheers

[rhadnum]

Not spam at all, the more I can try the better! 😄

Brilliant yeah I never thought of building the edgetpu runtime myself, completely overlooked that.

This is more of a hobby project for me, curious to see what the performance improvements look like with the usb accelerator as without it's 0.07 fps when doing object detection on a sample model (which is understandable considering the memory limitations of the RPI0).

Once again, many thanks for the quick responses and the suggestions :)

[rhadnum]

Managed to get it working on my raspberry pi zero now following the steps mentioned by @Namburger. Performance is still subpar which is to be expected but a definite increase in fps at around 0.7 / 0.8 (roughly 10x with the usb accelerator!)

[Namburger]

Awesome!

[Namburger]

@rhadnum hey I was wondering if you also built your own tflite_runtime package for 2.3.1? I have another user who is trying to work with the pi0 and this may helps

[rhadnum]

Hi @Namburger - yeah I did manage to build it but it took a while to do (and I don't quite remember how I did it 😄 )

All the .whl files that I tried can be found here (only the 2.3.1 one worked in the end for me): https://drive.google.com/drive/folders/1VtkKbpKnv9vHylQ62ekuGvyco2Jg9pQe?usp=sharing

To build the libedgetpu runtime I:

1. got the commit hash for this specific build by running:

python3 -c 'print(__import__("tflite_runtime").__git_version__)'

1. Get the full commit hash by running:

git rev-parse {short-hash-here}

1. Then just build the libedgetpu runtime by using your instructions above and replacing the hash in this file with the commit hash returned from the above command.

Hope this helps!

[Namburger]

@rhadnum thanks for the package, you're amazing!

[sim-kelly]

Hi @rhadnum
Just wondering how you got the full hash of the tflite_runtime from the SHA? (perhaps I'm missing what git directory I'm supposed to me in)

[Namburger]

@sim-kelly I think after having the SHORT_HASH, you can use

git rev-parse SHORT_HASH

[sim-kelly]

OK - thanks so much for the help with this!!
So I think I'm just missing something in following this.
I have installed the specific .whl that @rhadnum linked (I now have 2.3.1 tflite_runtime).
When I get the short hash, following the instructions, I do not follow which .git directory (if any, and on what machine (the armv6 or x86_64) I should be in to run the following command

git rev-parse SHORT_HASH

[Namburger]

Oh, so after installing tflite_runtime 2.3.1, you need to run this:

$ python3 -c 'print(__import__("tflite_runtime").__git_version__)'

That give you the SHORT_HASH, then you can run git rev-parse to get the full hash

[Edit] tbh, I believe 2.3.1 should be this hash: fcc4b966f1265f466e82617020af93670141b009
But since I didn't build that package, there's no way for me to really know

[sim-kelly]

Yes - I have the short hash, but I cannot run the git rev-parse from the root directory?? (and I dont know which directory I should be in as I installed the .whl without touching git)

Oh - and @Namburger thanks for the hash, but I cannot get it to build with that hash [Edit: I also changed line 41 (TENSORFLOW_SHA256) to match the commit, and things compiled!]

[Namburger]

@sim-kelly so git rev-parse should works anywhere that you have git installed :)

I also changed line 41 (TENSORFLOW_SHA256) to match the commit, and things compiled!

Right, I forgot you need to change that too.

Now that you have it compiled, you can move the new libedgetpu.so to /usr/lib/arm-linux-gnueabihf. If the commits I sent is correct then everything should works!

[ricardodeazambuja]

These are the steps that worked for me:

Download tflite_runtime-2.3.1-cp37-cp37m-linux_armv6l.whl to the RPI Zero. Install it using:
$ sudo pip3 install tflite_runtime-2.3.1-cp37-cp37m-linux_armv6l.whl
Get the SHORT_HASH:
$ python3 -c 'print(__import__("tflite_runtime").__git_version__)'
It was 0.6.0-88810-g5681c179ef in this situation.

Clone the tensorflow repo and run:
$ git rev-parse 0.6.0-88810-g5681c179ef
It returned 5681c179eff80bce00e526303950b67b23cad14c.

Generate the SHA256 (the filename is the value returned by git rev-parse with tar.gz added):
wget https://github.com/tensorflow/tensorflow/archive/5681c179eff80bce00e526303950b67b23cad14c.tar.gz
shasum -a 5681c179eff80bce00e526303950b67b23cad14c.tar.gz
It returned 18c5798270ea27c6991ee9bbf1357884d8f91589554799f4b4e7abff0ab75278

Clone the libedgetpu repo, open the file workspace.bzl, find the lines:

TENSORFLOW_COMMIT = ...
TENSORFLOW_SHA256 = ...

and replace with

TENSORFLOW_COMMIT = "5681c179eff80bce00e526303950b67b23cad14c"
TENSORFLOW_SHA256 = "18c5798270ea27c6991ee9bbf1357884d8f91589554799f4b4e7abff0ab75278"

Now, still inside libedgetpu repo, run:
DOCKER_CPUS="armv6" DOCKER_TARGETS=libedgetpu make docker-build

It takes a while, but at the end it generates the out/direct/armv6/libedgetpu.so.1. Copy it to the RPI Zero:
scp out/direct/armv6/libedgetpu.so.1 pi@<RPI_IP>:/home/pi/.

Log into the RPI Zero (ssh pi@<RPI_IP>). Move the libedgetpu.so.1:
sudo mv libedgetpu.so.1 /usr/lib/arm-linux-gnueabihf/.

It was necessary to add udev rules:
sudo vi /etc/udev/rules.d/99-edgetpu-accelerator.rules

SUBSYSTEM=="usb",ATTRS{idVendor}=="1a6e",GROUP="plugdev"
SUBSYSTEM=="usb",ATTRS{idVendor}=="18d1",GROUP="plugdev"

And add the user to plugdev group:
sudo usermod -aG plugdev $USER
Connect the Coral TPU USB to a micro USB to USB OTG adapter (no external power)...

...after all that, it worked!
I tested it according to the Coral docs and I found it quite fast. On my laptop it was not that much faster.

On the RPI Zero W:

----INFERENCE TIME----
Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory.
117.7ms
11.7ms
13.4ms
11.8ms
13.4ms
-------RESULTS--------
Ara macao (Scarlet Macaw): 0.77734

On my laptop (USB3, beefy CPU...):

----INFERENCE TIME----
Note: The first inference on Edge TPU is slow because it includes loading the model into Edge TPU memory.
12.7ms
4.4ms
4.3ms
4.3ms
4.1ms
-------RESULTS--------
Ara macao (Scarlet Macaw): 0.77734

However, when I try to generate my own pip package (commit 2be88a7f1a42ca8ff623e6953d7439ec284f67b8, generating tflite_runtime-2.5.0-cp37-cp37m-linux_armv6l.whl), it needs the LD_PRELOAD=/usr/lib/arm-linux-gnueabihf/libatomic.so.1.2.0 and it always gives the Illegal instruction after import tflite_runtime.interpreter as tflite.

Here is my build_pip_package_with_bazel.sh in case someone can point me the way:

#!/usr/bin/env bash
# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
set -ex

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PYTHON="${CI_BUILD_PYTHON:-python3}"
VERSION_SUFFIX=${VERSION_SUFFIX:-}
export TENSORFLOW_DIR="${SCRIPT_DIR}/../../../.."
TENSORFLOW_LITE_DIR="${TENSORFLOW_DIR}/tensorflow/lite"
TENSORFLOW_VERSION=$(grep "_VERSION = " "${TENSORFLOW_DIR}/tensorflow/tools/pip_package/setup.py" | cut -d= -f2 | sed "s/[ '-]//g")
export PACKAGE_VERSION="${TENSORFLOW_VERSION}${VERSION_SUFFIX}"
BUILD_DIR="${SCRIPT_DIR}/gen/tflite_pip/${PYTHON}"
TENSORFLOW_TARGET=$1

# Fix container image for cross build.
if [ ! -z "${CI_BUILD_HOME}" ] && [ `pwd` = "/workspace" ]; then
  # Fix for curl build problem in 32-bit, see https://stackoverflow.com/questions/35181744/size-of-array-curl-rule-01-is-negative
  if [ "${TENSORFLOW_TARGET}" = "armhf" ]; then
    sudo sed -i 's/define CURL_SIZEOF_LONG 8/define CURL_SIZEOF_LONG 4/g' /usr/include/curl/curlbuild.h
    sudo sed -i 's/define CURL_SIZEOF_CURL_OFF_T 8/define CURL_SIZEOF_CURL_OFF_T 4/g' /usr/include/curl/curlbuild.h
  fi

  # The system-installed OpenSSL headers get pulled in by the latest BoringSSL
  # release on this configuration, so move them before we build:
  if [ -d /usr/include/openssl ]; then
    sudo mv /usr/include/openssl /usr/include/openssl.original
  fi
fi

# Build source tree.
rm -rf "${BUILD_DIR}" && mkdir -p "${BUILD_DIR}/tflite_runtime"
cp -r "${TENSORFLOW_LITE_DIR}/tools/pip_package/debian" \
      "${TENSORFLOW_LITE_DIR}/tools/pip_package/setup_with_bazel.py" \
      "${TENSORFLOW_LITE_DIR}/tools/pip_package/MANIFEST.in" \
      "${TENSORFLOW_LITE_DIR}/python/interpreter_wrapper" \
      "${BUILD_DIR}"
cp "${TENSORFLOW_LITE_DIR}/python/interpreter.py" \
   "${BUILD_DIR}/tflite_runtime"
echo "__version__ = '${PACKAGE_VERSION}'" >> "${BUILD_DIR}/tflite_runtime/__init__.py"
echo "__git_version__ = '$(git -C "${TENSORFLOW_DIR}" describe)'" >> "${BUILD_DIR}/tflite_runtime/__init__.py"

# Build python interpreter_wrapper.
cd "${BUILD_DIR}"
case "${TENSORFLOW_TARGET}" in
  armhf)
    BAZEL_FLAGS="--config=elinux_armhf
      --copt=-march=armv6
      --copt=-O3 --copt=-fno-tree-pre --copt=-fpermissive
      --define tensorflow_mkldnn_contraction_kernel=0
      --define=raspberry_pi_with_neon=false"
    ;;
  aarch64)
    BAZEL_FLAGS="--config=elinux_aarch64
      --define tensorflow_mkldnn_contraction_kernel=0
      --copt=-O3"
    ;;
  native)
    BAZEL_FLAGS="--copt=-O3 --copt=-march=native"
    ;;
  *)
    BAZEL_FLAGS="--copt=-O3"
    ;;
esac

# We need to pass down the environment variable with a possible alternate Python
# include path for Python 3.x builds to work.
export CROSSTOOL_PYTHON_INCLUDE_PATH

case "${TENSORFLOW_TARGET}" in
  windows)
    LIBRARY_EXTENSION=".pyd"
    ;;
  *)
    LIBRARY_EXTENSION=".so"
    ;;
esac

bazel build -c opt -s --config=monolithic --config=noaws --config=nogcp --config=nohdfs --config=nonccl \
  ${BAZEL_FLAGS} ${CUSTOM_BAZEL_FLAGS} //tensorflow/lite/python/interpreter_wrapper:_pywrap_tensorflow_interpreter_wrapper
cp "${TENSORFLOW_DIR}/bazel-bin/tensorflow/lite/python/interpreter_wrapper/_pywrap_tensorflow_interpreter_wrapper${LIBRARY_EXTENSION}" \
   "${BUILD_DIR}/tflite_runtime"
# Bazel generates the wrapper library with r-x permissions for user.
# At least on Windows, we need write permissions to delete the file.
# Without this, setuptools fails to clean the build directory.
chmod u+w "${BUILD_DIR}/tflite_runtime/_pywrap_tensorflow_interpreter_wrapper${LIBRARY_EXTENSION}"

# Build python wheel.
cd "${BUILD_DIR}"
case "${TENSORFLOW_TARGET}" in
  armhf)
    ${PYTHON} setup_with_bazel.py bdist --plat-name=linux-armv6l \
                       bdist_wheel --plat-name=linux-armv6l
    ;;
  aarch64)
    ${PYTHON} setup_with_bazel.py bdist --plat-name=linux-aarch64 \
                       bdist_wheel --plat-name=linux-aarch64
    ;;
  *)
    if [[ -n "${TENSORFLOW_TARGET}" ]] && [[ -n "${TENSORFLOW_TARGET_ARCH}" ]]; then
      ${PYTHON} setup_with_bazel.py bdist --plat-name=${TENSORFLOW_TARGET}-${TENSORFLOW_TARGET_ARCH} \
                         bdist_wheel --plat-name=${TENSORFLOW_TARGET}-${TENSORFLOW_TARGET_ARCH}
    else
      ${PYTHON} setup_with_bazel.py bdist bdist_wheel
    fi
    ;;
esac

echo "Output can be found here:"
find "${BUILD_DIR}"

# Build debian package.
if [[ "${BUILD_DEB}" != "y" ]]; then
  exit 0
fi

PYTHON_VERSION=$(${PYTHON} -c "import sys;print(sys.version_info.major)")
if [[ ${PYTHON_VERSION} != 3 ]]; then
  echo "Debian package can only be generated for python3." >&2
  exit 1
fi

DEB_VERSION=$(dpkg-parsechangelog --show-field Version | cut -d- -f1)
if [[ "${DEB_VERSION}" != "${PACKAGE_VERSION}" ]]; then
  cat << EOF > "${BUILD_DIR}/debian/changelog"
tflite-runtime (${PACKAGE_VERSION}-1) unstable; urgency=low

  * Bump version to ${PACKAGE_VERSION}.

 -- TensorFlow team <[email protected]>  $(date -R)

$(<"${BUILD_DIR}/debian/changelog")
EOF
fi

case "${TENSORFLOW_TARGET}" in
  armhf)
    dpkg-buildpackage -b -rfakeroot -us -uc -tc -d -a armhf
    ;;
  aarch64)
    dpkg-buildpackage -b -rfakeroot -us -uc -tc -d -a arm64
    ;;
  *)
    dpkg-buildpackage -b -rfakeroot -us -uc -tc -d
    ;;
esac

cat "${BUILD_DIR}/debian/changelog"

I changed the lines:

--copt=-march=armv7-a --copt=-mfpu=neon-vfpv4

to

--copt=-march=armv6

...and

--define=raspberry_pi_with_neon=true"

to

--define=raspberry_pi_with_neon=false"

...and finally

    ${PYTHON} setup_with_bazel.py bdist --plat-name=linux-armv7l \
                       bdist_wheel --plat-name=linux-armv7l

to

    ${PYTHON} setup_with_bazel.py bdist --plat-name=linux-armv6l \
                       bdist_wheel --plat-name=linux-armv6l

And this is the command I'm using inside the tensorflow repo:

$ tensorflow/tools/ci_build/ci_build.sh PI-PYTHON37 tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh armhf

It goes until the end and generates, among other things, tensorflow/lite/tools/pip_package/gen/tflite_pip/python3.7/dist/tflite_runtime-2.5.0-cp37-cp37m-linux_armv6l.whl, but as I said before, it doesn't work.

[Namburger]

@ricardodeazambuja wow this is actually very nice, many many thanks!