Tvm: Add tvm.contrib.sort.argsort to iOS RPC

[KayneWest]

HARDWARE

MacBook Pro (13-inch, 2016). mojave v10.14.3. Python 3.6.8 |Anaconda custom (64-bit). TVM version: '0.6.dev'

ISSUE

I can get gluoncv yolov3 and gluoncv ssd to run just fine with TVM on macOS as I have built tvm with USE_SORT=ON during cmake (https://discuss.tvm.ai/t/meet-an-error-when-run-the-deploy-ssd-tutorial/847). However, ssd and yolov3 are both throwing the same error when attempting to run through the ios_rpc app: Cannot find function tvm.contrib.sort.argsort in the imported modules or global registry. Also to note, https://github.com/dmlc/tvm/tree/master/apps/ios_rpc for both metal and cpu work fine on my test iPhone.

I have looked through the issues and discussions but can't seem to find a solution for this problem and I'm not sure if it's a user-error or a bug; so preemptive apologies if it's been covered.

ERROR READOUT

2019-04-25 12:00:47.904729-0500 tvmrpc[1726:285858] [12:00:47] /Users/mkrzus/github/tvm/apps/ios_rpc/tvmrpc/TVMRuntime.mm:150: Load module from /var/containers/Bundle/Application/39EC33ED-BE66-48A9-8F90-7158E4274DAB/tvmrpc.app/Frameworks/tvm/net.dylib ...
Traceback (most recent call last):
  File "tests/mxnet_ssd_test.py", line 152, in <module>
    class_ids, scores, bounding_boxs = run(graph, rlib, params, ctx)
  File "tests/mxnet_ssd_test.py", line 81, in run
    m.run()
  File "/Users/mkrzus/.local/lib/python3.6/site-packages/tvm-0.6.dev0-py3.6-macosx-10.7-x86_64.egg/tvm/contrib/graph_runtime.py", line 167, in run
    self._run()
  File "tvm/_ffi/_cython/./function.pxi", line 304, in tvm._ffi._cy3.core.FunctionBase.__call__
  File "tvm/_ffi/_cython/./function.pxi", line 239, in tvm._ffi._cy3.core.FuncCall
  File "tvm/_ffi/_cython/./function.pxi", line 228, in tvm._ffi._cy3.core.FuncCall3
  File "tvm/_ffi/_cython/./base.pxi", line 168, in tvm._ffi._cy3.core.CALL
tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (8) 9   tvmrpc                              0x0000000102348b04 tvm::runtime::GraphRuntime::CreateTVMOp(tvm::runtime::TVMOpParam const&, std::__1::vector<DLTensor, std::__1::allocator<DLTensor> > const&, unsigned long)::$_38::operator()() const + 172
  [bt] (7) 8   tvmrpc                              0x000000010226bad4 tvm::runtime::PackedFunc::CallPacked(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const + 60
  [bt] (6) 7   tvmrpc                              0x0000000102291988 std::__1::function<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const + 112
  [bt] (5) 6   tvmrpc                              0x00000001022f4098 std::__1::__function::__func<tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7, std::__1::allocator<tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7>, void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) + 92
  [bt] (4) 5   tvmrpc                              0x00000001022f54bc void std::__1::__invoke_void_return_wrapper<void>::__call<tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7&, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*>(tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7&&&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) + 88
  [bt] (3) 4   tvmrpc                              0x00000001022f554c decltype(std::__1::forward<tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7&>(fp)(std::__1::forward<tvm::runtime::TVMArgs>(fp0), std::__1::forward<tvm::runtime::TVMRetValue*>(fp0))) std::__1::__invoke<tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7&, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*>(tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7&&&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) + 96
  [bt] (2) 3   tvmrpc                              0x00000001022f569c tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const + 304
  [bt] (1) 2   tvmrpc                              0x00000001022621b4 dmlc::LogMessageFatal::~LogMessageFatal() + 28
  [bt] (0) 1   tvmrpc                              0x00000001022639bc dmlc::LogMessageFatal::~LogMessageFatal() + 80
  [bt] (8) 9   tvmrpc                              0x00000001022f4098 std::__1::__function::__func<tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7, std::__1::allocator<tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7>, void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) + 92
  [bt] (7) 8   tvmrpc                              0x00000001022f54bc void std::__1::__invoke_void_return_wrapper<void>::__call<tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7&, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*>(tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7&&&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) + 88
  [bt] (6) 7   tvmrpc                              0x00000001022f554c decltype(std::__1::forward<tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7&>(fp)(std::__1::forward<tvm::runtime::TVMArgs>(fp0), std::__1::forward<tvm::runtime::TVMRetValue*>(fp0))) std::__1::__invoke<tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7&, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*>(tvm::runtime::WrapPackedFunc(int (*)(void*, int*, int), std::__1::shared_ptr<tvm::runtime::ModuleNode> const&)::$_7&&&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) + 96
  [bt] (5) 6   net.dylib                           0x00000001059916bc fused_vision_non_max_suppression_1 + 584
  [bt] (4) 5   net.dylib                           0x0000000105991c3c fused_vision_non_max_suppression_1 + 1992
  [bt] (3) 4   tvmrpc                              0x000000010226b13c TVMBackendGetFuncFromEnv + 64
  [bt] (2) 3   tvmrpc                              0x000000010226b458 tvm::runtime::ModuleNode::GetFuncFromEnv(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) + 596
  [bt] (1) 2   tvmrpc                              0x00000001022621b4 dmlc::LogMessageFatal::~LogMessageFatal() + 28
  [bt] (0) 1   tvmrpc                              0x00000001022639bc dmlc::LogMessageFatal::~LogMessageFatal() + 80
  File "/Users/mkrzus/github/tvm/apps/ios_rpc/tvmrpc/../../../src/runtime/module.cc", line 111
TVMError: Except caught from RPC call: [12:02:12] /Users/mkrzus/github/tvm/apps/ios_rpc/tvmrpc/../../../src/runtime/module_util.cc:73: Check failed: ret == 0 (-1 vs. 0) : TVMError: Check failed: f != nullptr: Cannot find function tvm.contrib.sort.argsort in the imported modules or global registry
Test Case '-[tvmrpcLauncher testRPC]' passed (86.484 seconds).
Test Suite 'tvmrpcLauncher' passed at 2019-04-25 12:02:13.079.
     Executed 1 test, with 0 failures (0 unexpected) in 86.484 (86.488) seconds
Test Suite 'tvmrpcLauncher.xctest' passed at 2019-04-25 12:02:13.084.
     Executed 1 test, with 0 failures (0 unexpected) in 86.484 (86.493) seconds
Test Suite 'All tests' passed at 2019-04-25 12:02:13.089.
     Executed 1 test, with 0 failures (0 unexpected) in 86.484 (86.498) seconds
2019-04-25 12:02:13.155 xcodebuild[18489:879520] [MT] IDETestOperationsObserverDebug: 94.530 elapsed -- Testing started completed.
2019-04-25 12:02:13.155 xcodebuild[18489:879520] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2019-04-25 12:02:13.155 xcodebuild[18489:879520] [MT] IDETestOperationsObserverDebug: 94.530 sec, +94.530 sec -- end

CODE

```import os
import re
import sys
import argparse

from matplotlib import pyplot as plt

import tvm
from tvm import rpc
from tvm.contrib import util, xcode
from tvm.relay.testing.config import ctx_list
from tvm import relay
from tvm.contrib import graph_runtime
from tvm.contrib.download import download_testdata

from gluoncv import model_zoo, data, utils

parser = argparse.ArgumentParser()
parser.add_argument("--metal", default=False, type=bool,
help="to test with metal in addition to cpu")
parser.add_argument("--proxy-host", default="XXX.XXX.XXX.XXX", type=str,
help="the host path you entered in the rpc start function \
exp: python -m tvm.exec.rpc_proxy --host XXX.XXX.XXXX")
parser.add_argument("--destination",
default="platform=iOS,id=XXXXXXXXXX",
type=str, help="Example of expected string:\
platform=iOS,id=1234567890abcabcabcabc1234567890abcabcab")
parser.add_argument("--team-id", default="XXXXXXXXX",
type=str, help="apple code-sign for team-id")
parser.add_argument("--proxy-port", default=9090,
type=int, help="the port from the rpc launcher")
parser.add_argument("--key", default="iphone",
type=str, help="key from the rpc launcher")
parser.add_argument("--arch", default="arm64",
type=str, help="architecture of the given iPhone - defaulted to 6s")
parser.add_argument("--sdk", default="iphoneos",
type=str, help="the sdk used for this deployment")
args = parser.parse_args()

def compile(target):
'''
compile graph
'''
net, params = relay.frontend.from_mxnet(block, {"data": dshape})
with relay.build_config(opt_level=3):
if args.metal:
graph, lib, params = relay.build(net, 'metal', target_host=target, params=params)
else:
graph, lib, params = relay.build(net, target, params=params)
return graph, lib, params

def run(graph, lib, params, ctx):
'''Build TVM runtime'''
m = graph_runtime.create(graph, lib, ctx)
tvm_input = tvm.nd.array(x.asnumpy(), ctx=ctx)
m.set_input('data', tvm_input)
m.set_input(**params)
# execute
m.run()
# get outputs
class_ids, scores, bounding_boxs = m.get_output(0), m.get_output(1), m.get_output(2)
return class_ids, scores, bounding_boxs

if args.metal:
# override metal compiler to compile to iphone
@tvm.register_func("tvm_callback_metal_compile")
def compile_metal(src):
'''compile metal code'''
return xcode.compile_metal(src, sdk=args.sdk)

RUN YOLOV3 ON CPU

model_name = "yolo3_mobilenet1.0_coco"
dshape = (1, 3, 512, 512)
dtype = "float32"
target_list = ctx_list()
im_fname = download_testdata('https://github.com/dmlc/web-data/blob/master/' +
'gluoncv/detection/street_small.jpg?raw=true',
'street_small.jpg', module='data')
x, img = data.transforms.presets.ssd.load_test(im_fname, short=512)
block = model_zoo.get_model(model_name, pretrained=True)

for target, ctx in target_list:
if target == "cuda":
print("GPU not supported yet, skip.")
continue
graph, lib, params = compile(target)
class_IDs, scores, bounding_boxs = run(graph, lib, params, ctx)
print(bounding_boxs.asnumpy()[0][0])

run YOLOV3 ON ios rpc

os.environ['TVM_IOS_CODESIGN'] = args.team_id
os.environ['TVM_IOS_RPC_ROOT'] = '/Users/mkrzus/github/tvm/apps/ios_rpc'
os.environ['TVM_IOS_RPC_PROXY_HOST'] = args.proxy_host
os.environ["TVM_IOS_RPC_DESTINATION"] = args.destination

if not re.match(r"^platform=.,id=.$", args.destination):
print("Bad format: {}".format(args.destination))
print("Example of expected string: platform=iOS,id=1234567890abcabcabcabc1234567890abcabcab")
sys.exit(1)

TARGET = "llvm -target=%s-apple-darwin" % args.arch

only the cpu version

graph, lib, params = compile(TARGET)

Save the library at local temporary directory.

tmp = util.tempdir()
lib_fname = tmp.relpath('net.dylib')
lib.export_library(lib_fname, xcode.create_dylib,
arch=args.arch, sdk=args.sdk)
xcode.codesign(lib_fname)

server = xcode.popen_test_rpc(args.proxy_host, args.proxy_port, args.key,
destination=args.destination,
libs=[lib_fname])
remote = rpc.connect(args.proxy_host, args.proxy_port, key=args.key)

if args.metal:
ctx = remote.metal(0)
else:
ctx = remote.cpu(0)
rlib = remote.load_module("net.dylib")

class_ids, scores, bounding_boxs = run(graph, rlib, params, ctx)

ax = utils.viz.plot_bbox(img, bounding_boxs.asnumpy()[0], scores.asnumpy()[0],
class_ids.asnumpy()[0], class_names=block.classes)
plt.show()
```

TL;DR:

ios rpc app deployment is throwing: Cannot find function tvm.contrib.sort.argsort in the imported modules or global registry for both yolov3 and ssd while standard macOS cpu is not.

[tqchen]

The solution is to add the inclusion of the file to https://github.com/dmlc/tvm/blob/master/apps/ios_rpc/tvmrpc/TVMRuntime.mm, a contribution is more than welcomed

[tqchen]

Note that for general troubleshooting and discussion, we recommend starting a thread in https://discuss.tvm.ai/ :) Please do so for future posts

[KayneWest]

Thanks @tqchen!

added the sort inclusion file to https://github.com/dmlc/tvm/blob/master/apps/ios_rpc/tvmrpc/TVMRuntime.mm

// Include Sort
#include "../../../src/contrib/sort/sort.cc"

[tqchen]

PR is more than welcomed!

[KayneWest]

@tqchen I was debating adding a PR but wasn’t sure considering that tvm.contrib.sort probably wasn’t added in the first place because it was out of scope for the ios_rpc example. That said, I’d be more than happy to create a PR if you think it would help. Additionally, do you think it would be beneficial if I submitted a PR for an IOS version of android_deploy with yolov3?

[tqchen]

If you get it to work, a PR is more than welcomed