Tvm: [RFC][Rust][Roadmap] Expand Rust capabilities

[nhynes]

The one thing that common to Rust programmers is that, if it's not written in Rust, it ~probably~ almost definitely should be. In the case of TVM, we can do this rather painlessly and incrementally since the "frontend" (i.e. _bindings_) made by @ehsanmok allow us to replicate the functionality of c++ TVM until the pure-Rust portion (currently a CPU-only static runtime) is feature complete.

The target for the next TVM release (0.6) will be (in order):

• [x] upgrade to Rust 2018 (#2524)
• [ ] unify bindings and runtime under a single tvm namespace
  
  
  
  • use single bindgen-generated ffi in common, remove tvm-sys
  
  
  • use common types for packed_func, module, (DL)Tensor
  
  
• [ ] make bindings optional using feature flag and conditional compilation
• [ ] use newer failure crate for errors
• [ ] Rust DeviceAPI (#2525)
• [ ] Relay graph runtime
• [ ] assess feasibility of porting compiler (incl. python bindings)

For those wondering "why even??," my main response would be that rust is more ergonomic than C++ (having powerful macros, a package manager, a sane standard library); safety is also nice to have but mostly for the runtime. In an ideal world, all of the boilerplate associated with creating new passes and operators is handled by a few proc_macros and some build.rs scripts--same with generating the python bindings (complete with docs; ref: https://github.com/dmlc/tvm/pull/2328#issuecomment-450001679).

[Ravenwater]

If TVM moves to Rust, I will certainly cease to be interested.

[nhynes]

If TVM moves to Rust, I will certainly cease to be interested.

That wasn't necessarily the proposal, nor do I have the authority to declare this so. However, I would say that proper use of C interop allows bindings in other languages. Having the core in Rust would improve maintainability, though. Also, the Rust standard library has been ported to more architectures including aarch, riscv, and wasm, which would help extend TVM device support.

Would you mind expanding on the reasoning behind your comment a bit? Are you a python proponent? If not, what about C++ does your use case require?

[ehsanmok]

@nhynes I like the idea very much and can see how an elegant rewrite would help the future directions of TVM. Definitely, leveraging the power of proc_macro/attr/derive can make adding new features seamless. Right now, I think (others have echoed the same) that the tight relation between C++ and Python is perhaps going too far and more likely it will become a major dev bottleneck when new features would be added. I envision that Rust can become a standalone component and _satisfy_ the research/prototype nature of Python as well as the backend deployment nature of C++ for both application oriented use cases and compiler developments.

Also one more +1 for mentioning failure crate and getting rid of the backtrace overhead of error_chain.

@Ravenwater

If TVM moves to Rust, I will certainly cease to be interested.

This proposal is not about moving TVM to Rust entirely for now (though it could happen later). Hopefully when we figure this out, the end result would be appealing enough that would catch your interests.

[Ravenwater]

@nhynes @ehsanmok the reason that Rust is unattractive in anything we do with TVM/VTA is that we are interfacing with systems that are 10M LoC of C++ that took hundreds of millions of dollars to create. These systems are not going to be Rust compatible for the rest of their economic life. Most of these lines are implementations of complex optimization algorithms where clever memory management techniques are the only way these algorithms produce any results.

I have nothing against Rust, but any Rust related dependencies in TVM will make our use of the TVM/VTA stack much less attractive as compared to TF et. al.

[nhynes]

not going to be Rust compatible

would C++ bindings not solve that? it sounds like your (completely reasonable) issue with Rust is just that it doesn't have a C++ ABI.

[Ravenwater]

@nhynes C++ bindings would at least make it compatible. However, the additional complexity of yet another language/compiler/build/package environment that needs to be supported among two dozen compiler and hardware environments is so very unattractive. We like to do science, not spent 95% of our time keeping the build going, :-(

If Rust would be an independent add-on, all would be fine. We could just exclude it from the builds and regressions and we would be on our way.

[yzhliu]

-1. Besides the pain that @Ravenwater mentioned, rapidly growing a community requires a stable core stack, which is essential to new contributors, and to experienced developers/researchers who keep building new features upon that. IMHO TVM is in its critical moment of growing, we'd better not introduce extra distraction which can have large impact on our contributors and users.

[nhynes]

we'd better not introduce extra distraction

I didn't intend the porting effort as a top-level project. It's more of a thing I was hoping to help with _anyway_. The idea was that a Rust impl would eventually get good enough that people would choose to use it over C++ or Python. If you think that, even as a side project, it would detract from contributions to the main project, then I will gladly table the idea. Ideally, we'd focus on tight interop between all languages such that it's possible to freely use whatever language that's convenient.

[FrozenGene]

What I only say is -1. TVM is not toy project and is not the place expressing the favor of one programming language and try to introduce it into project. TVM is growing rapidly and many companies are trying to use or have used it. We shouldn’t introduce any unnessary distraction for developers and make other companies not use it because of this.

[nhynes]

@FrozenGene understood. I will extract the high-level idea of your comment and say that the development of any Rust functionality will be independent of the main project. Although I jokingly suggested that the reason to rewrite in Rust is because of the language alone, doing so would have the benefits of easier extensibility/prototyping and bringing any re-implemented functionality to new platforms like riscv. For instance, imagine having an in-browser editor for TVM modules backed by a Wasm compiler enabled by Rust; or a F1 instance running a riscv soft core with VTA. As @Ravenwater mentioned, many of us (myself included) are trying to do science. Having more options _as long as it's not a distraction to the core project_ can only help that goal.

[tqchen]

Great discussions :) First of all, as many people said, I think the core of TVM will stay C++ in the foreseeable future.

On the other hand, the question is that, can we enable the use of rust(optionally), along with C++ core? I think the answer is yes. Let me elaborate that a bit. The TVM project is a layered system: we have runtime, frontend and compiler core. So we should really ask what does it mean to introduce rust in each setting.

Rust Runtime Backend

Runtime backend means implementing the C APIs in include/tvm/runtime/c_runtime_api.h. TVM runtime is the standalone thing that allows us to implement all the runtime features, in this case, the answer is definitely yes. The main reason is rust have good WASM support as @nhynes mentioned and probably better SGX support. Having a rust runtime gives us good options in these settings. Also because the runtime is minimum, having a rust alternative version helps us to build good support for WASM and SGX.

Thanks to @nhynes , we already have good rust runtime backend support, and I think it makes sense to think about supporting graph to some extent to get good WASM support(without libc).

Rust Frontend Support

Frontend means wrap C APIs in include/tvm/runtime/c_runtime_api.h to provide a rust abstraction. It allow us to deploy models using rust. I think it also makes sense, and @ehsanmok has been working on that.

Compiler Core and Node System

So the most interesting questions come when we want to think about Rust for compiler cores. In particular, we should ask where should the AST(i.e. TVM's Node class) live. Nodes are important not only because they are in C++, but also because we have clean API to expose them to python. While I have no doubt it is possible to re-implement the Node system in rust, it makes C++ obsolete, because you cannot directly visit these memory object directly as struct pointer access in C++, so it is not a realistic option.

A more realistic, but still a challenging question, is whether can we enable writing passes in rust, and plug them in as PackedFunc to the C++ core. I have some serious thoughts in that direction. The simplest way is to use frontend API, like python. However, that is only good for prototyping purposes, because we need to do C API calls just like python did.

A Common Node System

What we really want in our case, is to make rust access the C++ AST as backend language, so the Node field access can be done natively via struct address accessing, without getting into C API. Because some folks in the community expressed interest in Rust, I gave some serious thought into this.

I think it can done, but also technically challenging. The general idea is to make use of C ABI. That is, we build most of the Node system as C objects, which then allows us to expose them directly to Rust. The primary AST and core are still in C++, but the rust language module can directly access these in-memory AST fields via pointer de-referencing. The key challenges here, however, is how can we solve the problem of virtual functions tables, since they are part of C++ ABI and not visible in Rust. For the base Node system, that is partially achievable by defining our own version of vtable struct. In that way, we might be able to expose some of the Nodes partially to rust. I have not tried it out, but it could be an interesting step as long as it does not change the current c++ code base.

Summary of My Take

• We should support rust runtime/frontend, and get most of the benefit @nhynes mentioned(riscv, wasm).
• We should not rewrite codebase in rust, but we could try to explore possible directions to enable rust module plugin alongside with the current c++ core.

Most of the current @nhynes 's proposal makes sense since they fall into support rust runtime, perhaps we should change the title to Enable rust support instead of rewriting :)

[ajtulloch]

I think @tqchen's summary makes sense - I think there's definitely space for alternative _runtime_ backends/implementations. @nhynes if you're interested, one thing that I think Rust could do that would make the Rust backend compelling would be enabling something like a no_std/embedded-style impl for code-size constrained use-cases, that would target something like O(10kb) or so of code for the entire runtime vs the current O(100kb) (I think this is doable, curious what you think?)

[ehsanmok]

I think now, this is more qualified as a _pre-RFC_ to be more specific IMHO. Some features included (2018 edition, failure crate namely) are more like enhancement rather than new features/functionalities and we should separate them from this proposal into issues. I'll be very useful if TVM can adopt or get idea from RFC mechanisms/template from other big projects such as Rust itself.

I should say that no_std is appealing to me and it's in my todo list to go through, however, based on my understanding it's very likely that it will lead to yet undiscovered area in embedded Rust, but it'd be great as a research at least. It'd also be very helpful if we could consult with Rust embedded group first!

[nhynes]

@ajtulloch a no-std runtime is a nice idea! It's actually possible to use the current implementation with a custom sysroot built using xargo that only pulls in core and exposes it as std. The size of the final binary depends on DCE and LTO. I'm not sure what that is, but I'm definitely curious; I'll post back with findings. I guess this is a good opportunity to see how well the RISC-V toolchain works, too :)

[tqchen]

close as many discussed feature landed. let us open new threads for incoming goals