Flatbuffers: [rust] Viability of using c++ verifier via ffi

[jean-airoldie]

This is an issue with the goal of answering the question of whether it is worth
the effort to reuse the c++ verifier logic from a c++ffi in rust or if we should
instead focus on a native rust implementation. This is related to #4916.

This is based on a email thread with @rw.

Motivation

Being able use a verifier in rust is crucial since get_root currently panics when the provided buffer is invalid. Using the c++ verifier would fill this gap, at least until a native rust implementation.

Inital thoughts

Creating and using cffi bindings in rust is fairly common and simple in rust. The general concept is have the c source code as a git submodule, link and compile it in rust from a build.rs script, automatically generate bindings using [rust-bindgen] and expose these bindings in a crate. Then you wrap these bindings in rust code to offer a proprer interface.

If the c++ code offers a c-style style interface, then its no different from
c code. If flatbuffers offered a c-style interface to the Verifier logic,
here is how I think it would work. We would need to generate the c++ code
along with the rust code using flatc. Then we would have to programmatically
link, bind and generate bindings for it. Then we would have to generate
rust code that would wrap these bindings so that users don't have to use
a c-style API.

However, as far as I can tell, the c++ verifier logic seems to be a proper
c++ interface, which is a whole different beast. For starters I can't find
a single good rust example of a c++ffi interface. Moreover, it seems like the
Verifier relies on template specifialization, which is currently
[not supported] by bindgen. Assuming it is indeed the case, we would have to
programmatically generate c++ffi bindings which sounds like a nightmare.

So, from what I understand, the viability of this approach depends on whether
or not flatbuffers offers a c-style verifier API, as opposed to a c++ one.

[rw]

@aardappel for comments

[aardappel]

On the one hand it makes sense, in that you can expose the whole verifier in a single C function call.

On the other hand, I personally wouldn't want to have to deal with this much more complex a build system, having to generate code for 2 languages etc, just to save not having to write a verifier in Rust. I certainly think the long term most ergonomic solution is to have a Rust verifier. Note that in C++, though ostensibly the verifier just returns bool, it can additionally in a debug build assert right where the bad data happens, so it can be used to debug bad data in more detail. It be clumsy to expose this through a C interface, since having to actually collect the context of a stack trace manually would really slow the verifier down. If it was in native Rust however, native Rust debugging tools could be used similarly.

That said, if you did decide to pull in a C++ verifier, you could make the deal sweeter by also pulling in other C++ features the Rust implementation lacks, like JSON -> FlatBuffer and FlatBuffer -> JSON converters.

[jean-airoldie]

On the other hand, I personally wouldn't want to have to deal with this much more complex a build system, having to generate code for 2 languages etc, just to save not having to write a verifier in Rust.

Yes, but that might be achievable if the rust crate ships with the rustc executable (or we build it from src), since we could automate the build process entirely. I agree that this is a complex problem however.

Note that in C++, though ostensibly the verifier just returns bool, it can additionally in a debug build assert right where the bad data happens, so it can be used to debug bad data in more detail.

Couldn't we just build & link the library in debug mode depending on the rust profile? Or maybe a separate verifier API?

That said, if you did decide to pull in a C++ verifier, you could make the deal sweeter by also pulling in other C++ features the Rust implementation lacks, like JSON -> FlatBuffer and FlatBuffer -> JSON converters.

Totally on board with that.

I also feel like if we decide to pull from the C++ verifier, it also opens the door for other languages to do so since cffi are quite common (I'm thinking go, python, etc). Using the c++ implementation through a cffi might be more per formant than a native implementation, albeit less portable.

[jcrevier]

I'm currently using the C++ verifier in Rust over ffi. It wasn't really too much hassle to get it up and working over ffi by just wrapping the C++ with C, and adding a few things to the build.rs to first build a static library that then is linked into my rust library.

Given this was fairly straightforward, my initial thought is that I'm not sure how much value added there is in complicating the Rust flatbuffers project in order to expose C++ bindings in the way @jean-airoldie described. Right now the model of how to use flatbuffers in Rust is fairly simple, you have code that's generated, and you also depend on the flat-buffers crate for common functionality. If I understand correctly, with this proposal you'd need to generate...an entire crate that will be compiled by rustc for the verifier (C++ exposed over ffi + rust bindings for it)? This certainly seems more complicated to me, and I can also see advantages to not having the project rely on something like rust-bindgen.

Sorry, if I'm jumping in the middle of the conversation only to provide seemingly negative comments to this approach -- not my intent and I hope I don't come off as uncharitable here. I see value though in making the verifier easier to use for Rust users, as I think the project would get more use by the Rust community if the verifier was accessible, and I'm not sure if this approach would accomplish that well.

I'd be open to committing some time to work on a native Rust verifier if that's a direction people are still may be interested in...though I can't say with certainty how far I'd get. I'd have to familiarize myself with both code-bases.

[rw]

1) Maybe we should have a tutorial for how to use C++ verifiers over FFI in Rust. We could demonstrate two approaches, by showing usage with both rust_bindgen and the newer cxx project.

2) I like the idea of pure-Rust verification code. My two main reasons are that (a) this would help bring Rust up to parity with C++ (a long-term goal I have), and (b) 100% Rust code would be helpful to people doing tricky things like cross-compilation.

@jean-airoldie @jcrevier

[jean-airoldie]

This certainly seems more complicated to me, and I can also see advantages to not having the project rely on something like rust-bindgen.

I agree that directly using the generated c++ code via a cffi is way easier, but its not really user friendly, which was my concern. I feel like its more of a short-term solution until a pure-rust verifier is implemented.

[aardappel]

Agree that in the long run people will be much happier having the verifier natively. When I originally designed it, I underestimated how often people would need it. Turns out, even in trusted environments, being able to turn it on for debugging/testing is super nice.

But until then, a tutorial on how to integrate with C++ would be cool. There's other things I could imagine a Rust user may want to grab from C++, such as JSON parsing/generation.

[github-actions[bot]]

This issue is stale because it has been open 6 months with no activity. Please comment or this will be closed in 14 days.

[rw]

Worth keeping this open.