Neon: Quest: N-API Support

[dherman]

Prepare, once and future Neon contributors, for our noblest quest yet!

We are going to port Neon to Node's new N-API!

I'll explain. N-API brings to Neon the promise of a stable, backwards-compatible ABI—binary compatibility across all future versions of Node.

This is a big deal.

Portability across Node versions means Neon will finally be practical for publishing libraries, not just apps: a few prebuilt binaries should be sufficient for all downstream customers to use your native library without ever knowing the difference.

The stuff of legend, no?

Our Quest

Step 1. Create the feature flag

• [x] Create a cargo feature flag to allow us to concurrently maintain the main Neon codebase along with the experimental N-API support in the same master branch. (Merged!)
• [x] Set up a test suite specifically for the N-API backend so each task can easily include adding tests (#449)

Step 2. Implement the port

• [x] Module contexts and initialization: Implement the neon::context::ModuleContext type and pass it to the module initialization function inside the register_module! macro defined in /src/lib.rs. The context struct will likely need to encapsulate the underlying napi_env and napi_value as private fields. This can be implemented before we implement functions, with an unimplemented export_function() method for now.
• [x] Functions: This is probably one of the subtler tasks. See the implementation of neon::types::JsFunction::new(). The Rust callback can be stored as the extra void* data passed to napi_create_function.
• [x] Function arguments: Implement CallContext::len() and CallContext::argument().
• [x] Function returns: Implement function return values.
• [x] this: Implement CallContext::this().
• [x] Call kinds: Implement CallContext::kind().
• [x] Function exports: Once we have module contexts and functions implemented, we can implement the ModuleContext::export_function() shorthand method.
• [x] Objects: See neon::types::JsObject::new() and the neon::object::Object methods.
• [x] Arrays: See neon::types::JsArray.
• [x] ArrayBuffers and Buffers: See neon::types::binary and the N-API functions for working with binary data, such as napi_create_arraybuffer, napi_create_buffer, etc.
• [x] Uninitialized and null: These should be pretty bite-sized. See neon::types::JsUndefined and neon::types::JsNull. @goto-bus-stop
• [x] Booleans: See neon::types::JsBoolean. @goto-bus-stop
• [x] Numbers: See neon::types::JsNumber.
• [x] Strings: See neon::types::JsString. We'll need to explore what binary string representations can be used between the NAN vs N-API runtimes for constructing JS strings.
• [x] ~Classes: This will require us to figure out how to do unique branding with N-API, but I believe napi_define_class supports this. (Here is one pure C example we can look to for inspiration.)~ <== not needed for functional completeness; see #596
• [x] Errors: See neon::types::error. We'll need to explore how N-API does throwing and catching errors. - @anshulrgoyal 🔒
• [x] Conversions: See the uses of neon_runtime::convert::* and the napi_coerce_* functions.
• [x] Scopes: Luckily, the N-API HandleScope mechanim matches V8's mechanism very closely. See neon::context and the uses of various HandleScope internal types.
• [x] Tag checks: See uses of neon_runtime::tag::*.
• [x] ~Task scheduling: See neon::task and neon::context::TaskContext, and the N-API "simply asynchronous operations" API, which uses the same underlying libuv thread pool as Neon's current backend, but with N-API's stable ABI.~ <== not needed for functional completeness; see #596
• [x] ~Thread-safe callbacks: This can be implemented for N-API once we've merged an implementation for RFC 25, using napi_make_callback.~ <== not needed for functional completeness; see #596
• [x] Windows Support: Windows requires linking against node.lib and win_delay_load_hook. Create a custom build script to link these on windows.

We have just a couple remaining items to finish up:

• [x] Equality comparison of handles - see #666
• [x] JsBuffer::uninitialized - see #664

Step 3. Deprecate the legacy runtime

Once we finish the complete port, we can switch the default feature flags to use the new runtime and publish a new 0.x minor version. Eventually after a few releases we can remove the old runtime completely.

How to Contribute

Building N-API-based projects

To experiment with the N-API runtime or do manual testing, you can create a Neon project that uses the right feature flags. To try it out, you can run:

neon new --no-default-features --features=napi-latest --neon=path/to/neon my-project

where path/to/neon is the path on your local filesystem to a local clone of the Neon repo.

Manual Steps

The output of neon new executed above will produce a project that fails to build. When using the neon backend, either neon-build should be used with a simple cargo build or neon-cli should be used and neon-build should be removed. If both are used, the project will fail to build.

There is an RFC (https://github.com/neon-bindings/rfcs/pull/36) to replace neon new which will correctly generate a project. The simplest change is to edit native/Cargo.toml:

• Remove the neon-build dependency
• Remove build = "build.rs"
• delete native/build.rs

Note: If you create a Neon project nested inside the directory tree of a clone of the Neon repo, you'll need to add the line

[workspace]

to your Neon project's native/Cargo.toml manifest in order to build the project.

Adding an N-API primitive

To add an N-API primitive, you should implement it in pure Rust (using unsafe as necessary, but only as necessary!) in crates/neon-runtime/napi, and call out to the N-API backend exposed through nodejs-sys.

When the Neon runtime needs to pass around a data structure, you can make two different definitions of the type, separated by testing the feature flag with #[cfg(feature = "...")]. You may sometimes need to refactor the types in the Neon runtime to accommodate differences between the legacy and N-API runtimes.

Adding a test

The test/napi directory is the space for adding N-API acceptance tests. You can add native Rust logic to test/napi/native/src and JS logic to test/napi/lib. You can get examples of existing acceptance tests in our existing backend in test/dynamic, which has the same structure.

Will You Join Us?

As you can see, the quest ahead of us will be no small feat!

Indeed, but fear not: we're here to help you if you get stuck. And many of these tasks can be a great way to get started with contributing to Neon and even learning Rust.

Claim one of the tasks today by leaving a comment below or pinging @dherman or @kjvalencik on Slack!

[ciscojeremy]

I'm interested.

[dherman]

@ciscojeremy Awesome! Are you on the community Slack? @kjvalencik is just now working on some refactoring so you might want to sync up with a chat.

[ciscojeremy]

Nope, I'm not a the community slack. How do I join?

On Thu, Oct 24, 2019, 7:53 PM Dave Herman [e-mail] wrote:

@ciscojeremy https://github.com/ciscojeremy Awesome! Are you on the
community Slack? @kjvalencik https://github.com/kjvalencik is just now
working on some refactoring so you might want to sync up with a chat.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/neon-bindings/neon/issues/444?email_source=notifications&email_token=AJBIX4KNNZBGQNQP7MGIVWDQQHVMBA5CNFSM4IZWOEP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECGCCLQ#issuecomment-546054446,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJBIX4P2FBURIZPHWWAN5JLQQHVMBANCNFSM4IZWOEPQ
.

[kjvalencik]

The Rust Bindings community Slack is open to all; use the Slackin app to receive an invitation.

[Brooooooklyn]

Maybe this repo could save your time. https://github.com/Brooooooklyn/napi-rs

[kjvalencik]

:wave: I'm working on ModuleContextand initialization. This work also covers some of scopes.

https://github.com/neon-bindings/neon/pull/458

[jrop]

This is the best dev-related news today! (came from Twitter)

[dio]

Wow. I’m so excited! I hope I can contribute something! _joining the community slack_

[qwelias]

I'd like to learn rust, but I have no experience yet, would like to contribute and have lots of experience with node tho. So if you think a total rust noob can be trusted with a certain task -- hit me :)

[kjvalencik]

@qwelias definitely! There will be some beginner friendly Rust tasks and likely some pure javascript as well. And, of course, testing and vetting of ideas are always needed!

[goto-bus-stop]

I'm looking at undefined/null for a start! Possibly bools too if they are as similar as I suspect, so all the global singleton values could be done in one go.

[timsuchanek]

What are some of the next possible steps here?

[goto-bus-stop]

i was planning on working on either objects or functions this Friday

[goto-bus-stop]

ah i thought numbers got in together with bools but i had them in a separate branch: https://github.com/neon-bindings/neon/pull/492

started working on objects and as i was writing a test case, i wanted to use tag checks like val.is_a::<JsNumber>() as well, so that's where i'm looking. the tag checks need lots of boring mechanical changes to thread the Context value through a dozen layers :upside_down_face:

[timsuchanek]

Nice! So it sounds like a first prototype-ish version would already be usable quite soon?

[goto-bus-stop]

I also started working on function calls (from JS into Rust for now) so parts of the existing test suite can be run on n-api. In-progress branch here: https://github.com/goto-bus-stop/neon/tree/functions

@timsuchanek once both objects and function calls are in, some _very limited_ use cases would be possible. I expect many real-world cases would want the Class stuff from Neon so you can pass Rust structs between JS and Rust, so they can live for longer than a single call. that would probably come next.

[goto-bus-stop]

Functions, function arguments, and function return values can be checked off now that #507 is merged :tada:

I would like to look at ArrayBuffers support next, since #507 already ported a few of the buffer tests that are currently being skipped.

[goto-bus-stop]

Looking into getting the N-API runtime going on Windows today!

[dherman]

We're almost there! I've struck out the items that are not needed for functional completeness and noted them in the new #596 issue for the deprecation plan. All that's left is to review the implementation of ComputeContext and ExecuteContext to make sure they properly create nested scopes, and we'll be able to close this issue!

[wspl]

Would neon n-api support Electron without electron-rebuild or electron-build-env?

[kjvalencik]

@wspl Correct, they would not be needed. There would be a minimum n-api version that we might need to provide to neon with alternate means if we use APIs that aren't in the Electron node version yet, but we could probably do that with feature flags.

[dherman]

With the merging of #664 and #666, the last two items on the list are done, and the quest is officially complete! Congratulations and a _huge_ thank you to everyone who contributed to making it happen!