Pyo3: Compile with stable rust

[fafhrd91]

PyO3 requires following unstable

• [x] const_fn rust-lang/rust#24111
• [x] specialization rust-lang/rust#31844
• [x] use_extern_macros rust-lang/rust#35896
• [x] proc_macro (proc_macro_attribute) rust-lang/rust#35896
• [x] associated_consts
• [x] Optional, for wrap_function!: concat_idents (rust-lang/rust#29599) Solved with mashup

[tailhook]

Doesn't proc macro work on stable rust? (or just partially) And there is proc-macro2

[messense]

I think currently only proc macro for custom derive works on stable Rust, proc_macro_attribute don't.

[fafhrd91]

proc_macro_attribute is unstable, info is in linked tracking issue.

[tailhook]

Can this trick be used instead of proc_macro_attribute ?

[fafhrd91]

no, it is not. pyo3 strictly requires proc_macro_attribute, because it requires access to token stream of impl sections.

[tailhook]

no, it is not. pyo3 strictly requires proc_macro_attribute, because it requires access to token stream of impl sections.

I think this is exactly what it does, i.e. given the example:

#[macro_use] extern crate procedural_masquerade;
extern crate proc_macro;

define_proc_macros! {
    #[allow(non_snake_case)]
    pub fn foo_internal__stringify_const(input: &str) -> String {
        format!("const STRINGIFIED: &'static str = {:?};", input)
    }
}

The foo_internal__stringify_const function receives original code and returns generated code. You get a token stream using syn crate as with macros 1.1.

And all this works by wrapping a body of the macro into a type definition to leverage macros 1.1, to get a token stream.

The only difference is that:

#[mod2init]
fn init_function() {
}

Would be written as:

mod2init! {
  fn init_function() {
  }
}

[fafhrd91]

I see. But that would be quite large task, I don’t think I can spend any time on it. Also there is specialization, I am not sure if it possible to avoid it

If you want to work on this task I can give you permission to repo

[fafhrd91]

this features are stable now, const_unsafe_cell_new, const_size_of, const_ptr_null, const_ptr_null_mut

[termoshtt]

Stabilization of TryFrom has been reverted https://github.com/rust-lang/rust/pull/50121

[fafhrd91]

pyo3 does not use TryFrom

[termoshtt]

https://github.com/PyO3/pyo3/blob/9dc76610b92c3f52f73ec4d26a9e9154407e6345/src/objects/mod.rs#L183
The macro pyobject_extract! uses TryFrom. To use PyO3 in stable rust, this library itself must be compiled with stable rustc.
IMHO, there is no merit to use this unstable feature, and should be dropped.

[konstin]

@termoshtt Good catch! pyo3 implements TryFrom, even though it currently uses PyTryFrom instead. I put a feature gate onto the implementation of TryFrom and removed the try_from feature. Once TryFrom becomes stable, we can than replace PyTryFrom.

[termoshtt]

Drop of fn_must_use in afcc87e82c44432cbc4f7944711bf8d35c956d08 causes many warnings

warning: `#[must_use]` on methods is experimental (see issue #43302)
   --> src/typeob.rs:154:5
    |
154 |     #[must_use]
    |     ^^^^^^^^^^^
    |
    = help: add #![feature(fn_must_use)] to the crate attributes to enable

[konstin]

Unfortunately not, specialization is still not stable

[pohutukawa]

I'm trying to compile a Python manylinux wheel using PyO3. I know that it requires a nightly build, but isn't that (by now) a thing of history? It states the requirement of 1.34.0-nightly, however Rust stable is now at 1.35.0, but I'm still seeing the build fail (see below), though in my perception 1.35.0 (stable) is >= 1.34.0-nightly.

Any insights?

Error: pyo3 requires a nightly or dev version of Rust.
Installed version is: 1.35.0 (2019-05-20). Minimum required: 1.34.0-nightly (2019-02-06).
thread 'main' panicked at 'Aborting compilation due to incompatible compiler.', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.6.0/build.rs:540:17

[bgw]

@pohutukawa

my perception 1.35.0 (stable) is >= 1.34.0-nightly.

Nope. Nightly builds have additional feature flags turned on, most of which won't make it to the next stable release, so 1.34.0-nightly has a lot more features than 1.35.0 (including specialization).

[ikamensh]

I find the message confusing due to the second line:

Error: pyo3 requires a nightly or dev version of Rust.
Installed version is: 1.35.0 (2019-05-20). Minimum required: 1.34.0-nightly (2019-02-06).

I wanted to create an issue about this, but luckily found this. Also, how about telling Python users who are newbies to rust how to get the nightly version right in the error message?

I did it using
rustup toolchain install nightly
rustup default nightly

[joelberkeley]

Is specialization required for both using Rust in Python _and_ Python in Rust? i.e. can I do either of these on stable?

[AndreaCatania]

The path to stabilize the specialization feature still seems long, the tracking issue was opened on February 2016 and still there are a lot of opened questions there.

In the meanwhile that we wait for this feature, would be cool try to not use the specialization at all and so allow anyone to compile with the stable compiler, even if this mean change some APIs.

Is really so necessary this feature?

[io12]

https://github.com/rust-lang/rust/issues/31844#issuecomment-557918823

[nickray]

@konstin any thoughts on whether dtolnay's trick https://github.com/dtolnay/case-studies/blob/master/autoref-specialization/README.md linked to by @io12 above would be a feasible/pragmatic way to for this library to compile on stable?

[TheButlah]

Having to use not only a new programming language, but the unstable version of that language, is not something I feel comfortable pitching to my coworkers (no matter how stable nightly actually may be). Because easy python interoperability is a necessity for any new language that my work would use, and because it seems like there aren't other options with the same level of python interoperability with Rust that Py03 provides, I cannot in good faith pitch using rust at work until this is using the stable version of rust 😢

[jcdyer]

@TheButlah: You can create libraries to interoperate with python without using PyO3. Don't confuse the library with the language. You can currently either create native library bindings for your rust code, and interface with them using cffi, or use https://github.com/dgrunwald/rust-cpython. PyO3 provides a nicer interface, it's true, but if your requirement is stable rust, that interface isn't available to you.

[davidclevenger]

Bump: Is there any ETA / update on stable rust toolchain?

[davidhewitt]

Progress slowly continues; we want to get there but need more volunteer time to get this over the finish line yet. (pyo3 maintainer time in the last few months has been spent mostly on soundness problems as well as bugfixes, which are also very important.)

Last items to resolve are in #697. I can offer mentoring advice to anyone interested in helping out with pieces of the solution in there.

[TheButlah]

@jcdyer The emphasis is on easy interoperability, not just any interoperability, which is the purpose of PyO3. That being said, I did pitch rust to my coworkers, and got enough leeway to experiment with it in a side project - because I realized that nightly wasn't as bad as it sounded.

That being said, I work in a research lab not an enterprise company and our risk tolerance for these sorts of things is significantly higher, so continuing to focus on using stable would be a big selling point for the language overall. Glad to see how active this community is - it says a lot about the language and ecosystem :)

[kngwyu]

we want to get there but need more volunteer time to get this over the finish line yet

Yeah, I also think it's a matter of time.
Now I have a serious NeurIPS deadline. Please wait for a bit.

[nacaclanga]

Hello. It appears to me, from reading #210, that specialisisation is mostly used to implement class declaration. So how hard would it be to introducing a feature no-unstable-class-support to pyo3 that simply switches off class declaration, but allows code to be compiled on stable rust? People that want to use pyo3 instead of rust-cpython and can live without classes could then do so without having to use unstable.

Once specialisation is stable or pyo3 managed to get rid of it, class declaration would be allowed in no-unstable-class-support and the feature would become identical to the default configuration again. I think this would also be very much in line on how rustc itself adds new features.

An alternative would be the other way around: Make no-unstable-class-support the default and introduce a feature allow-unstable-class-support.

[programmerjake]

if we're going to use a cargo feature, we should use the enable unstable code feature rather than the disable version, since cargo features should be additive since that's how cargo's designed.

as an example, if one crate uses the unstable code and another one doesn't, then using the enable version of the feature would correctly deduce that unstable code should be enabled for the crate that requires it. if it was the disable version then cargo would incorrectly disable the unstable code even though a crate requires it, leading to compile failure.

If anything requires unstable code, then you need to use the unstable compiler on all the crates anyway in order to have the compiler versions match, so that's not a limitation.

[nacaclanga]

Make sense. The reason I thought it up the other way round is, that it might make things appear from the documentation, but one should be able to work around this.

[sinetek]

Hi. I don't know a lot about Rust, but trying to run an application I use often that recently introduced bits of rust.
Why can't PyO3 work with Rust 1.43.1, released last week? My OS ships stable rust. This ticket is 3 years old now, this is annoying.

[davidhewitt]

Read comments above, work is ongoing to fix this but it takes time...

[jcdyer]

@sinetek The answer to the question you asked is in the thread above.

If you would like to get it working, the recommended way to work with rust as a developer is to use rustup to manage your rust toolchains, so you can have nightly and stable running side by side. You can get rustup from https://rustup.rs.

Then run the following:

rustup toolchain install nightly
rustup default nightly

Then when you want do run on stable, you can switch back with rustup default stable. You can also set per-directory overrides with rustup override. Run rustup override --help for details.

You can find more detail here: https://github.com/rust-lang/rustup#working-with-nightly-rust

[davidhewitt]

For all those following this thread... the current PyO3 master branch just had a PR merged which added support for stable Rust.

Extra special thanks to @kngwyu and @konstin who have been working to remove specialization from PyO3 for a long time - and as of this morning, we finally got there! 🎉 🚀 🐍 🦀

[ikamensh]

Cheers!

On Sun, Jun 21, 2020, 10:06 David Hewitt [e-mail] wrote:

For all those following this thread... the current PyO3 master branch just
had a PR merged which added support for stable Rust.

Extra special thanks to @kngwyu https://github.com/kngwyu and @konstin
https://github.com/konstin who have been working to remove
specialization from PyO3 for a long time - and as of this morning, we
finally got there! 🎉 🚀 🐍 🦀

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/PyO3/pyo3/issues/5#issuecomment-647094806, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AFPQGZCMZPNWNYAXGESIS3DRXW5RBANCNFSM4DMCSS4Q
.

[gilescope]

That's awesome progress you've been making this past 6 months! Huge congratulations on all your efforts.

[TheButlah]

excitement intensifies

[fafhrd91]

congrats!