Pyo3: Provide comparison between PyO3 and rust-cpython

[ssokolow]

When I come to the PyO3 frontpage, having never seen PyO3 before, my first questions are "how is this different from rust-cpython?" and "Why would I want to use this instead?"

So far, all I've been able to determine is:

1. Judging by #5 and my experience using rust-cpython on stable, this adds a dependency on nightly Rust.
2. This uses attributes rather than macros
3. I don't remember the list of ToPyObject impls being this long in rust-cpython.

It'd be a good idea to add a blurb to the README explaining:

1. What motivated the fork (ie. how PyO3's goals differ from rust-cpython's)
2. What PyO3 does better
3. What PyO3 does worse (ie. successfully compiling on stable)

[fafhrd91]

at this point, base concepts, compared to rust-cpython, are totally different.

1. All objects are owned by pyo3 library and all apis available with references. in rust-cpython, you own
   python objects. here is example of PyList api:

rust-cpython:

impl PyList {

   fn new(py: Python) -> PyList {...}

   fn get_item(&self, py: Python, index: isize) -> PyObject {...}
}

pyo3:

impl PyList {

   fn new(py: Python) -> &PyList {...}

   fn get_item(&self, index: isize) -> &PyObject {...}
}

Because pyo3 allows only references to python object, all refs uses Gil lifetime. So Python object
is not required, and it is safe to have fn py<'p>(&'p self) -> Python<'p> {} function.

[fafhrd91]

rust-cpython custom class

1. rust-cpython uses whole new language based on macros
2. it is very hard to extend py_class! macros, for example async/await support.
3. generated functions are used for access to struct attributes.
4. To drop PyObject GIL is required.

pyo3 custom class

1. use proc_macro and specialization for class implementation (nightly is required)
2. pyo3 does not modify rust struct. it is possible to define needed traits and make rust type compatible with python class without using #[py::class] macro.
3. class customization is based on specialization and traits and associated types. separate trait is defined for each type
   of customization (i.e. PyMappingProtocol, PyNumberProtocol). macro is used for simplicity, but it is possible to make rust type compatible with specific customization without using proc macros.
4. pyo3 does not inc ref for borrowed ptrs, PyObject does not need GIL to drop.
5. pyo3 at least 15% faster.

[fafhrd91]

motivation

1. seems rust-cpython is not maintained anymore. I tried to communicate with author, didn't get any response. I submitted multiple PR, none of them had any reaction.
2. I don't like class implementation based on macros. first of all, this is new language within rust, second it is very hard to extend and modify.
3. pyo3 is just rust. proc macros are used for automatically implement extra traits and wrapping code, so it possible to do the same manually.

downsides:

1. pyo3 depends on nightly rust (proc_macro, specialization, const_fn).
2. proc_macro error reporting is bad.

[ssokolow]

In that case, definitely something I'll be switching over to once it's possible to resolve #5.

[fafhrd91]

I used rust-cpython, and had occasional segfaults. never got any response from author.
I use pyo3 in production without this segfault.

[ssokolow]

In that case, once I've finished migrating as much of my project to the Rust side as possible, if I encounter segfaults and can't resolve them, maybe I'll drop to something lower-level like Snaek.

...or just port everything back to pure Python. Avoiding dependency on nightly Rust is non-negotiable.

[fafhrd91]

I hope proc_macro will be stabilized by the end of year.

[fafhrd91]

Just landed pretty big change.

rust-cpython requires Python parameter for PyErr, so error handling ergonomics is pretty bad.
It is not possible to use ? with rust errors.

pyo3 on other hand does not require Python for PyErr creation, it is required only if you want to set exception to python interpreter with PyErr::restore() method. so it is possible to use std::convert::From<Err> for PyErr trait and ? is supported automatically.

for example:

use pyo3::*;

fn parse_int(s: String) -> PyResult<usize> {
    Ok(s.parse::<usize>()?)
}

The code snippet above will raise ValueError in Python if String::parse() return an error.

[fafhrd91]

closing, there is link in readme

[MajidHeydari]

Does rust_cpytho support all Python packages like socket?

[ssokolow]

@MajidHeydari I don't see why it wouldn't.

There are two major reasons something might not support a Python package:

1. Packages might assume some aspect of CPython that isn't replicated in another implementation. (eg. Tripping over an area where PyPy hasn't perfectly replicated CPython's behaviour yet, or assuming that IronPython or Jython will support the same C FFI as CPython.)
2. The package might conflict with something else in the process's address space. (eg. The X-Chat IRC client is GTK+-based and has a Python plugin interface. Trying to use PyGTK in an X-Chat plugin will crash X-Chat.)

The former isn't a problem because rust-cpython embeds or produces compiled extensions for the actual CPython.

As for the latter, rust-cpython won't cause incompatibilities, but it's still up to you to avoid using Python extensions and Rust crates which depend on incompatible C libraries at the same time. (eg. Trying to use both the Python and Rust bindings for the same C library in the same process, rather than using just one and using rust-cpython to share it across the language barrier.)

[MajidHeydari]

can rust_cpytho build web framework ?

[ssokolow]

To what end?

Combining Python with another language isn't going to magically make it faster because the things holding back Python's speed are inherent and changing them would break compatibility with existing Python packages.

You can use rust-cpython to write compiled modules for a Python web framework to load, but things like Python's Global Interpreter Lock and highly dynamic nature aren't going to perform well if you try to write a web framework by embedding a Python runtime into Rust.

(And, if you've got CPU-bound tasks in your web app, you're supposed to move them out into something like Celery anyway so they don't block the part of the app handling the web requests.)

[MajidHeydari]

Does webassembly not make the server faster?

[ssokolow]

WebAssembly isn't magic sauce. You need to use it to replace the bits that are slow... and if all the low-hanging fruit has been picked and the remaining slowness is spread throughout the language because of things like extremely dynamic dispatch, then you can't really "replace" it without rewriting everything in another language.