Pyo3: Pickle Support

[SerialDev]

As of right now its not possible to pickle classes created by PyO3.
This feature would be invaluable for situations where some form of persistence would be desireable.

As of right now it has trouble pickling after I call

    #[new]
    fn __new__(obj: &PyRawObject) -> PyResult<()>{

Otherwise the .__dict__ attributes are maintained prior to initialization with __new__

[rth]

Would this mean implementing __getstate__ and __setstate__ methods (cf https://docs.python.org/3/library/pickle.html#pickling-class-instances)?

For instance, the way pickling works for,

• Boost.Python C++ library
• Cython extensions

might provide some examples.

[rth]

For instance, if we take the documentation example for MyClass,

# use pyo3::prelude::*;
# use pyo3::PyRawObject;
#[pyclass]
struct MyClass {
   num: i32,
}

#[pymethods]
impl MyClass {

     #[new]
     fn new(obj: &PyRawObject, num: i32) {
         obj.init({
             MyClass {
                 num,
             }
         });
     }
}

by default we get the following error when pickling this class,

        obj = MyClass()

>       pickle.dumps(obj)
E       TypeError: can't pickle MyClass objects

If we now add the __getstate__/ __setstate__ methods,

    fn __getstate__(&self) -> PyResult<(i32)> {
        Ok(self.num)
    }

    fn __setstate__(&mut self, state: i32) -> PyResult<()> {
        self.num = state;
        Ok(())
    }

we get another exception,

_pickle.PicklingError: Can't pickle <class 'MyClass'>: attribute lookup MyClass on builtins failed

There is some additional step I must be missing here.

[althonos]

@rth : this may be related to the fact that PyO3 exposes all classes as part of the builtins module, because the import mechanism has not been properly implemented, so pickle tries to use builtins.MyClass and fails with the error you reported.

[rth]

Thanks @althonos ! Opened a separate issue about it in #474

[rth]

So by subclassing , to set the __module__ correctly as suggested in https://github.com/PyO3/pyo3/issues/474#issuecomment-489521285, pickling seems to work.

Though, I get a segfault occasionally (i.e. it does seem to be random) at exit. For instance when running a pytest session where one test checks pickling,

gdb --args python3.7 -m pytest -k test_pickle
GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
[...]
Reading symbols from /opt/_internal/cpython-3.7.1/bin/python3.7...(no debugging symbols found)...done.
(gdb) run
Starting program: /opt/_internal/cpython-3.7.1/bin/python3.7 -m pytest -k test_pickle
warning: Error disabling address space randomization: Operation not permitted
============================================================= test session starts =============================================================
platform linux -- Python 3.7.1, pytest-4.4.1, py-1.8.0, pluggy-0.9.0 -- /opt/_internal/cpython-3.7.1/bin/python3.7
cachedir: .pytest_cache
rootdir: /src/python
collected 1 items  / 1 selected                                                                                               

my_module/test_pickle.py::test_pickle PASSED

=================================================== 1 passed  in 0.13 seconds ===================================================
During startup program terminated with signal SIGSEGV, Segmentation fault.
(gdb) bt
No stack.

and there is no backtrace. Will try to investigate it later.

[konstin]

The segfault likely occurs because subclassing is broken

[gilescope]

How about trying dill? Pickle can't handle lots of pure python serialisation cases.
https://pypi.org/project/dill/

[davidhewitt]

Not sure if it's interesting; this snippet just got shared on gitter. https://gist.github.com/ethanhs/fd4123487974c91c7e5960acc9aa2a77

[shaolo1]

I've got a simple struct that I need to deepcopy. I'm trying to figure out how to pickle my struct (after getting the TypeError: cannot pickle error). The gist above shows how to do it for a single member, but I'm too much of a newb to see how to do this with multiple members.

I tried

pub fn __getstate__(&self, py: Python) -> PyResult<PyObject> {
        Ok(PyBytes::new(py, &serialize(&self.foo).unwrap()).to_object(py))
        Ok(PyBytes::new(py, &serialize(&self.bar).unwrap()).to_object(py))
    }

..but get an error "expected one of ., ;, ?, }, or an operator" after the first OK.

[davidhewitt]

@shaolo1 I would just return the tuple of members:

    pub fn __getstate__(&self, py: Python) -> PyObject {
        (
            PyBytes::new(py, &serialize(&self.foo)?),
            PyBytes::new(py, &serialize(&self.bar)?),
        ).to_object(py)
    }

[shaolo1]

@davidhewitt Thanks. I'll try that if I encounter it again. I got around the problem by just implementing __deepcopy__ in the parent object and handling the copy there so that pickle support was not needed in my rust object.