Fiona: Crash with pypy

[kannes]

I tried running things with pypy and got crashes. They are weird and this is the minimal I got:

import fiona
import json

with fiona.open("ne_10m_admin_0_countries.shp") as source:
    countries = list(source)

with open("randomfilenameiuiubiqubeiquequze", "w") as sink:
    for country in countries:
        sink.write(json.dumps(country)+"\n")

The shapefile is from the latest http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_countries.zip

Running this with pypy3 I get three different tracebacks, seemingly on random.

RPython traceback:
File "rpython_jit_metainterp_1.c", line 41233, in assembler_call_helper_65
File "rpython_jit_metainterp_8.c", line 21619, in AbstractResumeGuardDescr_handle_fail
File "rpython_jit_metainterp_8.c", line 30933, in AbstractResumeGuardDescr__trace_and_compile_from
File "rpython_jit_metainterp_8.c", line 38152, in MetaInterp_handle_guard_failure
File "rpython_jit_metainterp_8.c", line 41192, in MetaInterp__handle_guard_failure
File "rpython_jit_metainterp_1.c", line 5322, in MetaInterp__interpret
File "rpython_jit_metainterp_2.c", line 22084, in MIFrame_run_one_step
File "rpython_jit_metainterp_4.c", line 13161, in handler_ref_return_1
File "rpython_jit_metainterp_4.c", line 61659, in MetaInterp_finishframe
* Invalid usage of a dying CPython object *

cpyext, the emulation layer, detected that while it is calling
an object's tp_dealloc, the C code calls back a function that
tries to recreate the PyPy version of the object. Usually it
means that tp_dealloc calls some general PyXxx() API. It is
a dangerous and potentially buggy thing to do: even in CPython
the PyXxx() function could, in theory, cause a reference to the
object to be taken and stored somewhere, for an amount of time
exceeding tp_dealloc itself. Afterwards, the object will be
freed, making that reference point to garbage.

PyPy could contain some workaround to still work if
you are lucky, but it is not done so far; better fix the bug in
the CPython extension.
Aborted (core dumped)

or

RPython traceback:
File "rpython_jit_metainterp_2.c", line 50548, in handle_jitexception_68
* Invalid usage of a dying CPython object *
(...)

or

RPython traceback:
File "pypy_module_pypyjit.c", line 183, in portal_12
File "pypy_interpreter_2.c", line 40731, in handle_bytecode__AccessDirect_None
File "pypy_interpreter_2.c", line 51912, in dispatch_bytecode__AccessDirect_None
* Invalid usage of a dying CPython object *
(...)

Fiona Fiona-1.7.11.post1-cp36-cp36m-manylinux1_x86_64.whl
pypy3 says: Python 3.5.3 (3f6eaa010fce78cc7973bdc1dfdb95970f08fed2, Feb 12 2018, 22:20:44) [PyPy 5.10.1 with GCC 7.3.0] on linux
gdal 2.2.3
Archlinux

[kannes]

Reading the same data but from a geojson (converted via Fiona as above but without pypy) it sometimes exits cleanly but I also get these:

RPython traceback:
File "pypy_interpreter_1.c", line 40991, in execute_frame
File "pypy_interpreter_1.c", line 51447, in GeneratorOrCoroutine_resume_execute_frame
File "pypy_interpreter_1.c", line 62087, in GeneratorOrCoroutine_next_yield_from
* Invalid usage of a dying CPython object *

RPython traceback:
File "rpython_jit_metainterp.c", line 33505, in ll_portal_runner__Signed_arrayPtr_Signed_rpython_1
File "rpython_rlib_rsre.c", line 24234, in portal_23
File "rpython_jit_metainterp_1.c", line 13854, in crash_in_jit_29
* Invalid usage of a dying CPython object *

💩💩💩

[rbuffat]

I'm not sure if Fiona is able to run with pypy, it is at least currently not tested for it.

It looks like that Cython (which is used by Fiona) has some issues with pypy. After reading [1] it looks suspiciously like a memory management issue.

[1] http://cython.readthedocs.io/en/latest/src/userguide/pypy.html#borrowed-references-and-data-pointers

[micahcochran]

If you are only using a shapefile, you might try pyshp, which is python-only code. Currently, it isn't tests for pypy, but it might not be too hard to get it to working.

[sgillies]

@kannes I'm going to close this one after noting in the readme that Fiona is unsupported on pypy. It's okay that it isn't: the goal of pypy is to make Python code run faster, and Fiona is mostly written in C and already as fast as it can be.

[kannes]

Total agreement here :)

[rbuffat]

@sgillies The counter argument would be, that somebody has some rather large computations, that he wants to speed up with pypy, and then export the results with fiona. But I think this will be a rather limited use case.

[sgillies]

Don't get me wrong, I'd love for this to work with pypy and it may not be a ton of work, but it will have to be low priority for now.

[kannes]

pypy say they might be interested: https://twitter.com/pypyproject/status/975301441735876609
If someone posts this to their bug tracker, they might take a look.