Fiona: GDAL 2.3 build issues: CPLSetErrorHandler and OGR_G_CreateGeometry

[schwehr]

Pardon the confusion with this bug report. I'm building with a custom bazel env and not using setup.py. But I think these will hit many people using a GDAL pre 2.3 release (and probably needs a stricter compiler config to run into trouble)

Sadly, I don't know enough about cython to figure out a patch to include with this report

Expected behavior and actual behavior.

Build failures...

I think that CPLErrorHandler needs a ctypedef for CPLSetErrorHandler

third_party/py/fiona/_drivers.pyx_cython_generated.cc:2174:3: error: no matching function for call to 'CPLSetErrorHandler'
  CPLSetErrorHandler(((void *)__pyx_f_5fiona_8_drivers_errorHandler));
  ^~~~~~~~~~~~~~~~~~
third_party/gdal/port/cpl_error.h:161:37: note: candidate function not viable: cannot convert argument of incomplete type 'void *' to 'CPLErrorHandler' (aka 'void (*)(CPLErr, int, const char *)') for 1st argument
CPLErrorHandler CPL_DLL CPL_STDCALL CPLSetErrorHandler( CPLErrorHandler );

and OGR_G_CreateGeometry can't cast an int to an enum. I copied _shim22.* to _shim.* in my build

third_party/py/fiona/_geometry.pyx_cython_generated.cc:2460:27: error: no matching function for call to 'OGR_G_CreateGeometry'
  __pyx_v_cogr_geometry = OGR_G_CreateGeometry(__pyx_v_wkbtype);
                          ^~~~~~~~~~~~~~~~~~~~
third_party/gdal/ogr/ogr_api.h:87:22: note: candidate function not viable: no known conversion from 'unsigned char' to 'OGRwkbGeometryType' for 1st argument
OGRGeometryH CPL_DLL OGR_G_CreateGeometry( OGRwkbGeometryType ) CPL_WARN_UNUSED_RESULT;
                     ^
third_party/py/fiona/_geometry.pyx_cython_generated.cc:4822:27: error: no matching function for call to 'OGR_G_CreateGeometry'
  __pyx_v_cogr_geometry = OGR_G_CreateGeometry(__pyx_v_geom_type);
                          ^~~~~~~~~~~~~~~~~~~~
third_party/gdal/ogr/ogr_api.h:87:22: note: candidate function not viable: no known conversion from 'int' to 'OGRwkbGeometryType' for 1st argument
OGRGeometryH CPL_DLL OGR_G_CreateGeometry( OGRwkbGeometryType ) CPL_WARN_UNUSED_RESULT;
                     ^

Steps to reproduce the problem.

Sorry. I haven't tried with a normal linux build.

Operating system

Linux + bazel + python 2.7 + cython 0.27 and building as C++11

Fiona and GDAL version and provenance

Fiona and GDAL from pretty close to 2018-Mar-10

• GDAL: r41665
• Fiona: https://github.com/Toblerity/Fiona/commit/16a6b289d1db953c2894b68d495df1a2aee08bfc

[rbuffat]

I'm not familiar with bazel, but Fiona seems to build against 2.3 trunk (but then fails on a few test cases).

The errors occur in the part of the code that is auto generated by Cython. How did you generate the c code? Could you provide a minimal working example of a bazel setup that reproduces this behavior?

[schwehr]

rbuffat, The bazel aspect of the build isn't really important. Likely it's that I'm building in cython C++ mode that really matters.

[rbuffat]

Would it be possible to provide a minimal minimal working example that leads to this behavior?

As said before, the standard installation method seems to work with gdal 2.3 on the CI. As the error occurs in the auto generated code by cython it could also be a cython and not a fiona issue.

Fiona wraps around the C-library gdal. As you said you compile with c++, annotating functions with 'extern "C"' could solve the issue. But I have to say that I lack the experience and without a base to start with it will be difficult to debug.

[schwehr]

Working on the issues a bit more. It requires getting a bit more explicit about types. Even and I have worked on tightening up the types in GDAL and my local build tries to be as careful with types as it can. Here are my notes so far. I'm pretty sure I'm not doing a good job of where to put things.

For OGR_G_CreateGeometry, I had to add this to _geometry.pxd:

cdef extern from "ogr_core.h":
  ctypedef enum OGRwkbGeometryType:
    wkbUnknown
    wkbPoint
    wkbLineString
         [snip]

And change the function prototype in _geometry.pxd from int to OGRwkbGeometryType:

    void *  OGR_G_CreateGeometry (OGRwkbGeometryType wkbtypecode)

For CPLSetErrorHandler, I modified _drivers.pyx (probably the wrong place for this!):

cdef extern from "cpl_error.h":
    ctypedef void (*CPLErrorHandler)(int, int, const char*);
    void CPLSetErrorHandler (CPLErrorHandler handler)

Then in GDALEnv, I had to cast to a CPLErrorHandler:

    def start(self):
        cdef const char *key_c = NULL
        cdef const char *val_c = NULL

        if GDALGetDriverCount() == 0:
            GDALAllRegister()
        if OGRGetDriverCount() == 0:
            OGRRegisterAll()
        CPLSetErrorHandler(<CPLErrorHandler>errorHandler)  # <-- cast here
        if OGRGetDriverCount() == 0:
            raise ValueError("Drivers not registered")
[snip]

Then

cannot initialize a parameter of type 'char *' with an lvalue of type 'const char *'
  __pyx_t_10 = __pyx_f_5fiona_5_shim_gdal_open_vector
 error: no matching function for call to 'OGR_L_SetIgnoredFields'
error: no matching function for call to 'GDALDatasetCreateLayer'
          __pyx_t_11 = __pyx_f_5fiona_4_err_exc_wrap_pointer(GDALDatasetCreateLayer(__pyx_v_self->__pyx_base.cogr_ds, __pyx_v_name_c, __pyx_v_cogr_srs, __pyx_v_geometry_code, __pyx_v_options)); if (unlikely(__pyx_t_11 == ((void *)NULL))) __PYX_ERR(0, 974, __pyx_L107_error)
[snip]

So in the shim, there is a missing const:

cdef void* gdal_open_vector(char* path_c, int mode, drivers, options) except NULL

becomes

cdef void* gdal_open_vector(const char* path_c, int mode, drivers, options) except NULL

Some info on my build:

    "-DWITHOUT_CPLDEBUG",
    "-DDISABLE_CVSID",
    # "-DSTRICT_OGRERR_TYPE",
    # "-DUSE_IN_GDAL",  # For pcraster/libcsf.
    "-DGDAL_COMPILATION",
    "-D_GNU_SOURCE",
    # "-DPAM_ENABLED",
    "-DHAVE_EXPAT",
    "-DHAVE_LIBZ",
    "-DHAVE_GEOS",
    "-DHAVE_MITAB",
    "-DBIGTIFF_SUPPORT",
    "-DGDAL_NO_AUTOLOAD",
    "-DGDAL_NO_HARDCODED_FIND",
    "-DINTERNAL_LIBGEOTIFF",  # Says to use the GDAL version of SetCSVFilenameHook.
    "-DUSE_CPL",
    "-DPROJ_STATIC",
    "-DPROJ_VERSION=5",

[sgillies]

@schwehr thanks for the notes! Adding typedefs for everything will be an all-around win.

[snorfalorpagus]

I can reproduce this on macOS by setting the cython language in setup.py to c++:

ext_options["language"] = "c++"
ext_options["extra_compile_args"] = ["-std=c++11"]

The edits from @schwehr seem to work, but I'm having some problems with the OGRFieldSubType type in _shim22.pyx.

[sgillies]

@snorfalorpagus it's not clear to me if #672 resolves this issue. What's left?

[sgillies]

Confirmed that building in c++ mode works.