Pybind11: Default named arguments for init functions

[rmjarvis]

I'm working on converting a project from Boost Python to PyBind11, mostly to try to make it pip installable, since the Boost dependency is kind of a stopper for that.

So far, one thing that boost.python does nicely that I can't figure out how to do easily in pybind11 is to give default values to the arguments of an __init__ method.

For instance, here is the code to wrap one of our classes using boost::python:

namespace bp = boost::python;
bp::class_<GaussianDeviate, bp::bases<BaseDeviate> >
        pyGaussianDeviate("GaussianDeviate", "", bp::no_init);
pyGaussianDeviate
    .def(bp::init<long, double, double>(
            (bp::arg("seed")=0, bp::arg("mean")=0., bp::arg("sigma")=1.)
    ))

(There's more than this of course, but this is the relevant bit.)

So far, my attempt to convert this to pybind11 API involves:

• change namespace to pybind11 (of course)
• remove no_init
• remove bases wrapper for BaseDeviate
• remove the parentheses around the args

which leads to

namespace bp = pybind11;
bp::class_<GaussianDeviate, BaseDeviate>
        pyGaussianDeviate("GaussianDeviate");
pyGaussianDeviate
    .def(bp::init<long, double, double>(
            bp::arg("seed")=0, bp::arg("mean")=0., bp::arg("sigma")=1.
    ))

However, this is still insufficient. I get the following error from the compiler (using pybind11 2.0.0rc1)

pysrc/Random.cpp:143:40: error: use of overloaded operator '=' is ambiguous (with operand types
      'bp::arg' and 'int')
                        bp::arg("seed")=0, bp::arg("mean")=0., bp::arg("sigma")=1.
                        ~~~~~~~~~~~~~~~^~
/Users/Mike/Library/Python/2.7/include/python2.7/pybind11/attr.h:20:8: note: candidate is the
      implicit move assignment operator
struct arg {
       ^
/Users/Mike/Library/Python/2.7/include/python2.7/pybind11/attr.h:20:8: note: candidate is the
      implicit copy assignment operator
/Users/Mike/Library/Python/2.7/include/python2.7/pybind11/attr.h:24:24: note: candidate function
      [with T = int]
    constexpr arg_t<T> operator=(const T &value) const { return {name, value}; }
                       ^

Can you please tell me the correct syntax to do this in pybind11? I couldn't find anything in the docs about constructors with default arguments. The above kind of syntax (bp::arg('name') = value) works for other functions. Just not init.

I think I could do this by writing a custom initialization function that I wrap as

    .def("__init__", &CustomGaussianDeviateInit,
            bp::arg("seed")=0, bp::arg("mean")=0., bp::arg("sigma")=1.
    ))

But that's a lot of extra boiler plate that I would hope I don't have to write. (There are quite a few classes that we do this kind of thing for.) So if this is the only current solution, please consider this issue a feature request to enable the old syntax that boost python had for this (or something similar).

Thanks!

[wjakob]

Hi @rmjarvis,

you almost got it right: just put the py::arg arguments to the right of the py::init<>() call. In pybind11, any tags/modifiers are passed as variadic extra arguments to .def() and class_<>().

-Wenzel

[rmjarvis]

Thanks for the quick response. I think this is what you are suggesting:

    .def(bp::init<long, double, double>(),
            bp::arg("seed")=0, bp::arg("mean")=0., bp::arg("sigma")=1.
    )

But that still gives me the same error:

pysrc/Random.cpp:143:40: error: use of overloaded operator '=' is ambiguous (with operand types
      'bp::arg' and 'int')
                        bp::arg("seed")=0, bp::arg("mean")=0., bp::arg("sigma")=1.
                        ~~~~~~~~~~~~~~~^~
/Users/Mike/Library/Python/2.7/include/python2.7/pybind11/attr.h:20:8: note: candidate is the
      implicit move assignment operator
struct arg {
       ^
/Users/Mike/Library/Python/2.7/include/python2.7/pybind11/attr.h:20:8: note: candidate is the
      implicit copy assignment operator
/Users/Mike/Library/Python/2.7/include/python2.7/pybind11/attr.h:24:24: note: candidate function
      [with T = int]
    constexpr arg_t<T> operator=(const T &value) const { return {name, value}; }
                       ^

[wjakob]

Is it possible that you're using an old 1.8.x release? This should work with master.

[wjakob]

(A 2.0.0 release will be out fairly soon in 1-2 weeks)

[rmjarvis]

I'm using the latest pip version 2.0.0rc1.

[rmjarvis]

Update: It works if I explicitly cast the 0 as int(0). So maybe there is something funny in the function resolution for 0, since it can be a void* maybe?

    .def(bp::init<long, double, double>(),
            bp::arg("seed")=int(0), bp::arg("mean")=0., bp::arg("sigma")=1.
    )

[wjakob]

That doesn't make any sense since 0 already has the type int by default. What compiler/OS are you using?

[rmjarvis]

Apple's clang masquerading as gcc

$ gcc --version
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin14.5.0
Thread model: posix

[rmjarvis]

Should I try to trim down to a minimal example that you can look at?

[wjakob]

Can you provide a self-contained piece of code that causes the issue along with your compilation flags? I can't reproduce the issue here with the latest XCode (I tried just adding a py::arg("test")=0 to a constructor).

[rmjarvis]

Will do.

[rmjarvis]

Here is a minimal example that reproduces the error for me:

#include <pybind11/pybind11.h>

class A
{
public:
    A(int i) {}
};

PYBIND11_PLUGIN(test)
{
    pybind11::module m("test");

    pybind11::class_<A>(m, "A")
        .def(pybind11::init<int>(), pybind11::arg("i")=0);
    return m.ptr();
}

My compile command just includes the directories for Python.h and pybind11/pybind11.h:

gcc -c -std=c++14 -fPIC -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/Users/Mike/Library/Python/2.7/include/python2.7 test_pybind11.cpp 

The output on my system (XCode version 7.2) is:

$ gcc -c -std=c++14 -fPIC -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/Users/Mike/Library/Python/2.7/include/python2.7 test_pybind11.cpp 
test_pybind11.cpp:15:55: error: use of overloaded operator '=' is ambiguous (with operand types
      'pybind11::arg' and 'int')
        .def(pybind11::init<int>(), pybind11::arg("i")=0);
                                    ~~~~~~~~~~~~~~~~~~^~
/Users/Mike/Library/Python/2.7/include/python2.7/pybind11/attr.h:20:8: note: candidate is the
      implicit move assignment operator
struct arg {
       ^
/Users/Mike/Library/Python/2.7/include/python2.7/pybind11/attr.h:20:8: note: candidate is the
      implicit copy assignment operator
/Users/Mike/Library/Python/2.7/include/python2.7/pybind11/attr.h:24:24: note: candidate function
      [with T = int]
    constexpr arg_t<T> operator=(const T &value) const { return {name, value}; }
                       ^
1 error generated.

As before, if I replace 0 with int(0) it compiles. Also, if the default value is 1 rather than 0, it also compiles. So there seems to be something special about 0.

[dean0x7d]

Based on this error message

/Users/Mike/Library/Python/2.7/include/python2.7/pybind11/attr.h:24:24: note: candidate function
      [with T = int]
    constexpr arg_t<T> operator=(const T &value) const { return {name, value}; }
                       ^

you're definitely using version 1.8.1. (The latest version implements this assignment differently and in a different location.)

If you're using pip, make sure to enable pre-release versions when installing packages: pip install --pre pybind11. The default pip install will only download stable releases (1.8.1) which is most likely the issue.

BTW The bug itself was fixed in #244 -- 0 was implicitly cast to a pointer.

[rmjarvis]

@dean0x7d Thanks for the hint.

I did have 2.0.0rc installed as a python module, but it looks like pip didn't update the header files properly. When I deleted both the pybind11 include directory and the pybind11 egg in the site-packages directory, then reinstalled with pip, it installed the correct header files and the above compilation worked.

I've seen pip get confused about things like this before, which I think is a bug in how either pip or setuptools handles the headers directive. If you're interested in a workaround for this, you might consider installing the header files as package_data within the python module directory, rather than as headers which get installed to some other directory outside of your module in a semi-unpredictable way.

Then get_include can return e.g. os.path.join(os.path.dirname(__file__), 'include') rather than whatever directory pip thinks it installed things in. You have more direct control of the distribution that way.

Anyway, that's all beside the point for this particular issue, which indeed is working on the latest (pre-release) version, so I'll just leave those comments as food for thought and go on my merry way now. :)

Thanks for the help, both!

[yungyuc]

python setup.py install_headers correctly installs the header files. But I don't know why setuptools doesn't call it when install.