Pybind11: Please enrich docs for splitting module into several files

[mmodenesi]

As stated in the docs _It’s good practice to split binding code over multiple files_. A small auto-contained example consisting of three files is provided.

Please add basic g++ command lines to compile and link that very example into a usable python module.

[molpopgen]

You basically add the different .cpp files to the sources list for the target in the CMakeLists.txt file, if using that system to build (which is covered elsewhere in the docs).

[mmodenesi]

Thanks for the reply. I need to incorporate pybind11 to a large project using make. I am struggling to make it work with plain g++.

[molpopgen]

There should be nothing special about using make directly. If you can compile one file, then you can compile many separately and then link them at the end. You just need to handle all the symbol visibility and FLTO stuff yourself.

[mmodenesi]

Can you provide the required commands that will fullfil that goal?

I am not aquainted with c++ compilation, maybe it's a trivial operation but I am trying random stuff getting "this symbol is not visible" here and there.

The issue is a request to make the documentation more complete and lower the barrier for new people to use pybind11.

[molpopgen]

Are you able to compile a single file? Your issue is about multiple, but
are you able to get a single one compiled?

On Fri, Sep 27, 2019, 6:36 AM mmodenesi [e-mail] wrote:

Can you provide the required commands that will fullfil that goal?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pybind/pybind11/issues/1934?email_source=notifications&email_token=ABQ6OH6TURNG3WDLB2XYOM3QLYD2LA5CNFSM4I2TLQPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7Y5GGY#issuecomment-535941915,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQ6OHZLV7IHYKWYQWEUQFDQLYD2LANCNFSM4I2TLQPA
.

[mmodenesi]

I don't see how that is relevant, but yes, I can compile a single unit and use the module. See if you wish this gist of mine.

The fisrt steps sections says:

On Linux, the above example can be compiled using the following command:

$ c++ -O3 -Wall -shared -std=c++11 -fPIC `python3 -m pybind11 --includes` example.cpp -o > example`python3-config --extension-suffix`

I suggest to complete the FAQ about splitting the module with a paragraph like this:

On Linux, the above example can be compiled using the following commands:

$ c++ --complie-one-unit-from-the-avobe-example
$ c++ --compile-the-other-as-well
$ c++ --link-them-at-the-end-handling-all-the-symbol-visibility-and-FLTO-stuff

voilà.

[sizmailov]

Scheme is same as for compiling an executable from multiple files. Here we go:

# compile multiple files one by one 
%.o : %.cpp
    @echo building module part
    $(CC) $(CCFLAGS) $(INCLUDES) $^ -o $@ $(LIBS)

# combine multiple files in one module
binding: someclass_1.o someclass_2.o
    @echo building $(MODULE)
    $(CC) $(CCFLAGS) $(INCLUDES) $^ -o $(MODULE) $(LIBS)

[molpopgen]

@sizmailov hit the relevance part--once you can compile one file with g++, you can compile multiple with a Makefile.

However, the example given should probably use CXX instead of CC. Unfortunately, when building things for Python, you may need to use a mix of CCFLAGS and/or CXXFLAGS.

[molpopgen]

Ah, I see that CC is used in the gist. So, this has nothing to do with pybind11, but you should look into the expected variables for compilers. The C++ compiler is CXX, the C compiler is CC. Likewise with the FLAGS variables. Using the expected names would be preferable because they may be set by the system, as is common on HPC systems, conda environments, etc..

In fact, one generally does not set CC nor CXX in a Makefile. Rather, it is expected to be supplied by the system. Likewise, the usual thing to do is to append to existing CFLAGS/CXXFLAGS variables rather than over-write them.

[mmodenesi]

$ cat example.cpp 
#include <pybind11/pybind11.h>

namespace py = pybind11;

void init_ex1(py::module &);
void init_ex2(py::module &);


PYBIND11_MODULE(example, m) {
        init_ex1(m);
        init_ex2(m);
}

$ cat ex1.cpp 
#include <pybind11/pybind11.h>

namespace py = pybind11;

void init_ex1(py::module &m) {
    m.def("add", [](int a, int b) { return a + b; });
}

$ cat ex2.cpp 
#include <pybind11/pybind11.h>

namespace py = pybind11;

void init_ex1(py::module &m) {
        m.def("sub", [](int a, int b) { return a - b; });
}

$ g++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) example.cpp -o example.o -lpython3.6m
$ g++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) ex1.cpp -o ex1.o -lpython3.6m
$ g++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) ex2.cpp -o ex2.o -lpython3.6m
$ g++ -O3 -Wall -shared -std=c++11 -fPIC *.o -o example$(python3-config --extension-suffix) -lpython3.6m
$ python3 -c 'import example'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dynamic module does not define module export function (PyInit_example)

I have not clue what I am doing, hence my request for a recipe in the docs.

[molpopgen]

This may help?

[mmodenesi]

I was able to compile the splitted example with the next commands. I am not saying it is the way to go, just that this works:

$ g++ -O3 -Wall -fvisibility=hidden -shared -std=c++14 -fPIC `python3 -m pybind11 --includes` -c ex1.cpp -o ex1.o
$ g++ -O3 -Wall -fvisibility=hidden -shared -std=c++14 -fPIC `python3 -m pybind11 --includes` -c ex2.cpp -o ex2.o
$ g++ -O3 -Wall -fvisibility=hidden -shared -std=c++14 -fPIC `python3 -m pybind11 --includes` -c example.cpp -o example.o
$ g++ -O3 -Wall -fvisibility=hidden -shared -std=c++14 -fPIC ex1.o ex2.o example.o -o example`python3-config --extension-suffix`

After that:

$ python3 -c "import example; print(dir(example))"
['__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'add', 'sub']

NOTE: the documentation has a tiny bug, both ex1.cpp and ex2.cpp define a function named init_ex1, and none of them defines a function named init_ex2.

I will leave the issue open for the owners of the project to consider the original request of improving documentation with the bare g++ commands required to compile and link the splitted module.

[rosamc]

Please could somebody give a clear example of a proper Makefile that allows compiling code splitted in multiple files, in a way that avoids unnecessary recompiling when changing just one file? Thanks in advance.

[molpopgen]

Please could somebody give a clear example of a proper Makefile that allows compiling code splitted in multiple files, in a way that avoids unnecessary recompiling when changing just one file? Thanks in advance.

This is in the manual here.

[rosamc]

I don't see the corresponding Makefile in the manual.
After some trial and error I ended up with this test case. It works, but is it the correct way to do it?

main.cpp:
`

include

include "functions.h"

namespace py=pybind11;

PYBIND11_MODULE(main, m) {
m.def("testsumarf", &testsumarf, "Sum two arrays that are the same size.",py::arg("ar1"),py::arg("ar2"));
m.def("testmultarf", &testmultarf, "Multiply two arrays that are the same size.",py::arg("ar1"),py::arg("ar2"));
}
`

testmultar.cpp:

`

include

include

include

include

include "functions.h"

using namespace std;
namespace py=pybind11;

py::array_t testmultarf(py::array_t array1, py::array_t array2) {
if (array1.size()!=array2.size()){
throw "Arrays are not the same size.";
}
auto ar1buf=array1.request();
double *ar1=(double *) ar1buf.ptr;
auto ar2buf=array2.request();
double *ar2=(double *) ar2buf.ptr;

int N=array1.size();
py::array_t<double> resultpy = py::array_t<double>(N);
py::buffer_info bufresultpy = resultpy.request();
double *ptrresultpy=(double *) bufresultpy.ptr;
for (int i=0;i<N;i++){
    ptrresultpy[i]=ar1[i]*ar2[i];
}
return  resultpy;

}
`

testsumar.cpp
`

include

include

include

include

include "functions.h"

using namespace std;
namespace py=pybind11;

py::array_t testsumarf(py::array_t array1, py::array_t array2) {
if (array1.size()!=array2.size()){
throw "Arrays are not the same size.";
}
auto ar1buf=array1.request();
double *ar1=(double *) ar1buf.ptr;
auto ar2buf=array2.request();
double *ar2=(double *) ar2buf.ptr;

int N=array1.size();
py::array_t<double> resultpy = py::array_t<double>(N);
py::buffer_info bufresultpy = resultpy.request();
double *ptrresultpy=(double *) bufresultpy.ptr;
for (int i=0;i<N;i++){
    ptrresultpy[i]=ar1[i]+ar2[i];
}
return  resultpy;

}
`

functions.h:
`

include

include

namespace py=pybind11;

py::array_t testmultarf(py::array_t array1, py::array_t array2);
py::array_t testsumarf(py::array_t array1, py::array_t array2);
`

Makefile:
`
CC=c++
IDIR:=$(shell python3 -m pybind11 --includes)
CPPFLAGS= -O2 -DNDEBUG -Wall -std=c++11 -fPIC $(IDIR)
LDLFLAGS= -shared -undefined dynamic_lookup
LAST:=$(shell python3-config --extension-suffix)
TARGET_LIB=main$(LAST)

DEPS=functions.h

SRCS=main.cpp testmultar.cpp testsumar.cpp
OBJS=$(SRCS:.cpp=.o)

all: $(TARGET_LIB)

$(TARGET_LIB):$(OBJS)
$(CC) $(CPPFLAGS) $(LDLFLAGS) $^ -o $@

$(SRCS:.cpp=.o):%.o:%.cpp $(DEPS)
$(CC) $(CPPFLAGS) $^ -c
`

[molpopgen]

I don't see the corresponding Makefile in the manual.

Use cmake to generate the Makefile, which is described elsewhere in the manual.

[rosamc]

Ok, thanks.

[bstaletic]

%.cpp:%.o
    $(CXX) $(CXXFLAGS) -fPIC `python -m pybind11 --includes` -c $< -o $@
module.so:$(DEPENDENCIES)
    $(CXX) $(LDFLAGS) -fPIC -shared -o $@ $^

The above should be a fairly good starting point for anyone who needs make.

@molpopgen Thanks for answering the other questions.