Vision: Using torchvision C++ API without installing python

[stu1130]

❓ Questions and Help

I have followed the document and built the libtorchvsion.dylib but when I ran otool -L libtorchvsion.dylib it showed

libtorchvision.dylib:
    @rpath/libtorchvision.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libc10.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libpython3.8.dylib (compatibility version 3.8.0, current version 3.8.0)
    @rpath/libtorch.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libtorch_cpu.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 902.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)

Is there a way to get rid of libpython3.8.dylib, I think it is required for using python custom operators and we will only use C++.

[lanking520]

I managed to build the so with the hack:

https://github.com/pytorch/vision/blob/master/torchvision/csrc/cuda/vision_cuda.h#L2 change this line to include torch/torch.h.

Same with CPU. Also disable the unncessary pybind API call.

• need change cmake to disable all python3 packages.

However I don't feel like this is the right way to do. Is there any side effect if I disabled all python access?

[fmassa]

@mthrok has been looking into something like that for torchaudio and torchtext, maybe his findings could apply for torchvision as well? We might need to move away from cpp_extensions for building the C++ extensions, and instead use CMake for that.

[mthrok]

I am working on gathering requirements and specing out cmake build for torchtext and torchaudio, which includes both Python extension module and C++ library. I thought torchvision has already have cmake build so did not look into it but if it's something beneficial to torchvision, I can include that one too.

So as to build C++ library, the source code cannot use things that are related to Python. This includes PyBind11 and <torch/extension.h>, which, in my understanding, is a wrapper of PyBind11 (I could be wrong here). So it will be easier if we can get rid of <torch/extension.h> and replace it with <torch/script.h>, but this will involve some code change on Python side too because the way C++ functions are exposed are different. That is, with PyBind11, the extension module is treated as a regular Python module, but TorchScript based binding put all the functions under torch.ops or torch.classes.

The way I am thinking of cmake build is some thing like the following;

• CMakeLists.txt exposes two targets; C++ library and Python extension.
• These two targets depend on internal targets which include all the third-party libraries and custom C++ objects that does not include any Python-related code.
• When building the Python extension, setup.py resorts to call cmake command with supplement arguments, like Python header/library locations and PyTorch specific configurations.
• With this CMakeLists.txt, one can just add domain library as third party and do add_subdirectory(vision/text/audio).

Here is an example how Python extension can be build with cmake via setup.py.

[fmassa]

We already solved the torch.ops part in torchvision, and we also depend only on <torch/script.h>https://github.com/pytorch/vision/blob/15848edbc3c5282ced487a62192fbc763d052019/torchvision/csrc/vision.cpp#L2, but IIRC we had to also include <Python.h> for some magical reasons such as https://github.com/pytorch/vision/blob/15848edbc3c5282ced487a62192fbc763d052019/torchvision/csrc/vision.cpp#L19-L35 (although maybe this is legacy and could be removed).
Due to those two points, our CMake file currently depends on Python https://github.com/pytorch/vision/blob/15848edbc3c5282ced487a62192fbc763d052019/CMakeLists.txt#L33 but it shouldn't be the case.

[mthrok]

Yeah, torchtext also depends on Python because they bind functions via PyBind alongside with TorchScript for performance reason.

I think if they can separate registration (binding) and implementation, then things could work nicely.
Here is the diagram to illustrate the idea

from the doc I am working (internal only)

[mthrok]

, but IIRC we had to also include <Python.h> for some magical reasons such as
https://github.com/pytorch/vision/blob/15848edbc3c5282ced487a62192fbc763d052019/torchvision/csrc/vision.cpp#L19-L35

(although maybe this is legacy and could be removed).

At a glance it looks like you can remove Python2 portion at least.

[lanking520]

@mthrok do you have any plan to fix this? Or can I presume it is okay to temporaily applied the fix I mentioned and will have no side effects?

[stu1130]

Could it be a CMake option like USE_PYTHON? When it is disabled, we skip including <torch/extension.h> with macro.

[mthrok]

@mthrok do you have any plan to fix this? Or can I presume it is okay to temporaily applied the fix I mentioned and will have no side effects?

I do not have a plan to fix this yet. My work is still proposal phase, and torchvision is more complicated than torchtext or torchaudio.

However if @fmassa's comment

We already solved the torch.ops part in torchvision, and we also depend only on

is correct, then I think changing <torch/extension.h> to <torch/torch.h> or <torch/script.h> should suffice, and there should be no side effect. (I could be wrong though)

Searching "pybind11" reveals there is one test code that relies on PyBind11, and that code should include <torch/extension.h> by itself.

[mthrok]

Could it be a CMake option like USE_PYTHON? When it is disabled, we skip including <torch/extension.h> with macro.

That's what I am working on, but in the case of torchvision, <torch/extension.h> should not be needed, I presume.

[fmassa]

Searching "pybind11" reveals there is one test code that relies on PyBind11, and that code should include by itself.

Good catch. This is actually an artifact of testing that should be changed -- we wrapped the C++ models in Python so that we could compare the outputs of the model. But torchvision library itself only uses torch/script.h.

[stu1130]

@fmassa @mthrok is there any update? Thanks

[datumbox]

@stu1130 This is still currently in progress. There are a few PRs that try to minimize the use of torch/script.h see https://github.com/pytorch/vision/pull/3087#discussion_r539307310 and #3146.