Ifcopenshell: Precompiled distribution of PythonOCC

[Moult]

It is very convenient to be able to just put the distributed ifcopenshell in python's site packages directory and then immediately be able to import ifcopenshell. This is a bit off-topic, but is it possible to do the same for PythonOCC so that we can do import OCC.foo?

That would help bundle python utilities that use both IfcOpenShell and PythonOCC easily, since installing PythonOCC is quite an involved process.

[aothms]

PythonOCC has some other dependencies such as GUI frameworks which makes bundling inherently more difficult.

They opted for conda, but unfortunately builds are not published for the recent python versions.

It seems @tpaviot and @jf--- are a bit on a break in terms of PythonOCC. One way to solve this is to put PythonOCC on conda-forge (like IfcOpenShell is due to a great contribution #582, also occt is now) perhaps @rainman110 has ideas on this as well, since he has contributed a lot in the past wrt conda.

[rainman110]

@aothms In fact we are working on a pythonocc 7.4.0 release, that targets OCCT 7.4.0.

This release currently targets python 3.6, 3.7 and 3.8.

We could easily make pyqt optional in the conda packages. Would that help?

[tpaviot]

@aothms no break at all :smiley:

[Moult]

@rainman110 Making PyQt optional would _definitely_ be a huge help! Qt is a big beast! I'd be more than happy to test it for you :)

So as of today, what's the current "easiest" method to get PythonOCC installed on Windows?

[tpaviot]

@Moult try conda install -c dlr-sc/label/dev pythonocc-core This is really easy, without using any double quote, dont' you think ?

[Moult]

@tpaviot thanks! I got OCC working - Edit: I think IfcOpenShell is still on pythonocc 0.17, whereas pythonocc has now moved to OCC-7.4.0. Are there plans to upgrade? @aothms

[aothms]

@Moult see #737

[Moult]

Awesome @aothms ! When it gets merged, I will migrate my code to use OCC 7.4.0 too! I assume there have been quite a few improvements since.

Will close this issue :) thank you everyone

[Moult]

Reopening, since I've been trying to package this with Blender so that it works out-of-the-box, with a few struggles. Note, that I am currently attempting to package pythonocc-core 0.17.3 with oce 0.17.2.

Here are the versions I'm currently grabbing for each dependency on each OS:

IfcOpenShell

https://github.com/IfcOpenBot/IfcOpenShell/commit/9bcd932bed48486bf5b5f48d24b49329c280462f
https://s3.amazonaws.com/ifcopenshell-builds/ifcblender-python-37-v0.6.0-9bcd932-win64.zip
https://s3.amazonaws.com/ifcopenshell-builds/ifcblender-python-37-v0.6.0-9bcd932-macos64.zip
https://s3.amazonaws.com/ifcopenshell-builds/ifcblender-python-37-v0.6.0-9bcd932-linux64.zip

Python-OCC

https://anaconda.org/DLR-SC/pythonocc-core/files?page=2
https://anaconda.org/DLR-SC/pythonocc-core/0.17.3/download/win-64/pythonocc-core-0.17.3-py37he980bc4_10.tar.bz2
https://anaconda.org/DLR-SC/pythonocc-core/0.17.3/download/osx-64/pythonocc-core-0.17.3-py37h04f5b5a_10.tar.bz2
https://anaconda.org/DLR-SC/pythonocc-core/0.17.3/download/linux-64/pythonocc-core-0.17.3-py37h6bb024c_10.tar.bz2

OCE

https://anaconda.org/DLR-SC/oce/files
https://anaconda.org/DLR-SC/oce/0.17.2/download/win-64/oce-0.17.2-he980bc4_14.tar.bz2
https://anaconda.org/DLR-SC/oce/0.17.2/download/osx-64/oce-0.17.2-h04f5b5a_12.tar.bz2
https://anaconda.org/DLR-SC/oce/0.17.2/download/linux-64/oce-0.17.2-h6bb024c_14.tar.bz2

TBB

https://anaconda.org/DLR-SC/tbb/files
https://anaconda.org/DLR-SC/tbb/2019.5/download/win-64/tbb-2019.5-he980bc4_0.tar.bz2
https://anaconda.org/DLR-SC/tbb/4.3.6/download/osx-64/tbb-4.3.6-0.tar.bz2
https://anaconda.org/DLR-SC/tbb/4.3.6/download/linux-64/tbb-4.3.6-0.tar.bz2

Blender comes with its own Python 3.7 interpreter, so the version must match. It specifies its own sys.path, too, in a folder called scripts/addons/modules/. IfcOpenShell has a Blender Python 3.7 build, so simply placing scripts/addons/modules/ifcopenshell/ is enough to allow me to do import ifcopenshell in Blender without issue.

PythonOCC is a bit more complex. Placing the site-packages/OCC into scripts/addons/modules/OCC is not enough, as it requires OCE and TBB. On Windows, this is easily solved, I can download the Conda build and place Library/bin/*.dll into scripts/addons/modules/OCC/. So when I do import OCC.gp in Blender's interpreter, it loads scripts/addons/modules/OCC/_gp.py, and then that manages to find and load scripts/addons/modules/OCC/TKMath.dll.

On Linux, naturally, there are no dll files. Instead, I have lib/*.so files. However, simply placing those *.so files into scripts/addons/modules/OCC/ is not enough for the Blender interpreter to find them. The only fix I know so far is to export LD_LIBRARY_PATH=/path/to/scripts/adodns/modules/OCC/ before running Blender (assuming that the *.so files are copied into there.

I am not sure about Mac, but I assume it's similar to Linux.

I'd like to be able to have Linux (and Mac?) simply find the *.so files without needing to export LD_LIBRARY_PRELOAD. Is this possible? Would it require a special Blender build, similar to how IfcOpenShell does it?

Note: I have not tried with 7.4.0. When IfcOpenShell has a build available with the merged fixes, I can start migrating code too, and then I might find the same issue with 7.4.0.

[aothms]

@Moult I'm not sure what additional magic is contained in the conda build process. It actually does some stuff with the library prefixes that might be biting you now. Perhaps it's safer to just do the build yourself?

I thought the linker env vars are not really necessary normally for python .so libs as I think Python just searches sys.path itself? Did you try adding prepending to sys.path?

[Moult]

I tried appending to sys.path without success. I think on Windows it treats the lib path and the module path as the same and it's happy to load DLLs from there. On Linux (and Mac), they seem to be separate. Not sure if it is due to Conda magic or due to simply how the systems work ... but ifcopenshell doesn't seem to have this problem.

I'd like to get it working with the Conda builds simply because I am extremely inexperienced with compilation, and also Conda compiles for all three OSes, whereas I only have access to 2.

[tpaviot]

@Moult FYI, there's a docker image running at https://mybinder.org/v2/gh/tpaviot/pythonocc-binderhub/master including latest pythonocc (7.4.0beta2), gmsh4.4.1 and latest ifcopenshell-v0.6.0

[aothms]

and also Conda compiles for all three OSes, whereas I only have access to 2.

Yeah OSX is really a hindrance for open source platforms due to their license terms forbidding virtualized use on non-mac hardware. But I also wouldn't spend too much time on packaging issues, at least not at the expense of actual development that's fun and rewarding. At some point somebody will stand up in the community to do all platform builds or somebody with more conda / runtime linker knowledge will provide guidance or somebody will provide instructions to build on OSX or use @tpaviot Docker image on OSX or ...

Edit: Or maybe a last actual suggestion, see if you can get conda / miniconda to work in Blender and install your deps lazily. The anaconda devs are also pretty helpful and interop between these two might benefit many.

[rainman110]

@aothms and @Moult Packing on OSX on VMs is actually possible. You can use either Travis or Azure. Both are using macOS VMs.

You can have a look to our conda recipe repository, that builds conda packages for all platforms (win, linux, mac):
https://github.com/DLR-SC/tigl-conda

[Moult]

I've managed to fix this :) To fix it, I checked with with chrpath -l _gp.so to see where its rpath was, and the Conda builds for both Mac and Linux have it set to $ORIGIN/../../.., where $ORIGIN, from what I read online, is the cwd of _gp.so. So ensuring that the libs were in scripts/addons/modules/OCC/../../../ fixes it - and can be verified with ldd _gp.so! An alternative fix would've been to download the Conda build then use chrpath to change the rpath location but I didn't really want to get into that.

Hopefully somebody can propose a better fix in the future :) Closing this again, thanks for all the suggestions!

[aothms]

@Moult sorry, I finally realized you weren't talking about the wrapper libraries generated by SWIG (.pyd on Windows), but the dependencies of those wrapper libs. Then indeed it's simply the OS runtime linker at play and indeed if you want to reuse the conda builds with hardcoded rpaths I think the options you describe are about as good as it gets.

@rainman110 Thanks, your conda scripts have been a source of inspiration all along :) I didn't know Travis offered macOS.