Ifcopenshell: Have a complete opensource and OpenBIM workflow in a production pipeline

[Moult]

So many things to do, I thought I might use this bug as a TODO list, instead of many smaller bugs. This is also good for community transparency instead of just a list in my head :)

Here's the list so far in absolutely no particular order at all.

1. Add support to generate a section / plan / RCP cut for construction documentation, with annotation
2. Package the IFC diff utility as a standalone binary, so that people can use it to see differences between IFC files and prevent the need to federate unchanged IFCs (huge time waster in a commercial environment!)
3. Create and package clash detection with filters. Relies on being able to build fcl for Windows.
4. Improve the UI to quickly generate and manage unit tests with BIMTester from Blender
5. Add Gherkin definitions for checking naming conventions with regex, checking the existence of properties, checking the value of properties, and checking classification systems
6. Implement and create a guide for running an OpenBIM server with Git (+LFS) which has a hook to call unit tests and shows reports. In progress: we have a full project on Git+LFS already.
7. Implement support to import and view BCF issues - in progress
8. Implement the construction sequencing entities
9. Allow you to choose a user profile so that owner history is recorded properly
10. Properly set application metadata don't just leave them blank. Basically fixed, but can be improved depending on bug #747
11. Rewrite the product types system to have its own Blender property and not rely on parenting, which doesn't scale on large projects, similarly, create the necessary operators to assign / unassign types. Related to #750
12. Have the ability to "patch" an IFC file instead of doing a full export, to fix bits and pieces of errors in other IFC files, otherwise known as an incremental write or partial write.
13. Test BlenderBIM integration with Speckle
14. Test BlenderBIM IFC suppport in some web-based viewer to then integrate with Git (note: proposed to xeokit dev to make xeokit frontend FOSS - rejected. Plans to test with BIMSync. Some tests done with BIMSurfer.)
15. Allow you to easily switch between and create geometric representation contexts for an entire scene or individual objects. Partially done.
16. Create an open-source IFC optimiser which recycles duplicate non-rooted elements.
17. Create more functions in Blender to visualise diffs in a nicer way.
18. Link Blender to an open-source asset repository so you can drag and drop IFC asset components
19. Round-trip simple parametric modifiers by agreeing to an unofficial standard for storing these, preferably in a way that FreeCAD can read too

20. Store IFC in a queriable efficient format for data warehousing (SQLite doesn't seem capable to store the complex relationships of IFC, and HDF5 support is promising, but currently not stable enough for production use)

    1. Connect open source BIM models to OSM and create an open source BIM model repository, perhaps called OpenBIMMaps.
    2. Create an open source asset repository
    3. Set up a build server that runs tests headlessly, published sheets headlessly, does diff's, and clashes headlessly and package it to be easy to install.
    4. Start packaging and recommending default Blender utilities for mechanical, pipe, and hard surface modeling, including things like this / this.
    5. Basic round-trip import/export in Blender itself. Depends on #750. Done.
    6. Set-up the foundation for BlenderBIM online documentation. Done. See #755
    7. Allow connection to an OpenCDE API, including auth and BCF management.
    8. Create ways to easily search, query, and visualise properties in an IFC file.
    9. Create import settings to easily get cleaned mesh models and less objects to make it easier for artistic users.
    10. Finish implementing the structural entities to integrate with the IFC2CA converter

    ... this issue is a WIP and will be edited heavily!

[EAzari]

1. Project setup, IfcXtreme
2. BCF 2.1
3. New design system, needs Everything Nodes
4. MaterialPass
5. glTF, WebGL, and WebGPU
6. ProductPass
7. FacilityPass

[aothms]

Test BlenderBIM IFC suppport in Xeokit

If I'm not mistaken, Xeokit uses IfcConvert so checking the DAE output would be a reasonable starting point for that.

Have the ability to "patch" an IFC file instead of doing a full export, to fix bits and pieces of errors in other IFC files

This is definitely an interesting use case. What's your plan for this?

[Moult]

It seems Xeokit is not completely open-source, as commercial uses require a commercial license, so I might look at other web IFC viewers instead of Xeokit (or use their underlying MIT-based engine with a different viewer).

The usecase is that we often get IFCs which have incorrect data where the authoring tool is incapable of exporting it correctly. Such as when objects are modeled using the curtain wall in Revit (which is very hard to change IFC class), or using multiple IfcClassification in Revit (which has limited support for classifications), or the big one, which is Revit not exporting georeferencing data. We need to be able to patch these IFC files to have the correct data, and doing a full import/export risks data loss.

Don't currently have a plan, but I think combining a little utility with the Gherkin syntax could be _very_ powerful and easy for people to use :)

[EAzari]

Test BlenderBIM IFC suppport in Xeokit (note: Xeokit is not 100% open source, so maybe use something else)

I think we need a workflow through JSON and IFC to glTF JSON-based and also Binary-based (which Thomas has developed before its Binary-based pipeline)

Also, there are a lot of invaluable efforts on glTF/WebGL from giant companies like Mozilla, Facebook, Google, etc.

Store IFC in a queriable efficient format for data warehousing (first I thought SQLite, but @aothms has his own thing I need to try out)

I clearly know Thomas' view and like many "properly" suggests OWL/RDF/Triple, which is good but just is for Linked-Data side, another side is BIG Data which can be or can't be based on OWL/RDF/Triple

So, my suggestion is SQL/SQLite that can handle JSON, so with JSON and JSON-LD we can handle OWL/RDF/Triple

This approach can handle structured, semi-structured and unstructured data together better, and cause we don't convert structured data (Tabular data) to semi-structured data (Object-Oriented)

[aothms]

I clearly know Thomas' view and like many "properly" suggests OWL/RDF/Triple, which is good but just is for Linked-Data side, another side is BIG Data which can be or can't be based on OWL/RDF/Triple

I agree with your concerns on big data in RDF and its inability to represent ordered data (tabular, lists) efficiently / effectively. But I think, but not sure, that @Moult is referring to HDF5, which has support for more complex types, such as a aggregates.

In terms of viewer. I have some plans to maintain the BIMSurfer v2 version. https://github.com/AECgeeks/bimsurfer2 previously https://github.com/opensourceBIM/BIMsurfer-before2019 and superseded by https://github.com/opensourceBIM/BIMsurfer but the last one no longer supports glTF. I'm adding ThreeJS and SVG backends all using the same API.

JSON in SQLite sounds interesting honestly, but it'd be interesting to see how it compares to the native (portable) json databases.

[Moult]

Yes, I think I was referring to HDF5 - @aothms mentioned it verbally to me previously but I forgot it :) I'll take a closer look at it to see how easy it is to dump IFC data into it, and write queries. If you have an IFC specific how-to lying around, @aothms I'd love to read it :) otherwise I'll just tinker :)

Last I tried bimsurfer, it had to load a model from a bimserver. Is it possible to make it more agnostic of the backend so I can just run bimsurfer standalone locally and do bimsurfer.load({ file: "foo.IFC"})?

[aothms]

I'll take a closer look at it to see how easy it is to dump IFC data into it, and write queries

I think currently there is only this. A version of IfcConvert never merged into master: https://github.com/ISBE-TUe/IfcOpenShell-HDF5/releases

There is a comprehensive set of files floating around here in case you want to have a look just at reading using h5py https://zenodo.org/record/1172520

Last I tried bimsurfer, it had to load a model from a bimserver

BIMSurfer v2 can read from glTF but it's not agnostic in the way you propose because the geometry processing happens in the back-end, either IfcConvert or BIMServer and in the latter case it can incrementally stream the triangulated meshes over web sockets.

[Moult]

Thanks @aothms - I'll investigate them.

I hope I don't misunderstand - are you saying that BIMSurfer v2 can use IfcConvert as a backend without BIMServer? How does this work? I don't see IfcConvert able to convert to glTF as a format here? Perhaps another format?

[EAzari]

@Moult - As long as I know IfcConvert can convert Ifc (geometry) to OBJ and then through another solution can convert OBJ to glTF. (Also, semantics to XML)

It seems that @aothms mainly was focused on COLLADA, but COLLADA wasn't successful internationally and glTF nominated

Thomas clearly knows the process to convert Ifc to glTF (JSON) plus semantics (JSON)

However, we need to think about direct conversion to glTF, WebGL and semantics, all JSON based (plus Binary based too)

---

Dion, another request: I think BIMTester is really interesting, but currently it's hard to work with it, especially for users who don't know programming.
So, is it possible redesign it to be as easy as possible to work with?
User can check the Ifc file, and it provides all issues in one place "as a result window" and user be able to edit and solve issues directly?

I think I requested this before here

[Moult]

@aothms can you elaborate on the process to generate a gltf file for bimsurfer? I experimented with a basic cube exported from Blender and it seems like it might need a specific structure, and perhaps be paired up with an XML created by IfcConvert from a first glance.

Is there any benefit to using bimsurfer, based on xeogl as opposed to the other engines like three.js, babylonjs, cesium, a-frame, etc directly, or just even use xeogl directly? I assume bimsurfer is a thin wrapper which adds 1) BIMserver integration, 2) a UI, and 3) IFC data interaction based on the IfcConvert XML export. If my usecase does not require BIMserver, and we want to heavily customise and rebuild the UI anyway, what other advantage does it provide? Perhaps we can just use one of the engines directly? Or is there other magic on top? For example, I know that xeokit's viewer has a sectioning tool, which is extremely useful.

Is there any support for glb?

@EAzari, your request for improving BIMTester's usability is part of item number 4 in the original post.

[EAzari]

MVD/IFC --> glTF/WebGL (JSON) + 3D Tiles (has limitation) + Semantics (JSON) --> An interactive viewer (we should choose the viewer carefully)

It's hard to understand what's "quickly generate and manage unit tests"

I mean when we import an .ifc file, it tests the whole file and if it has issues opens a new window and lists all issues, so user can solve those issues directly until all issues solves and file opens in Blender

For instance window shows this: (based on color legend)

#14=IfcPerson($,'Azari','Ehsan',$,$,$,$,$)

WARNING: An email address was not found for #14=IfcPerson($,'Azari','Ehsan',$,$,$,$,$)
WARNING: A phone was not found for #14=IfcPerson($,'Azari','Ehsan',$,$,$,$,$)
...

Red = Error
Orange = Preferred
Green = Optional

I click on each issue and solve it directly, for instance fill out email address, phone, or select from Enums, etc

[aothms]

Is there any benefit to using bimsurfer, based on xeogl as opposed to the other engines like three.js, babylonjs, cesium, a-frame

I still need to commit this, but one advantage is exactly the ability to choose a rendering back-end and have a uniform API. For now what's implemented is ThreeJS, xeogl and SVG and have selection synced across these backends. I'll see if I can finally get around to committing this this week.

Other advantages is indeed the pairing with IfcConvert's XML to have semantics (tree and metadata).

But sure it's all pretty basic.

[Moult]

@aothms sorry if I missed something obvious, but is there a specific procedure required to create the glTF file for bimsurfer v2? Is there a special converter? I assume the XML is just created with IfcConvert out of the box with no special settings.

Also, is there a precompiled version of the IfcConvert fork which has the HDF5 patches? Are there plans to merge these patches into the main repo?

[aothms]

The current version of bimsurfer v2 only takes a gltf v1. I'll push a version later that can take the binary glb v2 from IfcConvert directly.

Also, is there a precompiled version of the IfcConvert fork which has the HDF5 patches?

Windows only apparently.

https://github.com/ISBE-TUe/IfcOpenShell-HDF5/releases

But there is an old nix build script here

https://github.com/aothms/hdf5_bim_sparql_query/blob/master/run.sh#L80

Are there plans to merge these patches into the main repo?

Tbh I don't have direct plans for it. It's not overly hard or anything, but in IfcConvert only writing was implemented, reading happened in Python for the experiments. Write-only is not overly useful. To support reading would require some thinking of a good abstract virtual interface on top to support the multiple serializations. I don't think I'll find the time for that any time soon.

[Moult]

That's great @aothms ! I'd love to test the version that takes a glb from IfcConvert when it's ready, as I'm really on the lookout for a good open-source web IFC viewer.

I tested out the HDF5 conversion and though it worked on a small file, it crashed on Windows on a larger file. Not entirely sure why and I don't have any debug information. I managed to poke through the HDF5 files with h5py but found some of the referencing a bit strange. E.g a reference to an OwnerHistory might be (30, 0), which I'd have to do something like file['population'][list[file['population'].keys()][30]] just to resolve the 30, since h5py groups can't be sliced. Perhaps I missed a more direct way of finding what a reference was pointing to. Not sure if this overhead of converting to lists is going to complicate writing larger queries, or in general if hdf5 is good for writing queries.

I'd love to hear the results of your presentation in Oslo about HDF5, and if you'd like I'd be more than happy to do testing to ensure HDF5 is an appropriate solution to take as-built IFC files + point scans and place them in data warehouses. For now, I'm going to park the issue though, as the tools are still a bit experimental.

[aothms]

it crashed on Windows on a larger file

IIRC it might be a 32bit executable making it run out of memory rather soon. The current procedure is not very memory friendly.

just to resolve the 30

On the population group there is an attribute to resolve this. https://github.com/ISBE-TUe/IfcOpenShell-HDF5/blob/hdf5/src/ifcparse/IfcHdf5File.cpp#L1856 h5 attribute values are mapped to numpy arrays so you can slice and index them as you wish.

as the tools are still a bit experimental.

They are. Let's await the presentation and see if there are some opportunities for funded work on this.

[aothms]

That's great @aothms ! I'd love to test the version that takes a glb from IfcConvert when it's ready

@Moult I just committed a ThreeJS version that can take a glb https://github.com/AECgeeks/BIMsurfer2/ let me know if you have questions. It's a bit early stil. You can add engine: 'threejs' to the BIMSurfer constructor.

[EAzari]

@aothms Have you seen this?

https://github.com/playcanvas/engine

https://terria.io

Also:

https://github.com/Soft8Soft/verge3d-blender-addon

https://github.com/KhronosGroup/glTF-Blender-IO

Thoughts?

[aothms]

@EAzari haven't kept up with any of them. Good that effort seems to consolidate around Blender and glTF but otherwise no thoughts.

[htlcnn]

https://github.com/Gaia3D/mago3d
I have just stumbled upon this project. It seems to serve some steps of the workflow.

[Moult]

Thanks @htlcnn ! I had seen it before but forgot about it! I've added it to the list here: https://wiki.osarch.org/index.php?title=AEC_Free_Software_directory#GIS

[EAzari]

Hi, hope this two help with:

1. Finally, bSI has focused on JSON: https://github.com/IFCJSON-Team
2. IFC2B3DM which still has some obstacles, it converts geometric models to OBJ and then B3DM, hope someday someone works on current approach IfcOpenShell follows: https://github.com/Erfan-Shooraj/ifc2b3dm

[Moult]

Closing the bug, as this is superseded by the wiki roadmap entry: https://wiki.osarch.org/index.php?title=BlenderBIM_Add-on_Roadmap