Ifcopenshell: [Question] Why is geometry creation brep beam slower?

[FreakTheMighty]

This may be obvious, but I was a little surprised by how much slower geometry creation is on large IFC than on small IFC. I expected that the total conversion time would be greater, but I'm finding that each individual object task many times longer to convert.

• On an IFC that is tens of MB, creating an IFC beam take thousands of a second to create.
• On an IFC that is 360MB, each beam takes about 2 seconds to create

---

This is mainly tested through the python interface, on 0.6.0. I've tried both the iterator and create_shape with similar results. Finally, I've also seen this behavior of IfcCovert although I can't time the individual shape creations.

Do you have any thoughts about the underlying cause?

[Moult]

@FreakTheMighty just a check - is the beam identical in the small file vs the large file?

[FreakTheMighty]

@Moult I'm pretty sure there are many duplicate beams, but can you suggest a way to confirm this? That is, they look the same, but I suppose there's a chance they are somehow slightly different.

With that said, if I take only the first iteration and compare it to the create_shape, the difference is pretty small within each model. That is, even when were comparing un-cached geometry to un-cached geometry, there seems to be a pretty massive performance difference.

Small Model, First Iteration

Iterator

GlobalId 15VblFIVnCSQ5QHjNXWfIC
Seconds 0.03679509999999997

create_shape

GlobalId 15VblFIVnCSQ5QHjNXWfIC
Seconds 0.03940960000000038

Big Model, First Iteration

Iterator

GlobalId 1BpsA7udj5hP4Krbh94MYM
Seconds 1.3049752000000012

create_shape

GlobalId 1BpsA7udj5hP4Krbh94MYM
Seconds 1.2521649000000004

---

This isn't a perfect test. Both elements are IfcBeams, but they are not the exact same. I don't think either are terribly complicated, but maybe there's some crucial difference. I'll work on getting some sample data that I can share.

I think an ideal test would two IFCs with only beams in it, one with just a few and another with ton.

[FreakTheMighty]

I've reproduced the performance issue with a very simple scene. . . My assumption that the performance was related to file size was wrong. Instead it appears to be related to the IfcBeam representation being a Curve2D in one, and a BRep in the other. I've attached sample files. The zip contains a "fast" and "slow" IFC.

The slow IFC takes 2 seconds to convert a single beam, the fast IFC takes 0 seconds.

ifc.zip

---

Can anyone provide tips on how to deal with this? Is this an issue we could deal with on export? Is there any conversion option that I can explore for improving the speed? 😓 at this speed, it will take an hour a half just to process the beams.

[aothms]

Ah good call. Just Curve2D would not be processed by default. Default output only includes surface or solid. Is that what explains the difference? Or is there output visible?

Generally Extrusion is much faster to process than BRep. This is a bit counter intuitive, but BRep in IFC is very inefficient, so much more data to process. And the extrusion operation is much simpler than all the plane fitting going on in case of explicit BReps.

This is mostly a known issue. Handling explicit BReps is just slow because of the generic handling in OCCTs data model. Mesh based implementations will be much faster in this case.

[FreakTheMighty]

@aothms I’ve done more digging and have found that each beam is an IfcFacetedBrep. I think that these representations should receive special handling. Rather than being treated as a general brep, the shape can be created by tessellating each face related to the IfcFacretedBrep. In python I can construct a mesh from the brep faces in hundreds of a second. I’m sure these would be even fast in C++.

Unfortunately I still need to reconstruct the transform and material (without calling create_shape), which of course is not very DRY, it’s reproducing work you’ve done in the core library.

I think now that I have a better understanding of the performance problem, it deserves a new issue as this one really misrepresented the problem.

[aothms]

The reason we haven't advanced yet on this front is we need (a) support for triangulating faces (with holes) (b) these breps may take part in Boolean operations because of the bottom up processing ifcopenshell we don't really detect that at the point of shape construction (c) the 0.7.0 branch has an alternative implementation in CGAL https://github.com/aothms/IfcOpenShell/tree/v0.7.0-rewrite it's not the fastest solution out there either (we use it's arbitrary precision number type implemented in software for robustness) but will likely still be faster in this case.

[Moult]

@aothms a thought - Blender handles meshes well. If we can detect that a shape does not have any booleans, is it therefore possible to just read the IfcFacetedBrep directly without any OpenCascade / IfcOpenShell parsing? This would be a lot faster, and Blender loves meshes.

[aothms]

@Moult I think so. Assuming we can use something like tesselate_polygon() in case there are holes https://docs.blender.org/api/current/mathutils.geometry.html#mathutils.geometry.tessellate_polygon. Would be good though if we have something for

transform and material

as @FreakTheMighty mentioned. Although the latter is virtually impossible in case of styles assigned to the faces or shells. Perhaps introduce a setting like SKIP_EXPLICIT_MESHES in which case create_shape() or the iterator returns an element with empty verts/faces/normals but with the transform matrix and in case there is an associated material to the product that material is in the materials list (but material_ids empty).

[FreakTheMighty]

@Moult @aothms this sounds promising! I'm using trimesh + shapely and have some success handling IfcFaceOuterBounds. I even think there's a chance I could even handle IfcFaceBounds as shapely provides some boolean operations. So, while I look forward to finding a native solution in 0.7.0, I think it makes sense to see if there are ways to make the current API more flexible for these sort of home brew geometry scenarios.

To that end, @aothms, your suggestion for skipping meshes sounds good. I realize this _may_ be more work, but I like the idea of new methods like get_transform and get_material for a given product. It achieves the same end, allow a calling library to get one piece of data without getting the other, but moves towards API calls with fewer responsibilities.

[Moult]

A small note that I did a quick test to support faceted breps (only outer bounds right now) to see what would happen: https://github.com/IfcOpenShell/IfcOpenShell/commit/76e7205692b908187b31320bed72dadb61ef9dde

The result is actually slower than just letting IfcOpenShell.geom do the shape creation. On the plus side, however, we do preserve the ngons which is a good thing. Note that some of the speed difference (if not all) can be perhaps attributed to Python vs C++.

I haven't optimised the Python code yet - using Python lists and looping through the polys as I have is no doubt very, very slow. Maybe with some optimisation it will become comparable.

[aothms]

The result is actually slower

This is highly surprising to me. Maybe I'm wrong all along and OCCT is not the bottleneck. Is it my parser? It hasn't really been touched since eight years and the lazy on-demand evaluation of instance attributes perhaps needs to be reconsidered.

Doing face.Bounds[0].Bound.Polygon twice is definitely wasteful: there is a lot of pinging back and forth happening between Py and C++ in such as statement especially aggregates of instance references are slow to eval in ifopsh (again, lazy on-demand evaluation).

Did you measure relative time in bmesh_from_pydata vs the rest? Perhaps it's slower for non-triangular data?

How much slower is the in-python procedure compared to C++ + OCCT?

Perhaps ifcopenshell.file.traverse() might help, folding more of the py c++ interaction in a single statement. Filter on cartesianpoint and get a sequential dense list of points to index. Use numpy to vectorize the scale multiplication. But all of this is incremental only of course.

[FreakTheMighty]

@Moult @aothms I don't think the parsing is slow. I was able to get a speed up using IfcOpenShell for file parsing and TriMesh and mapbox_earcut for triangulation with holes.

[FreakTheMighty]

@Moult @aothms I think I found a scenario where my internal geometry creation is slower for faceted breps than using the internal engine. My profiling seems to confirms @aothms 's hunch that parsing, and getting the data to python becomes the bottle neck. Attached is a snippet of the profiler with the last two columns being Time (ms) | Own Time (ms) respectively.

It seems that in this case, the file manipulation, looping through each representation, each fbrep, each polygon and getting all points, is expensive enough to make my custom geometry engine not worth the effort.

| IfcOpenShell | Custom Geom Engine |
| ------------- | ------------- |
| 32 seconds | 47 seconds |

I have tried using traverse , but haven't found a significant speed up. I'm also pretty sure that I'm not re-accessing attributes, so I think the low-hanging optimizations might already be used. I'll update you all if I find something new.

[FreakTheMighty]

I'm looking a bit more into your traverse suggestion

Perhaps ifcopenshell.file.traverse() might help, folding more of the py c++ interaction in a single statement. Filter on cartesianpoint and get a sequential dense list of points to index. Use numpy to vectorize the scale multiplication. But all of this is incremental only of course.

One thing I can't quite makes sense of is how I would group cartesianpoints by the particular bounds they belong to. Is idea that the list returned by traverse encodes that relationship in its ordering?

[aothms]

My profiling seems to confirms

Just to confirm. You also observed this without profiler attached / instrumented code? As it can make a huge difference.

One thing I can't quite makes sense of is how I would group cartesianpoints by the particular bounds they belong to. Is idea that the list returned by traverse encodes that relationship in its ordering?

It should. It contains duplicates and is in order. So maybe grouping them by length of the bounds is quickest. That's in theory though as the Python interpreter can't optimize len(inst.Polygon) in a meaningful way as the C++ compiler would be able to. I don't know, maybe somebody just needs to implement a direct IfcOpenShell-numpy interface or sth for this to be really effective.

[FreakTheMighty]

I went down the path of using their order, but there are scenarios where this doesn’t hold. Nonetheless, I think I found that traverse was faster to collect the large number of entities - if only I could also get the relationships. My sense though is that it isn’t just getting the polygon that is slow, it’s the very large number of __getattr__ calls. In my case there are a vast number of faces.

I feel pretty confident that the profiler was showing fairly accurate results, my total times don’t change much when running without the profiler.

With all that said, I spent a bunch of time tuning our custom brep generation and got things close to back to “native” performance for the case that was previously slower. I guess my main finding was that I can’t really squeeze our more performance from parsing within python.

[aothms]

Maybe what needs to happen then is a native implementation of the ifcopenshell.entity_instance.get_info(recursive=True ) call? That would keep all information nicely in a tree in one Pythonic object, so no further pinging back to C++ necessary. You can also check if the current get_info(recursive=True) performs well enough, but I guess it doesn't.

[aothms]

Ok, as promised a C++ implementation of ifcopenshell.entity_instance.get_info(recursive=True). Maybe that helps.

import ifcopenshell, pprint
f = ifcopenshell.open(r"duplex.ifc")
inst = f.by_type("IfcConnectedFaceSet")[0]
pprint.pprint(inst.get_info_2(recursive=True))

{'CfsFaces': ({'Bounds': ({'Bound': {'Polygon': ({'Coordinates': (0.01904999999999946,
                                                                  0.0,
                                                                  0.580949999999934),
                                                  'id': 759,
                                                  'type': 'IfcCartesianPoint'},
                                                 {'Coordinates': (0.01904999999999946,
                                                                  0.0,
                                                                  0.0190499999999242),
                                                  'id': 705,
                                                  'type': 'IfcCartesianPoint'},
                                                 {'Coordinates': (0.98095,
                                                                  0.0,
                                                                  0.0190499999999242),
                                                  'id': 785,
                                                  'type': 'IfcCartesianPoint'},
                                                 {'Coordinates': (0.98095,
                                                                  0.0,
                                                                  0.580949999999934),
                                                  'id': 822,
                                                  'type': 'IfcCartesianPoint'}),
                                     'id': 21534,
                                     'type': 'IfcPolyLoop'},
                           'Orientation': True,
                           'id': 15388,
                           'type': 'IfcFaceOuterBound'},),
               'id': 10968,
               'type': 'IfcFace'},
              {'Bounds': ({'Bound': {'Polygon': ({'Coordinates': (0.01904999999999946,
                                                                  0.01905,
                                                                  0.580949999999934),
                                                  'id': 3476,
                                                  'type': 'IfcCartesianPoint'},
                                                 {'Coordinates': (0.01904999999999946,
                                                                  0.01905,
                                                                  0.0190499999999242),
...

I'll start a new build end of day.

[FreakTheMighty]

Doesn't seem like its built yet and I'm having some issues with our local build. I'll test as soon as I get one of these to go through.

[aothms]

Right, the new automated build system is still a bit buggy. Just restarted.

[aothms]

The builds are up https://github.com/IfcOpenBot/IfcOpenShell/commit/e1b68dafa469e1f7549f4df500266075db2030a0#commitcomment-45887584

[FreakTheMighty]

@aothms my initial tests show a 2 to 3 times speed up using this new method! That's incredible. This build, however, is seg-faulting on some objects when using the IfcOpenShell geometry engine with the iterator. I'll try to get you more details on the type, and if possible examples, of objects that are failing on tomorrow.

[aothms]

Oh cool! What commit were you previously on? I don't recall drastic geometry changes lately. Hope it's not related to the changes in the build system.

[FreakTheMighty]

We were on c15fdc7. I’m pretty sure the regression isn’t related to the build server. A colleague reported a segfault to me earlier, maybe last week. Apologies for not submitting an issue earlier.

[Moult]

@FreakTheMighty do you have some code to share by any chance of how you use the new get_info, and whether you wrote some code to exclude them from the geometry iterator? I might embark on implementing this in the Blender world :) It may be a bit of work, though, since things like styles will need to be considered.

[FreakTheMighty]

@Moult which parts of code would be most useful to you? Its integrated into a larger project, but I may be able to extract pieces. Also, please note, I got hung up on the segfault, so I paused on this improvement. The work is hanging out in a branch, but hasn't yet been integrated.

With regards to the filter, I have a list of representation types that we process with our internal engine, everything else uses the iterator. Here's how it works:

1. Get a list of all IfcProducts
2. Filter those into two lists, internal engine and IfcOpenShell engine. Our internal engine handles IfcTriangulatedFaceSet, IfcFacetedBrep and IfcFaceBasedSurfaceModel
   

   3. Pass a list of GlobalId into the IfcOpenShell iterator indicating which products it should process

Materials have been one of the more challenging bits.

[Moult]

So I had a shot at implementing this for faceted breps (using mathutils.geometry.tessellate_polygon) and got mixed results.

• For a single object of some high polygon monkeys, I see crazy good improvements (1 second with iterator vs 0.1seconds with get_info_2). Filesize 420kb. Yay!
• With another file where it has about 12.5k element (which use faceted breps) and I know it has some which have something like 61,000 triangles per faceted brep, get_info_2 loads it in about 120 seconds, whereas the iterator will take so long, I just give up waiting. Filesize 42mb. (Some mistakes with transformations, but I assume I can find out what I missed.) Yay!
• Another file with about 19k elements which use faceted breps however (but doesn't have crazy big meshes (everything under 5000 polys) takes 300 seconds with get_info_2... but takes only 175 seconds with the iterator... so the get_info_2 method is significantly slower!

A little lost here and not even sure if my methodology is correct - it's similar to yours @FreakTheMighty - but the devil is in the details I guess with looping through the get_info_2() results, how it keeps track of the transformation for each representation (which may be mapped), and also how to add styles.

I'd love to have a code review session with either of you @FreakTheMighty @aothms if you're interested to see if you can spot what I'm doing which is clearly wrong - because if this does get resolved, it'd be a huge win for end-users. Interested?

I'm also still curious about how much this could be done in C++ - maybe if we did the element filtering outside C++, the C++ could still be multithreaded and return geometry.{verts,edges,faces} ... hmmm. There's a lot of room for error for reimplementing all the matrix transformations outside C++ (maybe tessellate_polygon can be reimplemented in C++?) ... and it would benefit FreeCAD too (they're now heavily using the geom iterator).

[FreakTheMighty]

@Moult that all sounds right to me. I'm not sure I've done as much direct comparisons between the iterator and the python implementation. For my use case, the potential for sometimes having a 2X slow down by using python greatly out weights the potential for an eternal freeze 🧊 while in the iterator.

Nonetheless, I have done lots of profiling on my python implementation and keep finding ways to make it slightly faster. I'm sure your profile will be unique to your implementation, but here are the types of optimizations I've done.

• Lots of caching to ensure that if a facet, or product has been generated I don't regenerate it
• I found ways to reduce expense I had when mapping facet vertexes to a planar surface

With regards to what could be moved into C++. I feel like there's agreement that basically all of it should be :), but that there are some challenges and work to be done to get there. I don't think the element filtering has been the slow part for me. I do think, however, that some smaller function that calculate transforms and materials in C++ would reduce the likelihood of error when doing these implementations in python. I've had to re-implement material code in python, which isn't very DRY, and has resulted in bugs that I've slowly addressed.

[FreakTheMighty]

Oh, and I'd be happy to do a code review with you.

[Moult]

@FreakTheMighty I guess part of it would be to understand some easy ways to optimise the code, profiling techniques you use, as well as overall strategy (whether my approach to getting representations and transformation matrices are correct).

I'm available pretty much anytime during Sydney Australia waking hours with a bit of heads up. We can screenshare online if you ping me on the OSArch.org live chat (##architect channel on freenode IRC / matrix). Feel free to ping me anytime or let me know a time / day that works best for you.

[FreakTheMighty]

@Moult I'll send a calendar invite to the email listed on your github profile.

[Moult]

@FreakTheMighty awesome! I've accepted it, but might be 20 minutes late.

[Moult]

For the record we didn't get around to profiling, but it seems as though both @FreakTheMighty and I are using basically the same approach, so that's a good start. We'll organise another time to meetup and go through the profiling exercise and hopefully something comes out of that.

[Moult]

Also @aothms a thought, is the reason the processing is so slow because it is attempting to resolve a manifold, closed, mesh? If so, perhaps it should default to a (potentially) non-manifold soup of verts/edges/faces (i.e. merely a reformatting of the source faceted brep / triangles / tessellation), and when it detects a boolean, then it backtracks and recalculates the shape? Totally guessing here...

[aothms]

is so slow because it is attempting to resolve a manifold, closed, mesh?

Not only, for facetedbrep we have the faceset_helper that makes this more efficient, by associating undirected edges to pairs of cartesian point the more expensive generic OCCT topologic lookup is eliminated.

For tesselated/triangulated faceset we currently don't have this enabled yet, so the OCCT sewing is in place which is expensive.

So it's mostly actually the OCCT BRep datamodel which is just simply heavy for storing meshes. Calculating edge curves, parametrizing the vertices onto the curves etc. And then meshing that again :(

then it backtracks and recalculates the shape?

Yes, we kind of have this with ensure_fit_for_subtraction where we on demand convert the Compound of Face to a manifold Shell, but as said, it's basically from before we had faceset_helper

The only significant way out is to have hybrid geometry libraries (which is what we're aiming for). So have a really light weight geometry library that handles the easy cases which falls back to OCCT/CGAL for the tough ones.

[Moult]

Had a closer look at my implementation and ignoring optimisation for the time being, I've made mistakes related to determining when I need to recalculate a mesh and when I can reuse it. I guess I really need to check every mapped_item in the tree and check that the accumulative transform matrix indeed ends up being something I've previously used, or whether it's something new. Not really a fun exercise and prone to error.

Kinda stuck right now. Making it more correct would make it slower as it has to check more things, and it's already slower than C++ in the majority of situations... my current theory after further testing is:

1. When the file has a crazy mesh outliers that causes IfcOpenShell's process to go on and on, Python is faster.
2. When the file has a high ratio of high poly meshes to object count, Python is faster.
3. When the file has the opposite ratio, i.e. mostly low poly meshes, high object count, C++ is faster. Presumably due to multicore support as well.

Other things I have tried: numpy arrays for collecting coordinate xyzs - seems slower, in fact. Removing vertex welding: slight speedup, but clearly not the bottleneck. Took a look at numba but not 100% convinced as I still need to go back and forth between the Blender tessellate function...

[aothms]

I've overhauled the handling of the tesselated items to use the same mechanism as we have for the old ifc2x3 connected faceset. It'll be more efficient [don't expect miracles though, for detailed meshes there is still the price we have to pay for using a heavy brep modeller] and robust (some topology checks before creating shapes and more). It needs to be properly tested though. It's a fundamental change on common representation types.

cf68cf6f5c380fdfcbd93371c42fab466f31c7ca
3f9e5ca806f06cf9507975f655f21f2c21fdd0a7