Ifcopenshell: Decouple data and usecases from BlenderBIM Add-on

[Moult]

As the BlenderBIM Add-on now does native IFC authoring, a portion of its code is 100% agnostic of Blender. I believe there is value in:

1. separating the agnostic code into its own module
2. introducing unit tests to make sure the code does what it does
3. using that code to then generate IFCs which can then form unit tests for whatever else we think is useful instead of having huge test model repos
4. polishing a super duper user friendly api for users over time to manipulate and generate ifcs at a higher abstracted level (that also considers schema differences out of the box) than the current offerings (e.g. pure ifcopenshell, and ifcopenshell utils) - this means refining how these usecases are called and even writing (gasp) documentation!
5. Maybe we can even generate some visual nodes for this. Neat.
6. Some higher level functions are useful in other utilities (e.g. currently ifccsv does its own pset and attribute editing - but with this higher level API it could do it better by considering ownership histories.)

The current system architecture is formed by dividing up the IFC schema into "modules" that delineate logical aspects of an IFC model that you might modify. For example, there is a module that manipulates the psets concept, and this is totally separate from a module that manipulates the classification references concepts.

Each module treats IFC as the source of truth. Two things are provided. There is a "Data" class for each module, which offers an abstraction to load useful information related to that module. It reads data from the IFC, and returns simple Python dictionaries and regular Python data types. The user can then use this as a cache of data, or a way to bulk load useful data without needing much knowledge of the IFC spec.

Each module also provides zero or more usecases. A usecase is a particular scenario where you might want to modify / write data to the IFC. It is higher level as it will take into account things like type conversions, maintaining ownership histories, purging trees as necessary, etc.

Here's a diagram of how the Blender-agnostic "Data" reading and "Usecase" writing concepts (purple) currently interacts with the BlenderBIM Add-on code (in orange)

Purple lines are data that use IfcOpenShell objects. Dark grey lines are data that are plain old Python Objects (POPOs) that aren't magical at all. Orange lines are data that uses Blender-specific data structures. Note how there is no coupling between Data (reading) and Usecase (writing).

Here is an example usecase for editing georeferencing and a data class for georeferencing.

The code is already written, but I want to propose splitting the BlenderBIM Add-on code from this core code, as currently the files are in the same place. Also the imports currently all have "blenderbim" in the namespace which isn't quite semantically correct :) This makes it difficult for a newcomer to simply include this library when they don't care about Blender.

So, this issue is to find out:

1. Is there anyone who thinks this shouldn't be done?
2. Any preferences on how this is bundled? Should it be part of the ifcopenshell python module, or should it be another module that is like an "api" to IfcOpenShell? Maybe import ifcopenapi? :) (literally thought of that as I'm typing this, no ideas yet)
3. Does this sound like an initiative that have some direct impact to something you're working on? Perhaps with the unit tests?

I was inspired to write this since @fbpyr seems to already be doing a lot of magical IFC generation and is using these usecases (currently through Blender operators, but effectively he is using these usecases).

Refactoring this is currently a low priority though, as nobody is jumping up and down for this just yet :) And the code kinda just needs moving and search/replace for imports.

[aothms]

I think this somewhat relates to the "Definitions applying to General Usage" stuff in the docs http://ifc43-docs.standards.buildingsmart.org/IFC/RELEASE/IFC4x3/HTML/lexical/IfcDoor.htm currently these are locked up in some sort of mvdxml / ifcdoc thing. Small subgraphs that relate to a functional component fulfilled by the schema (although to be fair it's actually mostly just markdown snippets).

The main question to ask (and probably the answer is "no"). Should this be implemented in some declarative schema language instead of python code? Conceptually mvdXML could be used, we have a reasonable parser in IfcOpenShell, but I don't think there is a realistic authoring option, it's also not the best basis probably.

Maybe there is some sort of graph language we could use? gremlin comes to mind (https://pypi.org/project/gremlinpython/) or of course SPARQL. Ultimately I think there is value in this as an additional query/projection step to decouple storage. Extend adoption of your usecases outside of python implementations. And perhaps more flexibility later when it comes to implementing the usecases based on different IFC schemas where the projection normalizes the schema differences.

[Moult]

@aothms absolutely, this does relate to those general usage instructions. Without a higher level API, it makes it so much more prone to inconsistent implementations.

Should this be implemented outside Python? In theory yes I would agree. In practice, it probably makes more sense to first do it in Python, as I've already written a ton in Python, and we can battle test it in real life scenarios first, and later on make it more academically correct :)

[Moult]

Ping @brunopostle - who recently sent me an email about porting aspects of Homemaker to IfcOpenShell. This issue is highly relevant as it would cut down a lot of code rework, so perhaps we should do it sooner rather than later. @aothms do you have any namespace preferences?

1. Option 1: move this stuff into src/ifcopenapi so that people can call import ifcopenapi.georeference for example?
2. Option 2: move into src/ifcopenshell-python/ifcopenshell/api/ so it becomes import ifcopenshell.api.georeference (my personal preference - one less moving part, and the only disadvantage is a small filesize increase)

A list of projects who might benefit from this:

• Homemaker
• DXF2IFC
• IFC2CA (and potentially future CA2IFC)
• P62IFC
• MSProject2IFC
• @fbpyr 's Rhino2IFC toys
• IfcCSV
• IfcSverchok
• IfcServer (planned owner history based multi-user system)

[aothms]

I tend to gravitate towards option 2 as well. As I said I really hope for these API-data-definitions to become implementation agnostic at some point. I think option 2 best reflects that. It's IfcOpenShell's implementation of the API and later some group of people can generalize that as part of a different project.

Let me know what I can do to help. I have a few snippets that convert small IFC sub graphs to very readable graphviz hierarchical dot graphs. Maybe that's a nice way of documenting the data definitions of the APIs?

[fbpyr]

I agree with @aothms on preference of option2 , as it reflects its api nature in name. 🙂 (zen of python: explicit better than implicit)

@Moult there are actually even a few more blender addons that I wrote/am writing, that would benefit from it as well.
(namely ifc_connector (fastapi/mongodb/brython webapp for building element data to db connection - currently focuses on rooms) and ifc_void_data (ProvisionForVoid data aggregation, which I still need to adapt to BlenderBIM post-refactor))

On the language: I think having the functionality exposed to Python - being it native or via wrapper -, is a huge plus for the large set of people, who their main or only programming language is Python (which I find fantastic not only for prototyping).

@aothms the ifc-sub-graph-to-graphviz-conversion sounds quite interesting - do you have a snippet that does that, somewhere as a gist by any chance?

[Moult]

@aothms I've done a first pass: https://twitter.com/BlenderBIM/status/1374897927680712707

There are still many things to massage (e.g. a standardised interface for shape representation adding), and of course a ton of docs to write, and overall standardisation of the input settings conventions... but I think this is really exciting! I look forward to when it is possible to define the scope of the usecases (i.e. the subgraphs it will manipulate) programmatically and agnostic of ifcopenshell, and turn it into a standardised API that can turn authoring tools into a client-server model, so that we can handle multi-user editing, version control rollbacks, etc. I'll add it to the todo list to put together a proof of concept of a VCS / multi-user support.

[aothms]

Very cool!

Quick comments:

• Isn't there a way to eliminate all these Usecase() things in the user scripts. I agree it's great how it's modularized, but maybe in the api folder there can be some python magic to import all the Usecases under their module names, like (untested):

for x in os.listdir(os.path.dirname(__file__)):
    mdl = importlib.import_module(x)
    globals[x] = mdl.Usecase

to go from

create_file.Usecase().execute() to create_file().execute

Another way to solve this would be to make api a class so that it has a __getattr__() to lazily import.

• Maybe I'd also eliminate the execute. If you like to have a functor you call create_file_obj instead of create_file.

So ...create_file() and ...create_file_obj().execute() with the same result

• create_product(class=IfcProject) IfcProject is not a 'product'.

• I would suggest keyword args instead of a plain dictionary, so instead of settings={} just **settings. Cleans up the user script

Usecase(f, {"ifc_class": X}) to Usecase(f, ifc_class=X)

• In light of don't touching each other's privates I'd say don't assign to api.owner.settings.settings['person'] but rather have some sort of API call to do that. Or perhaps better don't rely on global state (or have an api object to collect global state more locally). Working on multiple files simultaneously is otherwise virtually impossible with global state.

[Moult]

Agree with all points @aothms - I'll reopen the issue so that its more visible to the public to track these getting resolved. Other known issues is the current coupling to blender in the representation use cases.

I'll try to find a solution without magic too :)

I already moved ownership persons and organisations to a function get persons which can be monkey patched. Not convinced its the best solution either.

[Moult]

BTW @aothms - @FreakTheMighty mentioned how the Usecase execute() function implies the existence of say... an undo() function? :) This is an interesting concept which may help solve some issues of how to handle multi-user version control (local staging + conflict management) as well as managing undos through an decoupled authoring app like Blender / FreeCAD ... I thought it was a great idea to consider! Would be awesome to see how this namespace evolves and matures over time!

[aothms]

Oh indeed, maybe another way to look at that is some sort of transactional API inside the lower-level IfcOpenShell libraries with a rollback option, but that sounds complicated.

[Moult]

Proposed new syntax:

import ifcopenshell.api

# Before vs after
f = ifcopenshell.api.project.create_file.Usecase().execute()
f = ifcopenshell.api.run("project.create_file")

# Before vs after
project = ifcopenshell.api.root.create_product.Usecase(f, {"ifc_class": "IfcProject"}).execute()
project = ifcopenshell.api.run("root.create_entity", f, ifc_class="IfcProject")

With a dedicated run function instead of accessing the class directly, the app can switch between a regular run which operates on the local "f" variable, and a special run which instead posts to a listening daemon / server which will be the first step for multi-user authoring!

[aothms]

Sweet!

To refrain from magic, one other approach I can imagine is to do some sort of code generation during packaging. That we just print our own quick python file with proper function signatures for autocompletion and static typing with e.g mypy

Feel free to assign something to me btw.

[Moult]

Closing as mostly solved, but of course will continually evolve over time :)

[Jesusbill]

f = ifcopenshell.api.run("project.create_file")
project = ifcopenshell.api.run("root.create_entity", f, ifc_class="IfcProject")

Looks Blender-ish :) :+1:

[FreakTheMighty]

At the moment I don’t see a benefit to using a UseCase class rather than a function. It seems to me that a user would be expected to pass whatever reference the function requires, the file, some root entity, etc.

On a related note, what’s the benefit of using a dictionary or the spread operator to pass arguments? The execute method still needs to unpack those, so why not just put them in a function interface. It provides some self documentation and prevents a user from passing unused arguments, too few arguments, etc.

[Moult]

@FreakTheMighty yep :) At the moment there is no benefit to using a class rather than a function. If, however, an undo() is implemented, then depending on the implementation, it may be more beneficial to use a class as it can store its instance-specific "undo" instructions. This is all speculative. Part of the logic behind adding the run() function is so that the end-user doesn't need to care too much about how the actual usecase is structured, so I can change it at a later date. I plan to hold off changing it for a little while until I have a proof of concept of undo, as well as multi-user editing, which will inform what is the best approach.

There is no benefit to using a dictionary. I originally used a dictionary due to habits from other languages which don't have keyword args. The spread operator is also not as good as just using plain old keyword args, but it was easier to do a bulk search replace settings={} with **settings :) In the future, perhaps when these start getting documentation, I do plan to switch to keyword args in the function itself.