Minetest: Add support/API for mods written in C++

[adamryczkowski]

I think it shouldn't be too difficult to add some rudimentary support for add-on modules, that work the same way as kernel modules on Linux:

• Adding/removing the module (i.e. C++ written mod) would be possible to be done in-game, via menu.
• The module would need to be complied against the minetest's source code and would produce a shared library (.dll / .so) or downloaded in binary form.
• The module would need to conform to some API, that would be similar (or more powerful ?) to the already existing API for LUA mods.

The benefits, I think, are obvious:

• The modding community would be able to work on things, that now would require engine improvements. (Many things modders want to do are just too inefficient in LUA, which spur discussions about feature requests that don't really add extra functionality to the engine, but offloads mod- or game-specific tasks to the engine).
• Ability to write mods in in strong typed environment. No more nil crashes when you mistyped a variable.
• Easier learning curve from modder to engine developer, that would potentially bring more features to the engine (I personally fancy about dynamic lighting coming to life).

Edit:

It might be better (more compatible) to let the interface be in C. But then we sacrifice expressiveness of C++.

[SmallJoker]

The idea is good but it would cause a much higher work to check and maintain the compatibility with all by Minetest supported platforms. In addition, LuaJIT is not slow, so writing mods in C(++) will not give a very huge performance increase.

[adamryczkowski]

I agree about your point about speed. That's why I didn't add "speed" as a benefit for adding support for C++ mods.

Could you please elaborate more about your point about the amount of work with checking and maintaining the compatibility?

[nerzhul]

@SmallJoker lua 5.3 is not slow and is a good replacement for luajit

[lisacvuk]

Wouldn't we loose portability if we start writing mods in C++? Modder would need to compile mods for both windows and linux, or provide source code and hope that the user can compile it on his own.

[adamryczkowski]

@lisacvuk Yes, but for these very reasons C++ mods will be written only by those, who really need advanced features, like e.g. multi threading or interaction with the system (make blocks represent real-world objects, like files or appliances, like Arduinos), or use any numeric/simulation library they want, since C++ can link with virtually any other language in the world.

[SmallJoker]

@nerzhul That's good to hear but currently not the main topic here.

Could you please elaborate more about your point about the amount of work with checking and maintaining the compatibility?

With this I mean checks on all platforms to see if the mod even works there.

@lisacvuk Yes, it would require to pre-build them because not everybody has a compiler installed on their system.

[adamryczkowski]

@SmallJoker I Agree with you. When/if this feature is ready, we could address the problem of slipstreaming the process of installation normal and C++ mods with some sort of add-on manager. But that is a different topic.

[Zeno-]

_If_ this happened I'd be more supportive of a C interface/API (somebody else could make a C++ wrapper) rather than C++. That at least would allow both C and C++ "mods"

[sfan5]

There are two ways this can be done:

• 1) Link mods to the entirety of minetest and allow modders to modify everything they want

This method has the downside of being very incompatible. Mods will probably need updating everytime some changes are made to the engine, which means that modding compatibility will be as unflexible as in Minecraft. The upside is obviously that modders can really change everything in the engine/game.

• 2) Have a C++ API that is just like the Lua API

This method would be way more compatible but wouldn't allow modders to do what they want. As said above it would just be the Lua API but with C++ instead. (Which doesn't have that many advantages)

Generally adding a C++ API is not easy and will probably take some time.

[adamryczkowski]

I agree with @Zeno. C might be better. I'll redact the issue subject.

[adamryczkowski]

@sfan5

Except for the problems with maintainability I think the best approach would be to publish _different_, but roughly equivalent API for C++ users. The API would use all C++ features for better expressiveness.

But in reality, C API might be good enough.

Maybe it is possible to write C++ templates, that would automatically ensure the same interface for both C and Lua. At least I can see how it can be done in D (another programming language).

[neoascetic]

Oh boy, let's implement good Lua API at least and then extend with something harder to implement, such as C++ API.

[C1ffisme]

If we do make an API for C++ mods, can we make one that doesn't invalidate the lua API? Some of us are very poor at coding at all in a compiled language.

[sofar]

@C1ffisme I don't think that such a design would be acceptable (having it invalidate the lua API).

Having 2 API's will undoubtably fail as it will be almost impossible to have mods cooperate like they do right now.

[adamryczkowski]

@sofar I agree, I didn't think about it. But thinking again, I came to conclusion that what we really need is a tool that formally defines mods' API, just like .def file defines dlls' exports. This way it wouldn't really matter in what language a mod is written and we would have perfect mod cooperation, and no more mod mutual incompatibilities.

...

and that brings us to the point, which led Microsoft to devise COM in 1992... and that might be even more "non-trivial" than it is already.

I'll sleep over this subject, and maybe I come to something doable.

[sofar]

@adamryczkowski that would effectively break all the existing Lua mods.

I think this is dead to begin with. If you would introduce this to do something highly specific that doesn't need to interoperate with existing mods, then I could maybe see it work, but you're effectively planning to duplicate the entire Lua API with the promise of making it an equal implementation.

[ShadowNinja]

@Zeno-: C isn't a very practical option, since you'd have to, eg, create server_* functions instead of just passing around a Server object. This would be a lot of work.

A C++ API that simply provides a way for code to hook into the existing codebase would be fine, but mods wouldn't be able to do much because the engine isn't really designed to be modular.

[Ferk]

Is there any real purpose for this?

For anything specialized you can already use specific libraries from lua that can depend on external programs to do any heavy lifting... see RealTerrain mod with its use of imageMagick / MagickWand / imlib2 for image processing.

It's better to find the particular problem, without mentioning any specific technology (like "C/C++"). Don't assume you already know the solution, if what you want is performance for a particular task, then explain what task and why do you think it's too slow. Then people may come up with ways to make it faster... like adding a new API or maybe some other better solution.

Imho, it should be more important to make the current engine more modular, well documented and easier to work with than to add a C API.

[t4im]

Consider, that a JIT like LuaJIT can do runtime optimizations, that an AoT compiler cannot.
Things like loop-unfolding and code-inlining across mods and builtin won't happen with conventional native binaries, because the relationships are unknown at compilation time. But with a capable JIT they will. These are exactly the few situations in which a JIT (or any runtime optimization, really) could outperform equivalent native code.

Also consider the security problems when distributing unreviewable native code, that can easily be patched during transfer, as opposed to reviewable code, somewhat protected by git's merkle-tree.

[numberZero]

Native server-side plugins may be very helpful, if they expose some API to Lua.

compatibility with all by Minetest supported platforms.

If the plugin is limited to MT API, it doesn’t depend on the OS. So one image per architecture, not per platform (arch+OS), is possible (but requires a loader to be built into the MT core). And as long as you don’t mess with pointer arithmetic, compatibility problems are rare and usually (IMO) limited to various IO usage.

But note that while I like the idea of C++ API very much, it has a downside of that its ABI is not even nearly as flexible as the API. That may cause some trouble.

[numberZero]

Client-side plugins may be useful as well, for something more or less heavy. While I don’t like the idea of embedding a web browser into MT, there may be some better usage. But that may need more flexible network protocol to cooperate with server-side mods and plugins.

[Cazadorro]

I would suggest if you are going to do this, do C API, not C++, because of the ABI incompatibility issues it means any one not developing in the same compiler and version you used to generate minetest would potentially not be able to link to it. A C api would also allow other languages to be able to interact with minetest, and you could potentially have the lua api sit on top of the c api instead, potentially allowing full LUA, C-api compatibility, and compatibility with any language that can utilize the c-api.

[benrob0329]

I would much rather have changes be made to the core, rather than servers and mods requiring binary blobs.

[Ferk]

one image per architecture, not per platform (arch+OS), is possible

With time and effort you might find a hacky way to circumvent the OS-dependent binary loading (ELF in Linux, PE in Windows and Mach-O in Apple), but an i386-compatible processor cannot execute ARM instructions and vice-versa.

[paramat]

Closing due to disapproval but feel free to continue discussion.

[nerzhul]

i re-open the issue as i will start soon to work on a similar thing for my own needs, but it's not intended to be hosted by contentdb or something like this :)
I will start to work on an abstract for the callbacks first, then we will see what can we do

[nerzhul]

i think we can make lua api and c++ api plugins on the core engine with a central router

[sfan5]

Working on a C++ API will be wasted time until you know exactly in which situations Lua isn't enough.
And if such situations are identified it's better to add a generalized API so a solution can be used from Lua (such as with e.g. pathfinding, raycast, vmanip).

[rubenwardy]

Just because you want something "for your needs" doesn't mean it should happen/be given priority. I support refactors to separate concerns and simplify the server, and to provide isolation between the server and scripting, but something like this should not be the priority.

If the usecases aren't currently compelling enough for upstream, then once you have those refactors it could be easier to maintain a fork for your needs.

Also, what happened to securing the client and working on SSCSM?

[nerzhul]

as you know, we work on part with interest as we are purely FOSS, and on bugs we can fix.
In 1 year of absence nobody take the SSCSM part ? maybe in fact any CSM is pretty useless if nobody propose a PR on it and prefers to focus on server ? Where is the contentdb thing for people who wants to extend (safely for servers owners) their client ?

I'm already working on the abstraction layer, and as a side note, having this split with the abstraction layer should permit any language extension, not only c++ parts. And with C++ parts i will add for my needs i may find interesting ways to offload generics things into core

[nerzhul]

and another problem is that each time i propose you a high level call, which can reduce the overhead, on any tiny part and prevent reinventing the wheel it's refused.
anyway i'm thinking about a minetest-staging like wine-staging for unofficial patches on minetest to add some of those patches for whoever is interested in.

[rubenwardy]

In 1 year of absence nobody take the SSCSM part

Because it needs to discussion and prerequisite work

Where is the contentdb thing for people who wants to extend (safely for servers owners) their client ?

client-side modding is not officially supported, therefore ContentDB doesn't have a type for it

and another problem is that each time i propose you a high level call, which can reduce the overhead, on any tiny part and prevent reinventing the wheel it's refused

I don't recall any examples

[nerzhul]

in 1 year no discussion around it ?
Recently i proposed has_priv high level (and yeah tiny) call, to prevent each mod to write its own bug non-free implementation to verify if a player has a single privilege (instead of loading the whole privilege list and checking inside and returning things we don't care)

[rubenwardy]

Recently i proposed has_priv high level (and yeah tiny) call, to prevent each mod to write its own bug non-free implementation to verify if a player has a single privilege (instead of loading the whole privilege list and checking inside and returning things we don't care)

This issue for that is that privileges is auth data, not player data, and there's already:

if minetest.check_player_privs(player, "priv") then

Not exactly hard to make bug free.

I'm also not strongly enough opposed to reject it, I just call you out on your arguments of performance

[nerzhul]

did you see what this call implies ?
call the core to get privs, which gets privs to lua (fully) and then parse the retrieved lua through cpp through lua privs table and return a table with key = value.
It's not what we intend in terms of good call, sorry :)

[nerzhul]

as auth is in handled by various AuthenticationDatabase backend we can make both calls more efficient by at least remove the lua useless storage and let the core store the privs with the session (like it's done but not in core, in lua), to prevent core to lua callback, and the has_priv will be simpliy a std::set count check which is lightweight compared to lua full table comparison

[nerzhul]

and: don't talk me about LuaJIT, can we rely on a nearly abandonned tech ? (i know we use irrlicht too, but we can modify it and make it evolve for our needs, at least)

[rubenwardy]

It's not a good optimisation because there's no evidence that it's a hot spot, it requires modder changes, and it changes the responsibility of that API

in 1 year no discussion around it ?

I have been placing a higher priority on fixing the GUI

[benrob0329]

This seems really offtopic and better suited to IRC.

---

Looking back on this (I had :+1:'ed it) I think that it will be a waste of time unless the Lua API somehow gets a big refactor and the engine is modularized or something (which is still probably a waste of time since there are bigger issues given the limited dev time MT seems to have).

[nerzhul]

@benrob0329 can you explain what is lua api refactor for you ? As nobody wants to create new high level calls except me in the team ? :)

[nerzhul]

As a side note, this PR is also a good time to see what is our callback list, and see what can we do better inside our API

[paramat]

i re-open the issue as i will start soon to work on a similar thing for my own needs,

I'm already working on the abstraction layer

And with C++ parts i will add for my needs i may find interesting ways to offload generics things into core

Is there is a communication problem here?
"my (own) needs" sounds like you are working on a major MT feature primarily for your own benefit, with little consideration of whether it is generally beneficial for MT users. That would of course be unacceptable.
Also, surely such a major feature should be discussed first and agreed upon? Considering the lack of enthusiasm in this thread, it seems unlikely that other core devs support the idea, so it seems odd that you are already starting work.
Maybe i misunderstand? This comes over as arrogance, and is not how MT development works.

No objection to the issue being reopened, it is supported by a core dev so can remain open.
But i think this needs discussion and agreement first, as it is such a big feature.

For the feature request as initially stated, i am also leaning towards disapproval.

[benrob0329]

@benrob0329 can you explain what is lua api refactor for you ? As nobody wants to create new high level calls except me in the team ? :)

Mainly consistency, and bringing us more up-to-date with current Lua. Supporting require would be nice (I know there was effort for this, don't know if it was merged or not though) and having API versioning so that old mods don't break every time the API gets fixed would be nice. I'd like to see the entity API not be super inconsistent (things like having to get the object only _sometimes_ is really annoying).

Ask experienced modders what bugs them about the API, what common problems they have to work around are, you'll find more than enough to get an idea of what a refactor could look like :-)

Personally I think our graphics engine needs an overhaul moreso than the modding API (bugfixes and API additions, not fancy effects), but that's mostly because I've dealt with it the most in my time as a game-dev.

[nerzhul]

@paramat i have a server, and i'm experimenting things on it, then it's my own needs. No problem with that, and i will retribute a PR which will be discussed until open.
@benrob0329 i agree with your about the graphics but it's out of my skill scope, i'm a backend engineer :).
I think a such feature (like i'm implementing) will enjoy mainly high skilled c++ developers which wants to add more backends, not only c++, but with what i'm implementing they can implement Javascript or Python, or Ruby, or PHP, or any other language binding support on Minetest servers for their own needs (in my case it's just a C++ API)

[sfan5]

Supporting require would be nice

The PR you are thinking of is #7621

having API versioning so that old mods don't break every time the API gets fixed

As usual the engine always provides full API compatibility. Exceptions to this are undocumented features or when old features are removed (rarely).

Do you have a particular breakage in mind with this suggestion?

[benrob0329]

Do you have a particular breakage in mind with this suggestion?

If you loaded up a mod from 2013, there's a pretty good chance it just wouldn't work in current MT. Now, this is despite the supposed "compatibility" that the API is supposed to keep. Functions get removed, arguments get changed, MT still breaks it's API and has no way around that other than to update the mod/game. But despite that, we still can't have many quality of life improvements because they break the API.

[sfan5]

Functions get removed, arguments get changed

Can you name some instance(s) of that happening?
I agree with your complaints in principle, but "just" adding API versioning without a clear goal doesn't make sense.

we still can't have many quality of life improvements because they break the API

Is that so? I'm not aware of any PRs that have had this problem, usually when larger changes are made a new slightly different API function is added.

[nerzhul]

maybe @benrob0329 refers the various PR we add recently which break mods just because we add security against bad practice, it's not the first time, around 5.0 we did a such thing too if i remember. An API have a lifetime and should have a support contract, we should for example define API is stable for 2 years, and we warn 2 versions before about a breakage (which mean 1 year in terms of MT lifetime), but provide the compat tools. Currently we are trying to do our best but it's not perfect

[IvanTurgenev]

i would get rid of lua and make mods compiled c++, its 2020 we got docker etc compiling stuff is easy, besides 99% of mods are used serverside, and performance is crucial, making a fun and interesting server needs a ton of mods and the overhead get insane

[IvanTurgenev]

or make a c api for all programming languages

[numberZero]

@IvanTurgenev Compiling is easy. As easy as writing crashy leaky code in C++. Trust me, it’s way easier than in Lua. Take a look at this innocent example: char const *to_c_string(std::string s) { return s.c_str(); } (based on real-world code). Security is relevant too, non-trusted mods run in a restricted environment currently. Sandboxing compiled code is non-trivial (yes there are containers, but you definitely don’t want mods to talk over TCP, that would kill performance again). So no, C++-only is for closed games where single company controls the whole codebase.

But adding C API while keeping Lua (maybe making it a layer over that C API) would be great for those few but often highly popular mods which do need maximum throughput (like MG, Mesecons, Techinc). Other mods would continue using Lua for its simplicity and flexibility.

P.S. As compiling is easy you can build your mods directly into the Minetest code =)

[benrob0329]

If there really was a bottleneck in Lua execution speed for your mod, couldn't you write a Lua module and require() that from a trusted environment?

[Emojigit]

I think can add a function: minetest.do_cpp(cpp_place)
If you place this into init.lua, the game will auto load your cpp.