Godot: Include libffi

[ghost]

libffi is a cross-platform library which allows calling C (C++ can also be called but in many cases name mangling makes it hard) code at runtime.

Foreign function interfaces are implemented by nearly all dynamic languages (Python implements it in the ctypes module, for example).

This would allow writing wrapper plugins in GDScript directly instead of writing modules in C++, which would make it possible to both use dynamic libraries from GDScript in a cross-platform manner (#3936) and distribute wrappers to libraries directly from the asset store (since a library like GDNet could be written in GDScript instead of C++).

Implementing libffi is extremely trivial, and it could be implemented as a module shipped with Godot by default.

[punto-]

Can this be used to solve the ABI problem? For example come up with a
protocol to pass variants through the interface? On your module you'd have
to use this to call any methods on an Object that you get from the host
engine.. What APIs would be natively available to the module? Variant?

On 5 March 2016 at 21:10, paper-pauper [email protected] wrote:

libffi is a cross-platform library which allows calling C code at runtime.

Foreign function interfaces are implemented by nearly all dynamic
languages (Python implements it in the ctypes module, for example).

This would allow writing wrapper plugins in GDScript directly instead of
writing modules in C++, which would make it possible to both used dynamic
libraries from GDScript in a cross-platform manner (#3936
https://github.com/godotengine/godot/issues/3936) and distribute
wrappers to libraries directly from the asset store (since a library like
GDNet could be written in GDScript instead of C++).

Implementing libffi is extremely trivial, and it could be implemented as a
module shipped with Godot by default.

—
Reply to this email directly or view it on GitHub
https://github.com/godotengine/godot/issues/3943.

[est31]

Yes it can solve the ABI problem, but it doesnt have many advantages over simple C interfaces.

In fact these are all types possible with libffi: http://www.atmark-techno.com/~yashi/libffi.html#Types
Very C-like.

[ghost]

Yes it can solve the ABI problem, but it doesnt have many advantages over simple C interfaces.

I would say that being able to write wrappers directly in GDScript (allowing them to be distributed on the asset store) and being able to use thousands of C libraries from GDScript are pretty big advantages over having to recompile Godot each time (as is the case with modules).

[reduz]

It's C and really low level... but might solve the issue with external
libraries.

Still though, this is GPL licensed, so forget it.

On Sun, Mar 6, 2016 at 8:52 PM, paper-pauper [email protected]
wrote:

Yes it can solve the ABI problem, but it doesnt have many advantages over
simple C interfaces.

I would say that being able to write wrappers directly in GDScript
(allowing them to be distributed on the asset store) and being able to use
thousands of C libraries from GDScript are pretty big advantages over
having to recompile Godot each time (as is the case with modules).

—
Reply to this email directly or view it on GitHub
https://github.com/godotengine/godot/issues/3943#issuecomment-193017998.

[reduz]

seems the license changed:

https://sourceware.org/libffi/

On Sun, Mar 6, 2016 at 9:19 PM, Juan Linietsky [email protected] wrote:

It's C and really low level... but might solve the issue with external
libraries.

Still though, this is GPL licensed, so forget it.

On Sun, Mar 6, 2016 at 8:52 PM, paper-pauper [email protected]
wrote:

Yes it can solve the ABI problem, but it doesnt have many advantages over
simple C interfaces.

I would say that being able to write wrappers directly in GDScript
(allowing them to be distributed on the asset store) and being able to use
thousands of C libraries from GDScript are pretty big advantages over
having to recompile Godot each time (as is the case with modules).

—
Reply to this email directly or view it on GitHub
https://github.com/godotengine/godot/issues/3943#issuecomment-193017998
.

[bojidar-bg]

If the library is <1-2 MB, we might easily include as a core module, since many plugins would benefit from it.

[est31]

Thats wasted space IMO. what the library does is easily doable with a simple C API as well. C APIs are ABI stable afaik. The problem is just the ++ part :)

[bojidar-bg]

@est31 ~ Ok, then not so many plugins would benefit from it enough...

[ghost]

What C API would allow dynamic calls to C code from GDScript in a cross-platform way, and without recompiling the engine?

Even if it's not C++, the possibilities for writing bindings are huge. The issue with C++ is name mangling, which is not standardized. C is not C++ but at least it offers a sufficient number of important libraries, from ENet to libyaml, all which could be wrapped easily directly in GDScript, without writing any C code.

Edit: The Itanium ABI standardizes C++ name mangling and makes it possible to call C++ from libffi. Of course, the libraries targeted should be compiled with Itanium support, but both GCC and Clang offer it already.

For this reason, if someone knows for sure a library was built with Itanium support (for example by bundling it along with the game), libffi could be used to call C++ code from GDScript.

[ghost]

Also, libffi is ~65K, way smaller than 1MB.

[bojidar-bg]

Ok, I read a bit more about it ( http://stackoverflow.com/a/145649/4168713 , above comments, etc. ), and to me it seems like this would be a nice addition indeed, unless we can ( @est31 ) write our own library to do the same.

Also, using Itanium seems like a good idea, but if it is unreasonable for various reasons, we would have to switch to huge extern "C" bindings (probably generated automatically)....

[ghost]

bumping this

This is an easy solution to implement ABI. Already there, portable, with compatible license and working. And it covers everything Godot can export to.

And here's the thing CPython uses libffi for C extensions. PyPy decided to make their own (called cffi) instead of using libffi and their C support is not as good.

Why reinvent the wheel when you already have this lil package, that has been proven to work, ready to go?

[est31]

Why reinvent the wheel when you already have this lil package, that has been proven to work, ready to go?

libffi _only_ gives you as much access to writing ABIs as c does. It basically gives other languages the functionality of C to write ABIs.

[ghost]

@est31 According to my research libffi is highly optimized C code. Is not just writing interfaces in C but exporting them compiling them using certain flags. This is why PyPy's own version lags behind in numerics for example or why CPython's C extension numerics are top notch.

[akien-mga]

Is this still relevant in GDNative times @karroffel @bojidar-bg?

[cjxgm]

One more advantage of libffi is that you do not need a C/C++ compiler to use a DLL in GDScript, so that I do not need to install VS/MSVC. And when developing cross-platform (Linux+Win) plugins, I can work in Linux completely by testing via Wine for Windows compatibility. (VS or MSVC build tools can't be installed in wine).

[Calinou]

I believe this is solved with the inclusion of GDNative, which allows developers to use native libraries at run-time.

@cjxgm Whether you need a C/C++ compiler to use them depends on the availability of compiled libraries. Most developers currently don't distribute compiled libraries, but this could improve if someone made a continuous integration template that developers could use and adapt in their repository.

(VS or MSVC build tools can't be installed in wine).

Technically, they can :smiley: – it's definitely quite involved though.

[bojidar-bg]

I think this issue might be still relevant, though less relevant than it used to be before GDNative.
libffi (or a suitable replacement) would allow developers to use native functions from GDScript without any prior bindings in GDNative.

[and3rson]

I agree, having FFI would make it much easier to bind to existing shared libraries, especially the proprietary ones like SteamWorks SDK.

We're currently building a game for publishing on Steam and we rely a lot on libsteam_api library. We had to write our own "intermediate" shared library with GDNative which behaves like a bridge between precompiled proprietary steam library & our game. This works great, but involves this extra step of maintaining a custom library just to expose proprietary library in a way that's understood by Godot. Having FFI would make it much easier to do this sort of things via foreign function calls directly from GDScript.

I think doing something like this would be wonderful then:

onready var lib = SharedLibrary("libsteam_api.so")  # or maybe preload("libsteam_api.so")
func foo() -> void:
    lib.invoke('SteamAPI_Init')

Python's ctypes.CDLL is a great implementation. Of course it needs some tackling with argument & return types, but it's still worth it.

EDIT: Hang on... Instead of amalgamating Godot core, we can actually have this as a separate GDNative lib that will use FFI itself under the hood... Yeah, that would be a wrapper for a wrapper, but it will work and will let you use FFI without recompiling bridge libs. GDNative is truly beautiful.

[follower]

Instead of amalgamating Godot core, we can actually have this as a separate GDNative lib that will use FFI itself under the hood...

@and3rson Did you happen to try out this approach?

Am interested in FFI in relation to the Epic Online Services SDK which exposes a C APi with Linux/Mac/Win binary libraries.

Had already wondered about taking the approach you suggested so would like to avoid re-inventing the wheel if you've already done it. :)

[and3rson]

@follower @akien-mga I'd like to work on this.

I'm currently writing a proxy library that allows to load shared libraries during runtime with an interface similar to Python's ctypes.CDLL.

The question is: is it better to do this as a seperate third-party library via GDNative distributed seperately from Godot or would the community want me to write it into the Godot core and contribute as a pull request?

[and3rson]

I think I'll proceed with a third-party library for the time being. I believe it will be pretty trivial to integrate it into core later if you guys would like.

So far I was able to do some basic loading via dlopen & dlsym (will add support for the Windows API later as well) and call methods from a Steam shared library with this code (my library's working title is "Foreigner"):

    var foreigner = preload('res://contrib/foreigner/foreigner.gdns').new()
    lib = foreigner.open('libsteam_api.so')
    prints(lib.invoke('SteamAPI_Init'))  # Prints 1
    prints(lib.invoke_str('SteamAPI_GetSteamInstallPath'))  # Prints .
    prints(lib.invoke_bool('SteamAPI_IsSteamRunning'))  # Prints True

Of course those _str & _bool suffixes are just temporary to test things around. I'm switching to ffi integration now which should provide a more robust way of doing this. I'm still thinking about the most readable usage of it from the Godot side. Some ideas are:

lib.set_signature("some_fn", [foreigner.INT, foreigner.FLOAT], foreigner.STRING)

# or maybe more Python-ish...
lib.some_fn.argtypes = [foreigner.INT, foreigner.FLOAT]
lib.some_fn.restype = foreigner.STRING

# or just passing them during the call, FFI-style:
lib.invoke("some_fn", foreigner.STRING, [foreigner.INT, foreigner.FLOAT], [34, 42.0])

Also I think it made sense for "open" to actually return a pointer to some other class instance, probably ForeignLibrary, and make all functions instances of ForeignFunction.

Thoughts are welcome!

[and3rson]

FFI came out to be really easy to use. I've started the development here:

https://github.com/and3rson/foreigner

It's currently capable of performing simple function calls via FFI. I've tested it with Steam shared library. I've also added some boilerplate code for Godot/FFI typing which can get tricky once we start dealing with pointers. So far it works! You can try it out yourself.

Your contributions are highly appreciated! (Well, not much to contribute to yet, but an extra pair of hands is always helpful!)

[and3rson]

Considering the fact that I've already hijacked this thread (sorry), here are some updates:

• I've got most of the basic stuff working - pointers, Variant/C conversion etc.
• Casting between Variant & ffi_type_* came out to be pretty simple thanks to all the overloaded operators for casting to primitives.
• Pointer support is done via uint64_t mode of Variant and allows GDScript to pass them between different invocations of foreign functions (Yay!)
• Some memory leaks are still present (like allocated FFI CIF structures), but that's mostly just me forgetting to free memory.

Anyway, the flight is stable.

[follower]

If only all "thread hijackers" were as productive as you. :D

Thanks for responding to my question...with the best answer: code! :)

A couple of thoughts from reading your issue comments & skimming the code:

• While I think it would be great to have this functionality builtin I think your decision to make it 3rd-party initially is probably the fastest way to making the functionality available at all. Once the code is stabilized and (widely! :) ) used I think it makes it easier to present a compelling case for being builtin (if that still ends up being considered ideal).

• Appreciate you listing potential approaches for specifying the function signatures. I think the FFI-style makes sense initially but a set_signature() approach seems like it would be nice so the signature only has to be specified once. (It probably also depends a bit on how people end up "wrapping" libraries on the Godot side.)

I'm keen to try out your code but haven't yet because I'm also trying not to let this distract me from what I need to be doing instead. :)

Anyway, just wanted to give you some positive feedback so you didn't feel like you were typing into the void. :)

Thanks!

[KoBeWi]

Feature and improvement proposals for the Godot Engine are now being discussed and reviewed in a dedicated Godot Improvement Proposals (GIP) (godotengine/godot-proposals) issue tracker. The GIP tracker has a detailed issue template designed so that proposals include all the relevant information to start a productive discussion and help the community assess the validity of the proposal for the engine.

The main (godotengine/godot) tracker is now solely dedicated to bug reports and Pull Requests, enabling contributors to have a better focus on bug fixing work. Therefore, we are now closing all older feature proposals on the main issue tracker.

If you are interested in this feature proposal, please open a new proposal on the GIP tracker following the given issue template (after checking that it doesn't exist already). Be sure to reference this closed issue if it includes any relevant discussion (which you are also encouraged to summarize in the new proposal). Thanks in advance!