Godot: The need for a better relationship with GDScript when using other languages

[mateusak]

GDScript has the role of being a fast to write, easy to use language. It does well in that area and it's sufficient for most game code. But sometimes you need a faster language (to do procedural generation, for example). The issue is that none of the languages available right now can interact with GDScript very well.

C#
Calls between C# and GDScript are slow; integrating C# into GDScript code means having to translate many of the advanced data structures of C# (like structs, multidimensional arrays and so on) to simpler data that GDScript understands which is time consuming; C# is also vastly undocumented in the sense of interacting with Godot types, which makes this job even harder.

C++/GDNative
Hard to setup and maintain; runs in the same problem as C# which is having to translate data structures (like structs in C++) to GDScript; even less documented than C# (to setup and write).

Possible solutions

• Making GDNative as easy to setup and write as GDScript (or just easier at least).
• Vastly documenting which X language types are which in GDScript (_which is not really feasible: the main documentation is already out of date with the current state of the engine, imagine the several unofficial languages_).
• Automatic conversion of common types (#26104)
• Adopting official prefixes for Godot's types in other languages, making clear what's from Godot and what's from the language. So Godot's Vector2 in C# would be GDVector2 for example.
• Adding some more common data structures to GDScript, like structs and multidimensional arrays, so they can be easily translated.
• Improving GDScript call speed, allowing better communication between itself and other languages.

[neikeq]

Calls between C# and GDScript are slow

That's because GDScript calls are slow. You will find the same problem for pretty much any other language trying to interact with GDScript, as you noticed is the case with languages integrated with GDNative.

integrating C# into GDScript code means having to translate many of the advanced data structures of C# (like structs, multidimensional arrays and so on) to simpler data that GDScript understands which is time consuming

There is nothing that can be done about this except perhaps some little improvements here and there. The same way you cannot bind every engine function to be usable from GDScript. You will find functions in the engine that are suffixed with _bind (or similar) that act as wrappers to be callable from GDScript.

• Improving GDScript drastically (_especially in for/while loops and array handling which are terribly slow right now_) rendering the need for a performant language non existant.

GDScript performance is most likely already tracked by another issue.

• Making GDScript a hybrid language, where using optional static indicators increases performance.

This is already implemented, and further improvements are most likely also already tracked by another issue.

• Making GDNative as easy to write as GDScript, integrating it into Godot's script editor.

I don't know what you mean by "as easy to write". I will assume you refer to the workflow in general for GDNative-powered language bindings. This is not really a GDNative issue. Some improvements may be done to GDNative/NativeScript and the Godot editor, but at the end of the day it depends on the people working on those language bindings.

• Replacing GDScript with a static and performant, yet easy to prototype custom language.

This is unlikely to happen. Instead, expect mode improvements on optional-typed GDScript and the possibility of implementing JITing or compiling AOT to improve performance.

• Creating another language alongside GDScript, one that focuses on performance.

This is not going to happen. Use C# or GDNative language bindings.

• Vastly documenting which X language types are which in GDScript

Can you give an example where this, if not documented, is not trivial to figure out?

[mateusak]

@neikeq

There is nothing that can be done about this except perhaps some little improvements here and there.

I know that, unfortunately. That's why I opened this issue. This issue is not about duplicating the intentions of all these other trackers for GDScript performance or whatsoever (_because even if it reaches a speed similar to C#, there's still the need to interact with other languages to make use of libraries, for example_). It's about how can we facilitate the interaction of languages with GDScript.

I don't know what you mean by "as easy to write"

I meant being able to create a C++/Nim/whatever the same way you create a GDScript file, and editing it in the same way too; using Godot's built-in script editor.

This is unlikely to happen

Most things on the list are, hahaha.

Can you give an example where this, if not documented, is not trivial to figure out?

How to pass a C# multidimensional array to GDScript? Passing array[,] of course doesn't work. But neither does array[][]. Is it just not possible? I wouldn't know, since it's not documented.

[avril-gh]

Most things on the list are, (...)

Because this topic is as old as godot, and it was allready discussed for years. There has been many languages implemented and droped and to put it short, after long time and many implementations GDScript has came out by evolution to be best approach from many angles.
so after most questions has been answered by life, speed is always the concern, and nowaday's disscussion are "to JIT or not to JIT", ect.

[mateusak]

@avril-gh I don't think you grasp exactly what's the issue here. The issue is how can we improve either the available languages or GDScript to interact better with each other. I guess putting suggestions like improving GDScript's speed was a misled by my part. Edited the post to better represent the issue.

[QbieShay]

Maybe it is possible to have a plugin that ships with whatever compiler your gdnative script needs

[avril-gh]

i just been moved by your other issue #25753 which sounded like "GDScript is bad, everyone hate it, lets abandon it". I personaly like it very much, its very good and i know many people think that way too.
Sorry if i missunderstand. My mistake.

[mateusak]

@avril-gh Oh no, not at all. I love GDScript. It does what it needs to do. My issue is exactly that it doesn't seem very usable together with other languages, which is a shame since I wanted to use GDScript as much as possible and use other languages just when needed 😄

[girng]

neikeq's response is really good

I meant being able to create a C++/Nim/whatever the same way you create a GDScript file, and editing it in the same way too; using Godot's built-in script editor.

yeah, i think you just want "official" godot editor executables for their respective language bindings? for example, nim, rust, crystal, etc? you have to find developers who excel in that language and godot to maintain those AFAIK. there's been some talk in crystal's gitter about crystal bindings.. but IMO the core
devs have much bigger fish to fry

but really, this "gdscript is slow" really hurts.. my heart sinks every time. gdscript is fast enough for 98% of games out there! and plus, the optional static typing and future JIT optimizations will make it perform even faster in the future. there is about zero reasons not to use it. really, it's such a blessing @mateusak, take advantage of it! time is gold

[mateusak]

@girng Not really official Godot editor executables, no. Just easier installation and use of GDNative. Yeah I know GDScript is sufficient for 98% of use cases... oh my god, I shouldn't have brought that up, it isn't even the topic. The thing is, in those 2% you need another language. I just want the interaction between GDScript and that language to be smoother, it's all I'm saying.

[girng]

@mateka i see. sorry if i misread 👍. i know how ya feel. the vid tutorial on gdnative does seem a bit complex now skimming over it. however, it's low level stuff so prob no way around that ;O

[vnen]

I don't know why you need to pass so much data to GDScript. Why do you need a multidimensional array for? If you are going to loop that in GDScript, you are already losing (some of) the performance you gain from using a faster language.

If you need to arbitrarily access data, you need to use Godot data structures (Arrays and Dictionaries), to take advantage of the easy access. I honestly don't think you need custom data structures in GDScript. You should use the other language to process that (since you are already using another language anyway). Passing data between languages should be reduced to a minimum.

For instance, if you are using C# to generate a map, you should not pass a bi-dimensional array to GDScript and construct a TileMap there, you should instead create the TileMap directly in C# and pass only the resulting Node, which can be easily accessed in GDScript.

Also, you technically can create a C# object and keep a reference to it in a GDScript variable, from which you can access properties and methods. So you should keep the data in the faster language and only expose an API for GDScript.

[vnen]

Making GDNative as easy to write as GDScript.

This is vague and technically impossible. GDNative is not a language, so you really can't "write" it. It's just a C-compatible interface.

We all agree docs should be better, but I don't think there's any way to make GDNative as easy as GDScript (and also don't think it's necessary). If you want to write C++ you need to know how to deal with a compiler as well, there's no way around it.

[mateusak]

@vnen Passing nodes around just means I have to duplicate everything I want to pass to GDScript. I'll pass a TileMap node to my TileMap node so it can rewrite it... 🤦‍♂️ . It even goes against good coding practices doing everything on a single C# script. What if I want to signal the resulting bool[,] to a bunch of nodes, that would use it for various purposes? Do I signal a TileMap node? Why? You can't actually call that a solution. Right now you can't pass a bool[,] or a bool[][] to GDscript. I agree that data flow between languages should be reduced to a minimum. But it should still be _possible_, _easy_ and _fast_.

Surely GDNative will always be more complicated than GDScript. The very nature of it guarantees it. But there are ways to make it better. There's no place right now that functions like an asset store for GDNative. Just making a place to allow people to share their GDNative projects organizedly, with the ability to list dependencies and such, would go a long way.

[QbieShay]

@mateusak I think for multidimensional arrays you can use a 1D array and have a get_2D(row, col, size_row, size_col) and get_3D(x, y, z, size_x, size_y, size_z) functions implemented onto array that does some index magic for you. Possibly you also have append2D and append3D that appends in a coherent order for the way that is then read from the other functions

[karroffel]

There's no place right now that functions like an asset store for GDNative. Just making a place to allow people to share their GDNative projects organizedly, with the ability to list dependencies and such, would go a long way.

I think I know what you mean here, but sharing on a "function" level will lead to a very high fragmentation of functionality and suddenly your project depends on 34 GDNative libraries which all do just one thing. You don't want left-pad or event-stream in Godot's ecosystem.

There actually is a place to share GDNative libraries - the Asset library. There are at least 4 GDNative libraries in there for providing VR drivers to Godot, there are also at least 2 libraries to bring Python support into Godot.

A GDNative setup has to include any binary dependencies it has, it's just that generally those dependencies are vendored into the package.

I think the general usage of GDNative should not be "here's this library which does one thing", because the management overhead of that will be quite big once you have two or three of those.

I'd say the best way to use GDNative is to have one library per project that you write (and any possible third party GDNative libraries (like VR)) and use Godot types as much as possible.

For example, as @QbieShay pointed out, you can have your generation of procedural data done in a GDNative class, but instead of using int filed[HEIGHT][WIDTH] inside you can use PoolIntVector field and have accessor functions that map your 2D grid onto that 1D array. Because it's a PoolIntVector you can painlessly pass it to Godot with basically 0 additional overhead, while inside of GDNative you can manipulate it using raw pointers.

[vnen]

Passing nodes around just means I have to duplicate everything I want to pass to GDScript. I'll pass a TileMap node to my TileMap node so it can rewrite it...

My point is that you shouldn't need to rewrite. Just create the TileMap in C#, pass it to GDScript to add to the tree (or even add directly from C#). As others have said, you should focus on structures available in Godot itself when you want to pass data around.

[mateusak]

@vnen

Also, you technically can create a C# object and keep a reference to it in a GDScript variable, from which you can access properties and methods. So you should keep the data in the faster language and only expose an API for GDScript.

Calling C# methods from GDScript takes a considerably amount of time (_maybe not as much as I thought (read latest comments), still: better avoid_).

My point is that you shouldn't need to rewrite. Just create the TileMap in C#, pass it to GDScript to add to the tree (or even add directly from C#). As others have said, you should focus on structures available in Godot itself when you want to pass data around.

Yeah I understood what you meant, haha. My counterpoint is that maybe it's not possible to add to the tree. What if my TileMap node is connected with signals? What if something else alters it before the map is generated?

I get it you're trying to help me saying what I could possibly do to make it work. The thing is, I can make it work right now. _You can always_ _make it work_. I'm just saying it could be more readable than maintaining a C# script in your GDScript or passing around TileMaps, faster than taking 0.05s to call a method and easier than having to pass structs as dictionaries or using monodimensional arrays with index magic.

[mateusak]

@QbieShay I could, but you would agree with me that isn't very elegant (or readable), wouldn't you? Like I said, I can make things work right now, I'm just advocating for it to be... better.

I get it you're trying to help me saying what I could possibly do to make it work. The thing is, I can make it work right now. _You can always_ _make it work_. I'm just saying it could be more readable than maintaining a C# script in your GDScript or passing around TileMaps, faster than taking 0.05s to call a method and easier than having to pass structs as dictionaries or using monodimensional arrays with index magic.

[QbieShay]

Yes, it is not elegant indeed. That said, I think it's super great that Godot dares to support multiple different languages, I guess making things nice to use is hard when so much flexibility is required.. but anyway, I believe something might be done to improve the workflow. Not sure about improving the calls performance, but maybe the workflow for cpp could be indeed improved

[mateusak]

@karroffel You _can_ post GDNative in the Asset Store? Well... okay then. Problem solved. Hahaha. I wonder why there's no more complex languages there as well, like Nim or Rust? I guess the developers are just not interested in publishing there.

Like I said to Qbie, using monodimensional arrays to simulate what I need is at best hackish. I hate doing it for A* already I can't imagine being sane after doing it for the entirety of my project. Oh just seeing those nasty pos2idx everywhere... oh my god... vietnam flashbacks.

[karroffel]

@mateusak It's not like people don't care to publish, it's having something to publish in the first place. In the case of the Rust bindings it's mostly a problem of them not being ergonomic enough that anyone would use them on a day-to-day basis. For nim I guess it's just size of target audience and need for something like that.

Currently the only things in the asset library made with GDNative are things that would usually be part of the engine, like VR drivers or scripting language implementations. If you just want a function or two then pulling in GDNative is just overkill IMO, and at that point you can write this function on your own and use GDNative for more things as you already have it set up.

Using continuous memory to simulate multi-dimensional arrays is actually the ONLY way to do them, it's just that languages hide that from you. That's because flat memory is all there is in your system, everything else is just abstraction. Now the problem is if you have one language that has that abstraction (C#, C++, Rust, ..) and a system that does not have it (Godot's Variant class + GDScript). In that case there's no way for you to do it, as there's just no way to express it.
So here we are not fighting for "we need a statically typed language that's fast" but for "there is a need for more expressive types".

In which case I agree with you, I'd love to see more expressiveness in GDScript, but I also don't expect that to happen since it doesn't really align with its goals.

Good language design is not about implementing cool features, but to create a coherent system.
C is great because of that, C++ isn't, Rust is, Haskell is, many other languages are, and I would say GDScript is too. (That doesn't mean I don't want typed arrays :yum: )

[mateusak]

@karroffel I guess the GDNative situation is just about maturity then. I'll wait and observe.

Using continuous memory to simulate multi-dimensional arrays is actually the ONLY way to do them, it's just that languages hide that from you.

But doing so in GDScript is a very poor choice since every function call (pos2idx) takes the entire duration of the Lord of the Rings trilogy... extended edition. Hahaha.

So here we are not fighting for "we need a statically typed language that's fast" but for "there is a need for more expressive types".

But that's the point I've been trying to make! What! All along I just want GDScript to communicate better. I know at first I came up with some unfeasible solutions, but now I've redeemed myself and I just want some common types to come to GDScript (and the other things I've listed in the post), like structs and multidimensional arrays, so it can receive data from other languages without hacks.

[neikeq]

... Calling C# methods from GDScript takes painfully long (_for reference calling an empty method takes 0.05s, ...

Are you sure about this? 1 ms for a simple call would already be pretty bad, but 50 ms? Can you provide an example benchmark were this can be confirmed?

[mateusak]

@neikeq Hmm... I just tested it again and it doesn't seem to take any time. Oops. I remember taking 0.05s when I tested. Although I also remember an unrelated error occurring somewhere else. Maybe the error altered the results. Sorry for the misinformation then, I guess. I'll edit it.

In a million interations, calling an empty C# function from GDScript took 1.632s and calling an empty GDScript function from GDScript took 0.153s.

Also in a million iterations, calling a create 2D array C# function from GDScript took 3.436s and calling a create 2D array GDScript function from GDScript took 99.128s. If only there was a way to pass that C# array back to GDScript... hahaha.

[QbieShay]

@mateusak have you tried cpp instead? Apparently c# is a 'special case' because the objects can't be easily translated due to csharp's own architecture.

[mateusak]

@QbieShay I wouldn't dare to use C++ to write procedural generation. I can barely detect errors in C#. Now say, Rust, I would give that a try. But unfortunately it doesn't seem very complete.

[mnn]

If you want to write C++ you need to know how to deal with a compiler as well, there's no way around it.

Current state (at least beta3) goes way beyond "just know how to use gcc". There is currently no "preset", no fully working template to get GDNative with C++ running in a few minutes. I spent literally half a day trying to make demo working...

We all agree docs should be better,

Yes, docs with C++ demo are broken and often confusing - on many places it is not clear about what directory the docs are talking (I think bindings repo had more clear guide, but I didn't find that out until I no longer needed it).

but I don't think there's any way to make GDNative as easy as GDScript (and also don't think it's necessary).

There should be easier way to get it set-up. Why are not cpp bindings (binaries for all platforms) for each official version prepared? (Seeing them even for beta releases would be nice.) Why couldn't we have fully working template project to download? Why at least all needed commands for first time are not in some script to simply run, not having to copy&paste many commands in different directories manually? I don't know scons at all, but isn't that thing a build tool? Coming from JavaScript world, where are scaffoldings/generators for many FE frameworks, even smaller ones, this is a giant contrast. It is literally two commands to set-up a new project: install package from npm and run generator which asks you few questions and generates everything, so you can just run dev server and start developing in literally under a minute from start.

In the end I found some random demo, which had broken installation guide (I had to manually download all dependencies, I really don't like git submodules). Yeah, but that whole process of setting-up GDNavetive with C++ took me like 4 or 5 hours, just to get a demo running...

I understand beta can have some bugs (like https://github.com/godotengine/godot/issues/25812), but why marking it as beta, so users can start testing it, when docs are not finished? I am not saying polished, but at least in a state when they reflect current version? I don't know, this process doesn't seem to me thought-out very well.

I am sorry if this sounds too negative, but as you can tell from the amount of wasted time, it was quite frustrating for me.

[mnn]

@QbieShay I wouldn't dare to use C++ to write procedural generation. I can barely detect errors in C#. Now say, Rust, I would give that a try. But unfortunately it doesn't seem very complete.

What a coincidence, the reason I am trying C++ is to finish benchmark for voxel related tasks. So far I got only data generation part for GDScript and C#. GDScript's performance was bad (as expected, slower more than by 11 times compared to C# in Godot), but C# + Godot was a nice surprise, since it was slightly faster than Unity + C# (bench does not count startup times, only measures time of the method which generates voxel data). Possibly Mono version difference? I also noticed perf regression of GDScript between stable and beta by ~1.75%, but my sample size is quite small (several runs with increasing "difficulty", but only one run per each difficulty; highest took GDScript over 2 minutes, C# was slightly over 10 seconds).

Now I am fighting a bit with C# <-> GDScript interoperability (https://github.com/godotengine/godot/issues/25825), probably will have to limit usage of GDScript as much as possible (in bench and game as well, if we decide to use Godot), because passing data is slow and works correctly only in some cases. Originally I wanted to write just world gen and mesher in C# or C++ (only if significantly faster, I am a C++ noob) and rest in GDScript, but that won't probably be the case :disappointed:.

[mateusak]

@mnn Yeah, for my case it doesn't really matter between C# and C++; I'm just manipulating small 2D arrays. C# is already fast enough (and about 500x faster than GDScript).

The way I'm going about it is to isolate the generation part completely in C# (everything from number crunching to building) and use GDScript for the rest (enemies, etc). If occasionally I have to communicate to the generation algorithm through GDScript I'll just hack it away.

[piratesephiroth]

I actually don't really see a reason to insist in using and improving GDScript (other than C# support being in alpha state)
All it does is save you from writing a bit of code. Also said code is full of ugly snake case names and indentation for blocks.

C# is already extremely simple and the performance is way better, so to me it's really a no-brainer.
I think Godot should just switch over to C# scripting.

[Zireael07]

1) Why is this not tagged mono?
2) I am thinking of moving more performance intensive parts of my biggest project to c#, and I have no idea how to make gdscript and c# play ball with each other (array problems notwithstanding). Can we maybe get a note on that in the docs?

[mateusak]

@Zireael07 What I can say to you, from toying and dancing with it for the last month, is that

1. You can't pass C# arrays to GDScript, with the exception of an object array. This means you have to convert your arrays into object[]. If you have a multidimensional array, you have to convert to object[][]. Object[,] will be converted, but it'll do so as a single dimension array in GDScript.

1. If you want to use Godot's random number generator, you need to use RandomNumberGenerator (which is not mentioned anywhere in the docs). I'm not exactly sure how it works yet, but it seems it creates a copy of Godot's generator (meaning if you call randi in GDScript and then in C# you'll get the same result); this means it's impratical in almost any of the scenarios where you'd need the Godot generator.

I hope this helps you.

[neikeq]

1) Why is this not tagged mono?

Why is this an issue with the mono module? If there is any improvement that can be done in the mono module in this regard, please open a new issue for that specific thing.

[Zireael07]

A better relationship between GDScript and C#, as listed here, may involve improvements to mono module?

[neikeq]

@mateusak You can pass a RandomNumberGenerator instance between C# and GDScript. I don't understand what is your issue with it.
As of ce67808818d7d3d17a3d6702f09d8f487e078e0b you can also use the default rng from C#. The equivalent of the rng global functions.

[neikeq]

@Zireael07 I already talked about this in my first comment. There is not much that can be done here, and if there is any specific thing that can be done in the mono module it should go into its own issue.

[mateusak]

@neikeq My issue:

this means it's impractical in almost any of the scenarios where you'd need the Godot generator

Not only that but the docs are misleading.

As of ce67808 you can also use the default rng from C#

I'm using Godot 3.1 Beta 5, unfortunately.

[neikeq]

this means it's impractical in almost any of the scenarios where you'd need the Godot generator

Why...? Like I said you can re-use the same RandomNumberGenerator instance.

[mateusak]

@neikeq Yes, but Godot's generator is supposed to be a global generator. If I'm going to use it as a global instance to achieve the same effect I can just use C#'s Random.

[neikeq]

I don't think I follow. What is the problem with RandomNumberGenerator or the default rng? What do you mean it's supposed to be a global generator? Sorry if the problem you're describing is obvious but I just don't understand it.

[mateusak]

@neikeq Hahaha, just forget it, it's unimportant as of ce67808 anyways. Thanks for that btw.

[Zireael07]

Just as an example, if I have a Math.cs which contains a Arc() function, how do I call said function from Gdscript? Most importantly, scripts where I need it already extend Godot classes, e.g. VehicleBody or Node2D.

[neikeq]

You can't access it. You can only access Godot objects from GDScript (Math would need to inherit Godot.Object).

[Zireael07]

And if it inherited Object, would it be possible to access?

[neikeq]

Yes. You would then access it as you would do with other GDScripts.

[vnen]

I believe this discussion has outlived it's usefulness. There's nothing really actionable from the discussion, and we usually don't like issues with multiple points since it's tough to track what has been done and when to close it.

For stuff that is still missing, please open a proposal at https://github.com/godotengine/godot-proposals