Godot: [TRACKER] Networking improvements

[mhilbrunner]

This issue is to collect and discuss improvements to the networking in Godot 3.

• [x] Methods on IP to get loopback, link local, or external addresses instead of all of them as an array (with get_local_addresses()) (likely not possible)
• [ ] Method to get networking stats (min, max, avg for ping/latency/packet loss/bandwidth) (https://github.com/godotengine/godot/issues/8763)
• [x] Lobby / LAN host autodiscovery using broadcasting (possible through broadcasting using PacketPeerUDP, fixed here: https://github.com/godotengine/godot/pull/19378)
• [ ] Docs for movement interpolation & reconciliation (lag compensation, see article) (https://github.com/godotengine/godot-docs/issues/1216)
• [x] Expose ENet (features like channels, but also the whole of ENet) to GDScript, to avoid users having to do their own wrappers when ENet is included anyway (https://github.com/godotengine/godot/issues/6642, https://github.com/godotengine/godot/issues/13206, PR: https://github.com/godotengine/godot/pull/18827)
• [ ] Support for launching multiple instances of Godot on play and/or server and client running at the same time in the same instance (https://github.com/godotengine/godot/issues/7008)
• [x] NAT Traversal/Punchthrough Support by allowing to set client port (https://github.com/godotengine/godot/issues/2970, PR: https://github.com/godotengine/godot/pull/18666)
• [x] UPnP support (https://github.com/godotengine/godot/issues/2970, PR: https://github.com/godotengine/godot/pull/18780)
• [ ] Relay server support or relay through other peers (article) for cases where UPnP/NAT Traversal fail
• [x] Ability to kick/disconnect peers as the server (https://github.com/godotengine/godot/issues/16452, PR: https://github.com/godotengine/godot/pull/18070)
• [ ] Auto node replication, allow joining mid-game (https://github.com/godotengine/godot/issues/16793)
• [x] Method to get the IP/address/port of a certain peer (https://github.com/godotengine/godot/issues/17798, PR: https://github.com/godotengine/godot/pull/18145)
• [x] NetworkedMultiplayerPeer should work with hostname / domain (https://github.com/godotengine/godot/issues/12512, PR: https://github.com/godotengine/godot/pull/18154)
• [x] Multiplayer API refactor, make extendable/overridable (PR: https://github.com/godotengine/godot/pull/17227)
• [x] Better error checking for RPCs based on types (https://github.com/godotengine/godot/issues/16733)
• [x] Allow sending raw data in multiplayer API for minimal overhead (PR: https://github.com/godotengine/godot/pull/18823)
• [x] RPCMode refactor, new keywords (remotesync, mastersync, slavesync) for safer RPCs (PR: https://github.com/godotengine/godot/pull/19021)
• [x] Fix UDP broadcasting on Linux (https://github.com/godotengine/godot/pull/19378)
• [x] Unify socket implementations across platforms, remove duplicated code (https://github.com/godotengine/godot/pull/21692)
• [x] Add clear error messages for networking (https://github.com/godotengine/godot/issues/22318)
• [ ] Docs for all these awesome new features :)

[groud]

1. Why not, but I wonder how you can manage to do that without dirty workarounds.
2. Would make sense, not to use the ICMP protocol, but having infos on the network makes sense to me.
3. For LAN that would be cool
4. Why not, but it is probably really hard to design a smart API for that.
5. Don't know for this one.
6. Like launching both from the editor ? Makes sense.
   7 & 8. I don't think UPnP is meant to be used for games. Maybe it would be better as a plugin. Anyway, we're not here to provide networking solution, we'd better go for simple IP based API and let the network engineers provide plugins for specific needs.

Most of the ideas seems cool :)

[mhilbrunner]

@groud: For exposing ENet: It just seems a shame that multiple people had to work on their own Godot ENet implementation when it is already there, just not exposed sufficiently.

E.g. https://github.com/PerduGames/gdnet3, https://github.com/Faless/godot-enet-better etc.

Thanks for the feedback!

[mhilbrunner]

Regarding 2: Would the best way be to implement a "heartbeat", a packet that gets periodically sent by each peer at a configurable interval to measure latency (and make that information then available on the peer/SceneTree)?

[Calinou]

I think "Support for movement interpolation & reconciliation (lag compensation, see article)" can be highly game-specific, and probably can't be implemented as-is, since the implementation methods may vary a lot depending on the game genre.

Providing built-in support for Hermite interpolation could be a good idea, and it can be used in areas other than networking, see this article for more details.

[mhilbrunner]

According to reduz, ENet channels are not supported on mobile. So if they can be exposed at all, this probably has to be taken into account.

[nasdf]

I'd like to help with launching multiple instances for server and clients. Does anyone have a good idea of how this should work? I can think of two solutions for this issue.

1. Have a button to run an instance in server mode separate from the main scene
2. Allowing the NetworkedMultiplayerENet to be attached to a child node instead of the scene tree

[Faless]

Opening some old wounds :wink: ...

1. Can we? We should check OS capability but I don't think we can, IPv6 link-local are already excluded (as they don't really works as expected), how can we identify loopback address in a cross platform way?
2. Will need to expand the multiplayer protocol for that, but sounds like a good idea, should be too hard.
3. That sounds too implementation specific.
4. Interpolation should be easy to implement for physics, reconciliation and lag compensation is complex, and sounds too implementation specific. Also, it's an entry point for cheaters so at least is should be possible to disable it.
5. No ENet DOES support channels on mobile. It's other API that does not (i.e. the Google Multiplayer API). I was a strong supporter of having this exposed to the multiplayer API itself, and just do custom implementation or just signal an error when using them with library that does not support them. It just seems not smart to hinder many implementation (WebRTC has channels too) just because few less used libraries does not have it.
6. #7008 is about launching two different instances, what you seems to want is to have both server and client in the same instance. Another old wound, I deeply believe that the network thing should not be on the SceneTree but on a specific MultiplayerNetworkNode, that behaves the same as SceneTree does now, but only handling its children. At some point I had a working implementation of this (probably broken now). See https://github.com/Faless/godot-enet-better/tree/rpc_enable . And the patch to godot: https://github.com/Faless/godot-enet-better/blob/rpc_enable/node_rpc.patch . This is important for games that wants to run both server and client in the same instance, a typical example is Quake III arena which used to run a server on localhost even when playing single player. (allows for less code duplication.)
7. Would be really nice to have, I did start studying it at some point, but never implemented something useful due to lack of time. This could also solve 3 in a way that does not clutter the engine too much.
8. That is technically already done by the multiplayer server peer as the API itself is not fully peer to peer (so to allow for authoritative servers). This is very implementation specific, depends on the underlying layer (eg. WebRTC with Websockets STUN servers) etc. Wouldn't make much sense to have a Godot instance running just as a STUN server basically.

@reduz sorry about bringing this up again, I was kinda dragged to it, but I really care about point 6 (and a to point 5 too :wink: ).

[nasdf]

@Faless Your patch for issue 6 is exactly what I was looking for. It's currently a huge pain point when developing games with dedicated servers.

[reduz]

Some answers I guess

1. No idea how to do this
2. Seems implementation specific,
3. Seems implementation specific, and likely outside the scope of multiplaye API
4. Seems game specific
5. Not against channels, but will not work for blutooth/online network on Google/Apple APIs. If you want to port your game there and you depend on the feature, it won't work.
6. Too game specific, but maybe a tool to launch multiple godot instances would be useful
7. Indeed would be nice
8. This is how it works currently.

[mhilbrunner]

@reduz Just to clarify, can you explain what you mean by "implementation specific"?

[reduz]

By the way, nothing prevents from adding extra functions to MultiplayerNetworkPeerENet, even if not in the base class. Something like set_send_channel() may work (not sure if there is any point on checking what channel the data has been received on, since RPC already handles this)

[reduz]

you could do something like.

peer.set_send_channel(10)
rpc("blabla")
peer.set_send_channel(0)

[reduz]

@mhilbrunner Not sure if all APIs measure ping, latency, packet loss, etc. the same way.
If ENet has methods for this, i would just expose them in the ENet object, not in the base class.
For a more generic way, you can do it by user code (just send a packet and see how long it takes to come back).

Discovery is also completely implementation specific. For IPV4 you can send packets to all your subnet to test, or broadcast. IPV6 is likely different, but you have already other network classes to do this, does not need to be part of the multiplayer API.

[mhilbrunner]

@reduz The above idea is interesting. But couldn't it potentially cause weird behaviour/race conditions in case of multithreading?

Just to compare to another engine, Unity has the following (the others are not applicable):

2 (Ping): https://docs.unity3d.com/ScriptReference/Network.GetAveragePing.html

3 (Discovery): https://docs.unity3d.com/Manual/UNetDiscovery.html

4 (Interpolation): https://docs.unity3d.com/ScriptReference/Networking.NetworkTransform.html

6 (Start Network+Client in same app): https://docs.unity3d.com/ScriptReference/Networking.NetworkManager.StartHost.html

7 & #8 (NAT traversal/punchthrough/relay servers): Supported in UNet + via plugins

The story for UE4 likely looks similar; I know it can launch a configurable number of clients at "Play" and supports interpolation, at least.

For now, I'll look into ping, discovery and the channels. I personally don't need NAT traversal currently :) If someone wants to chime in on any of these or help out, PRs welcome, of course :) As well as helpful pointers.

ENet seems to already have support for average pings, which may not be optimal but could be a starting point.

[reduz]

@mhilbrunner no, because the networking API is not meant to be used from multiple threads

Regarding those functions, ping may make sense given you need it to work on both ends, which is why I suggested checking if ENet has something like this.

But discovery, interpolation and host, we can provide a script in the Asset Library, given there isn't a single way to do it.

[Faless]

Regarding those functions, ping may make sense given you need it to work on both ends, which is why I suggested checking if ENet has something like this.

ENet does something internally, but I've read around it's not to be used as a good latency metric.

Addiontally, we might want to have it on other libraries too (e.g. Websocket, WebRTC).

Given that we already have a small protocol for managing peers (see SYSMSG_ADD_PEER, SYSMSG_REMOVE_PEER), adding SYSMSG_PING and SYSMSG_PONG and calculating avarage over time should be really simple.

1. Too game specific, but maybe a tool to launch multiple godot instances would be useful

@reduz , this is something that at the same time adds a new feature, makes debug easier, unclutters SceneTree, unclutters Object and most of all moves the Multiplayer related code in it's own class/classes where it belongs, I really can't understand why you are against it.

I'd love to do it myself and show it to you if there's the smallest chance you would change your mind (pretty please :wink: ).

[reduz]

Current API is meant to be simple, and I see no problem in adding a tool for launching (or even debugging) multiple instances to test. Making a worse API just for the hack of testing all scenes together does not seem good to me.

Again, feel free to make a separate library via gdnative if you want something more specific.

[Faless]

Current API is meant to be simple

And that is not going to change. The idea is to keep the same API, with the small addition of also allowing a node to the same work that is now done by SceneTree. The API itself will not change, current demos will keep working as they do now with no change.

feel free to make a separate library via gdnative if you want something more specific.

That is not going to be possible without huge and unmaintainable copy-paste from core files, unless we separate the Multiplayer protocol from the core.

I guess I'll try to make a PR that adds no more than few lines of code and respect those two conditions (same API, separated protocol) to at least allow for a gdnative implementation without rewriting a lot of code, I'm confident that once you see it, you will like it, as again there will be no change to the API (nor the current SceneTree behaviour).

[reduz]

Changes need to be made on a use case basis. If this work is only to make it easier to test multiplayer code, then It's definitely undesired.

[Faless]

Changes need to be made on a use case basis

The many developers that kindly asked to allow for running both client and server in the same instance because their game requires it, or because it makes their code way better decoupled and not redundant.

EDIT:
And I brought the example of Quake III, but any FPS or even turn based game would benefit from it

[reduz]

The current design is not meant to run as client and server with two simultaneous instances. The server, as things are now, can run as a client without any difficulty and without being redundant. If debugging is the problem, then the debugger can be modified to support launching and debugging multiple instances at the same time (it should be pretty easy to do this).

Again, what you are proposing is considerably more complex than what is in there and I don't really see any advantage. Supplying it externally should really not be a problem.

[reduz]

If all your miss are the RPC functions, a wrapper could easily be done so they call your node instead of SceneTree, then you can handle them as you want.

[reduz]

Regarding Quake III, this is really ancient multiplayer design. I'm not sure why you would want to go with that approach. Reminds me of Torque, which was completely unusable.

[Faless]

Regarding Quake III, this is really ancient multiplayer design

EDIT: As I said at the beginning, it's an old wound, I don't want to argue about this,

I'll see if I can at least unclutter SceneTree and Node to have the networking part separated in the code. We'll see then if it's worth merging or not, at this point I won't mind any wasted work

[reduz]

Let me rephrase myself. I see zero advantage over the current approach, which is simpler and can perfectly act as server authoritative.

Running both a client and a server in the same process only makes implementation more complex and provides nothing in exchange. Why should a considerably more bothersome approach need to be supported when the default one provided serves fine for large majority of use cases?

[Faless]

Why should a considerably more bothersome approach need to be supported when the default one provided serves fine for large majority of use cases?

Because it's not considerably more bothersome, because the current situation makes your life impossible in some (minority) of use cases and makes people that hates singletons like me cry :sob: .

:wink:

[reduz]

What minority of use cases are the problem? Why are singletons a problem?

[mhilbrunner]

Yes, running the client and server in the same process is some additional complexity.
However, for some apps, its really needed. If you are creating a game where everyone can create a server (and you don't host master servers), its really nice to be able to launch both in the same app.

Otherwise, you'd have to tell your customer "Uh, you need to first start this other application in the background and keep it running". There is a reason Unity supports it.

For example, both Minecraft and Klei's Don't Starve games do this for multiplayer:
The server is basically always running in the app, and if you play single-player, you just connect as a local client in the same app; if others want to join and you allow it, it just opens your local server up.

Additionally, I really agree with @Faless that decoupling networking from SceneTree a little more would make the code cleaner, too.

But yes, being able to start server and client (or server and multiple client) instances via pressingPlay in the editor would be a nice start.

[reduz]

@mhilbrunner I think you are misunderstanding the situation. Currently it works as you describe, the server is also a client and you don't need to launch two processes. What faless proposes is making the server not be a regular peer but a separate entity, and bundle both it and a client together.

This is how networking in old engines like Torque used to work. It's more complicated and there really isn't any advantage.

[mhilbrunner]

@reduz Oh, sorry, then I retract the above :) Are there docs on this somewhere? Still, launching multiple Godot instances via play would be cool.

[reduz]

@mhilbrunner Yeah, definitely need to add a way to launch and debug multiple instances

[Calinou]

@Faless's desired behavior could be implemented by making the client start a dedicated server, which will be possible once the headless server platform is re-implemented. That said, it's a convoluted way of doing things as the workflow @reduz mentioned works just as well in practice.

[Faless]

@Calinou @reduz I'm not talking about changing the current behaviour, I'm asking for something like this (i.e. moving the network code from SceneTree and Node to a MultiplayerAPI class, while allowing for customizations when needed). That stubby patch is fully compatible with the current system, you can try with the multiplayer bomber demo. (Although I have to say I'd prefer to see all net related signals and functions moved from get_tree() to get_tree()->get_multiplayer_api() which will also allow for a single point of documentation instead of having it amid SceneTree functions).
At the same time it allows gdnative/gdscript/whatever to customize it. In ~100 lines more, mainly headers and bindings.

See customization with gdscript here: multiplayer_demo.zip (combo.tscn is the customized version, standalone.tscn shows the compatible version).

[reduz]

@Faless I understand perfectly what you want to do, but the problem is not the amount of lines of code required to implement your approach, but the approach itself. Your arguments are solely about personal taste, not limitations on the current design that could justify making it more complex.

It's not a "this type of game can't be made with the current approach" or "this type of game is more difficult to do with the current approach". Your argument is is simply "I would like to run server and client on the same game instance", which sounds cool but in practice it's pretty useless. Then you shift your arguments to pure distaste for singletons, which is not really an argument with technical merit. Afterwards you try to make focus on the documentation, which is still not really a problem.

To sum up, I feel you are proposing something that doesn't have any practical use (you seem to be coming up with different reasons for it every time, which is even less convincing to me) and is more confusing than the current approach for no reason.

Please understand you are not giving me any meaningful reasons to change the current approach.

[Faless]

I haven't changed reasons, the rationale has always been the same: allow
customization from scripting language (either for creating client/server or
protocol extensions or whatever) while having better organized code.

You keep saying it's a more complex approach, I fail to see that, as it
won't change anything for people who uses the get_tree() system (as I
wrote, fully compatible with the current demos).
This would only allow for more customization IF and only IF a gamedev want,
at practically no cost.

I don't see why we shouldn't allow that when it actually changes nothing
for other regular users and makes the code more structured.

But alas, I feel I tried hard enough to convince you, I guess some things
just can't be changed.

On Dec 25, 2017 10:54, "Juan Linietsky" [e-mail] wrote:

@Faless https://github.com/faless I understand perfectly what you want
to do, but the problem is not the amount of lines of code required to
implement your approach, but the approach itself. Your arguments are solely
about personal taste, not limitations on the current design that could
justify making it more complex.

It's not a "this type of game can't be made with the current approach" or
"this type of game is more difficult to do with the current approach". Your
argument is is simply "I would like to run server and client on the same
game instance", which sounds cool but in practice it's pretty useless.

To sum up, I feel you are proposing something that doesn't have any
practical use (you seem to be coming up with different reasons for it every
time, which is even less convincing to me) and is more confusing than the
current approach for no reason.

Please understand you are not giving me any meaningful reasons to change
the current approach.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/13947#issuecomment-353855955,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABnBbuovIFMsYwYp04mMTrSKvY6BNS6sks5tD3DMgaJpZM4Q0IbV
.

[reduz]

@Faless I still think you don't understand my point of view. Let's agree on the following:

• The feature you want does not need to be used often, in fact it's for rare common cases.
• The feature you want, (be it slightly or a lot) adds complexity to the current system

As such, what I propose is making sure you have elements that would allow you to implement something like this, externally. The only thing that comes to mind is that you may want to reuse the rpc functions. If so, it may be possible to add some abstraction in the middle that allows you to intercept those.

But what is clear to me is:

• I don't see any reason to make this core, as it's clearly more code and more complex API for a highly corner case.
• I don't see any reason why this can't be implemented externally to Godot, via GDNative. If hooks are missing to allow you reuse more API, this can be added.

[Faless]

The feature you want does not need to be used often,

Agreed

The feature you want, (be it slightly or a lot) adds complexity to the current system

I totally disagree. No changes at all from a gamedev prospective, from an engine prospective if anything it lowers the complexity, as it keeps the same algorithm and protocol while moving all network related variables and functions in a single place which means easier to debug, easier to maintain.

I don't see any reason to make this core, as it's clearly more code and more complex API for a highly corner case.

Agreed, no need to make this core, just expose the functions to allow users to do it via script, since as of now it's IMPOSSIBLE to do, unless you want to copy paste those 800+ lines of code from SceneTree and Node (and patch them).

I don't see any reason why this can't be implemented externally to Godot, via GDNative. If hooks are missing to allow you reuse more API, this can be added.

That's exactly what the diff I linked does. Group functions, expose them so they can be used from GDNative/GDScript. No change other than that! (i.e. no protocol change, no user API change, full compatibility with current system, just grouped and exposed code)

[thimenesup]

I agree that we could have the ENet lib a bit more exposed, for example having raw access to the peer IP and port would let you build your own match discovery and/or NAT Traversal/Punchthrough

[Nodragem]

As a gamedev, I think the major points in your original todo-list are:

• Support for launching multiple instances of the game client / server (make testing/debugging our multiplayer games easier)
• Support for movement interpolation & reconciliation (make easy to optimize the networking for a fast paced action game or for a turn-by-turn game)

@reduz seems to argue that the second point is "game specific"... that is true, but it is unclear whether this is an argument against implementing it?? If so, I would say that rendering in 2D or 3D is game specific and you offer both.

Anyway, thank you all for your work, can't wait to see the next improvements to the multiplayer support.

[Neicul]

But discovery, interpolation and host, we can provide a script in the Asset Library, given there isn't a single way to do it.

That would be great, to at least give a starting point for inexperienced network programmers like me 🤓

[kyledayton]

I would like to see options in the editor to simulate latency, packet loss, jitter, bandwidth restrictions, etc. Being able to easily test various network conditions locally would be immensely useful.

[Enhex]

@kyledayton There's a standalone tool for that called Clumsy:
https://jagt.github.io/clumsy/

[Malkverbena]

Not against channels, but will not work for blutooth/online network on Google/Apple APIs. If you want to port your game there and you depend on the feature, it won't work.

This is a relevant feature of Enet and very useful.
Would not it be possible to enable the channels for those who need it and disable it for those who do not?
Or maybe expose this in GDScript? Something like "multiplayer_api.enable_channels".

[mhilbrunner]

@Malkverbena Yes this is possible for ENet. Looking into it.

[mhilbrunner]

ENet's channels are now exposed: https://github.com/godotengine/godot/pull/18827

[mhilbrunner]

fales did an awesome job removing lots of duplicated code by unifying the socket implementation across platforms: https://github.com/godotengine/godot/pull/21692

[Nodragem]

does the unification across platforms means that the networking will work for html5 games?

[Faless]

@Nodragem networking in general works in HTML5, high level networking is only supported by the websocket module (only suitable for non-realtime/turn based games).
An amazing work on WebRTC (for real time games) has been done by @brandonmakin during this summer, but it's still unmerged as it still needs some polishing.
We were discussing the possibility of including it in 3.1 as an "experimental feature" to get better user feedback on it, but it might as well be pushed to 3.2.

See #14627 .

[Ploppy3]

The above improvements seems to be pushed to 3.2

Since UPNP is not reliable, what are the alternatives?

[Faless]

The above improvements seems to be pushed to 3.2

@Ploppy3 if you mean WebRTC, yeah, sadly I didn't have enough time to finish clean it up and PR it in time ( :-1: for me) and it's postponed to 3.2.

Since UPNP is not reliable, what are the alternatives?

Not sure I understand you, UPNP is used mainly for discovery (and few other things) but it's not meant as a "transport protocol for user data", so I'm not sure what you mean by unreliable (beside that it uses UDP for discovery, but then again, you shouldn't bother about it, because you are not transmitting your game data with it).

Right now, there is no unreliable high level multiplayer peer for HTML5, only a reliable one: WebSocket (see docs).
So you can already use the RPC system in HTML5 exports, but it's not really suitable for real time games (it's worth noticing that a game like WoW used TCP, so it's still possible in same cases to have close to real time games using reliable transport. although I wouldn't recommend that in general).

[Calinou]

I think @Ploppy3 meant UPnP doesn't work everywhere (e.g. on mobile networks), so it cannot be relied upon in all cases if you want players to be able to host servers.

That said, I think it's a big improvement over not having anything of the sort :smiley:

[Faless]

Oh, I see, well UPNP only makes sense on local networks.
What problem are you trying to solve?
Peer discovery? In that case you need a discovery service of some kind hosted on a public server.
For peer discovery on the internet you need to host your own discovery service,
NAT punch-through? That depends on the PacketPeer you use, There was some work for ENet by another contributor in #16034 then merged in #18666 . But in general that's a hard problem to solve and it's unlikely to be 100% reliable (and you still need your own discovery service publicly available on the internet).

[Ploppy3]

UPNP is known to be unreliable, it won't work for everybody out of the box since some router don't support it or comes with it disabled by default.

Also, I asked for an alternative since I have a problem which seems to be related to the UPNP library used in Godot https://github.com/godotengine/godot/issues/25008

I'm kind of lost at this point and don't know what is the right approach to let someone host a game on Godot without having to do the port-forwarding himself.

NB: I'm new to network

[Faless]

it won't work for everybody out of the box since some router don't support it or comes with it disabled by default.

Of course, because that was finally acknowledged as a security issue and now most routers come with it disabled.

I'm kind of lost at this point and don't know what is the right approach to let someone host a game on Godot without having to do the port-forwarding himself.

You can't with 100% reliability, as you can't with any other piece of software, because there will always be some case where the router/network configuration does not allow it.

[Faless]

You can't with 100% reliability

That is basically why every game I know of tells you which port you need to open to host a match, so if you don't have working UPNP IGD you can manually open them.

[Ploppy3]

@Faless I understand that, though Warframe and Supreme Commander FAF both use UPNP and work without it. I'm asking for alternatives, what are these games solution to this problem?

I see SupComFAF is talking about proxy but why would a server be _magically_ allowed access on your router? I'm quite confused. (see the 'magic' part, that's where I'm lost ;) )

[Faless]

I'm asking for alternatives, what are these games solution to this problem?

Use UPnP, if that fails, (optionally try NAT punch-through, if that fails) ask the user to manually open ports. SupComFAF in fact has a The longer but definitive fix section where it asks the user to manually open ports

[trooper]

Docs for movement interpolation & reconciliation

I see there's a sub-issue opened for adding documentation / tutorials around interpolation/reconciliation, but no issue opened around adding any engine-level support for this. Was this concluded to be a non-goal for the engine (and if so, why)?

Regarding authority - it seems that the current approach is relying on clients to compute the physics and send the results to the server. I still dont fully understand how that would work and how potential conflicts between what multiple clients computed would be resolved. Furthermore, it seems very prone to cheaters / compromised clients.

Is there an issue # to track adding support for relying fully on server to simulate all the physics and having client-side prediction?

Thanks!

[Faless]

but no issue opened around adding any engine-level support for this. Was this concluded to be a non-goal for the engine (and if so, why)?

I guess the discussion is still ongoing, the question is mainly if this can be done in a general way that is useful in common cases. What I've encountered with my RT game is that some interpolations depends on game mechanics (e.g. custom integration), so it's kinda hard to generalize.

The way I do it is to set, in the clients, all the physics objects to static, disable the collisions, attach a Tween node that interpolate position/velocity/acceleration (sent by the server) but also apply the custom_integration function. The clients are then only allowed to send their inputs.

Works very smoothly in my case, but I'm not sure if this could be good as a general approach.

I do not like reconciliation, reconciliation makes the game worse for people on good connections, and it might even help cheaters. It's something I try to avoid unless it's absolutely necessary.
I've been saying this for a while, and gladly I've recently found out I'm not the only one:
https://youtu.be/vPjpiokAUas?t=221

Regarding authority [...] it seems very prone to cheaters / compromised clients.

That depends on the way you implement your network code.

The multiplayer example in the demos is non-authoritative (i.e. peers send their own positions/calls and the others accepts them blindly).

You can do an authoritative server by e.g. having the players only send input state to the server/host which applies them to the player nodes (and is master on all the other nodes too)

You can, if you want, implement other techniques like consensus algorithms, turn-based strategies like in some RTS networking, etc.

The decision is left to the game dev, as there's no fit for all solution.

We'll be discussing networking/multiplayer improvements at the upcoming Godot events, and we'll update the roadmap in regard.

As much as we understand the wishes for an easy-to-use system for some common networking cases, generalizing them is not an easy task, and sadly not always possible.

[raymoo]

19110 talks about some potential man in the middle attacks, but it occurred to me that Godot multiplayer may also be vulnerable to spoofed RPCs (a peer sends an RPC that looks like it came from another peer). This is a more pressing issue because then an attacker doesn't need any special power other than knowing another user's IP address, and then they could do things like disconnect the other player or make them send in-game resources to the attacker. Knowing someone's IP address is plausible since people may both host and join games, and they need to publish their IP address to host.

Protection against this kind of attack wouldn't need a full DTLS implementation (though I think a DTLS implementation would also protect against this). I think it would be enough to include an HMAC of each RPC, using a key (unique for each client) sent from the server to the client in plaintext at the beginning of the connection. Obviously this wouldn't protect against MITM attacks, but an attacker wouldn't be able to forge packets unless they can eavesdrop on the connection.

[Faless]

but it occurred to me that Godot multiplayer may also be vulnerable to spoofed RPCs (a peer sends an RPC that looks like it came from another peer).

There are checks for that.

This is a more pressing issue because then an attacker doesn't need any special power other than knowing another user's IP address.

And you need to be able to spoof your IP address which every legit ISP will not allow you to do, plus ENet has it's own extra session id, and you'll need to know a bunch of other parameters (like path caches, sequence numbers, etc).

I don't think implementing hmac would make sense under those circumstances.

[raymoo]

This stack exchange post (https://security.stackexchange.com/questions/1062/why-dont-isps-filter-on-source-address-to-prevent-spoofing) suggests ISPs typically didn't filter spoofed packets in 2010. Do you have a source reporting which ISPs currently filter spoofed packets?

The session ID in ENet is only 8 bits and the sequence number is 16 bits, so those could eventually be guessed. I am not sure whether they are initialized randomly or not, but if not, they could be guessed even more reliably. Do you happen to know if there is an official specification of ENet's protocol? I had trouble answering these questions to myself since I couldn't find one.

I don't really understand Godot's plumbing that well, so I couldn't say whether path caches would be sufficiently unguessable.

[Faless]

As stated in a more recent answer to that question an increasing number of
ISP have realized the impact of ip spoofing in terms of costs, and are thus
implementing anti-spoofing techniques.

Additionally, knowing the IP of the server is not really enough, you would
also need the IP of the client. Not only that, you would also need to know
the randomly assigned UDP port of the client (if we are talking about enet).

That said, if there's any proof of concept that works by just knowing the
server IP/port let me know.

About enet, the only documentation I know of is here:
http://enet.bespin.org/ , the rest I gathered from the source code.

On Wed, Jun 5, 2019, 18:48 raymoo [e-mail] wrote:

This stack exchange post (
https://security.stackexchange.com/questions/1062/why-dont-isps-filter-on-source-address-to-prevent-spoofing)
suggests ISPs typically didn't filter spoofed packets in 2010. Do you have
a source reporting which ISPs currently spoof packets?

The session ID in ENet is only 8 bits and the sequence number is 16 bits,
so those could eventually be guessed. I am not sure whether they are
initialized randomly or not, but if not, they could be guessed even more
reliably. Do you happen to know if there is an official specification of
ENet's protocol? I had trouble answering these questions to myself since I
couldn't find one.

I don't really understand Godot's plumbing that well, so I couldn't say
whether path caches would be sufficiently unguessable.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/13947?email_source=notifications&email_token=AAM4C3RZXHJE5D64PMKFPBTPY7U6TA5CNFSM4EGQQ3K2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXAKMPA#issuecomment-499164732,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAM4C3T77WQAVNDT4BMMHILPY7U6TANCNFSM4EGQQ3KQ
.

[raymoo]

As stated in a more recent answer to that question an increasing number of ISP have realized the impact of ip spoofing in terms of costs, and are thus implementing anti-spoofing techniques.

An increasing number is not the same as "every legit ISP". Without specific figures it could mean 10% were before, and 25% are now.

Additionally, knowing the IP of the server is not really enough, you would also need the IP of the client.

When I said an attacker knows the IP address of another user, I meant another client. In the same post I give a reason why it's plausible that an unprivileged attacker would know the IP address of another player.

Not only that, you would also need to know the randomly assigned UDP port of the client (if we are talking about enet).

The port number is 16-bit, which could be brute forced. However, the combination of port number + sequence number would be more difficult to brute force if both the port number and initial sequence number are assigned randomly. From Wikipedia I know that Linux has a port number range of around 16 bits, and Windows around 15 bits. I think on at least Linux it is assigned in some random way and probably can't be predicted easily unless the attacker can observe the other client's connection status. I don't know if ENet initializes its initial sequence number randomly, but if it does then I think the attacks I was worried about would take too long to be useful in a game.

About enet, the only documentation I know of is here:
http://enet.bespin.org/ , the rest I gathered from the source code.

I was hoping there was a specification of the protocol independent from the implementation, but thank you for checking.

[Faless]

An increasing number is not the same as "every legit ISP". Without
specific figures it could mean 10% were before, and 25% are now.

A non-complete list:
https://www.manrs.org/isps/participants/
Feel free to go over it, don't expect ISP names there, but bigger
consortiums, e.g. for Italy MiX includes all ISPs I know and some I've
never heard of.

When I said an attacker knows the IP address of another user, I meant
another client. In the same post I give a reason why it's plausible that an
unprivileged attacker would know the IP address of another player.

You mean the host+join scenario? Well, in that case, the join part is most
likely on loopback, which is pretty hard to spoof.

I don't know if ENet initializes its initial sequence number randomly, but
if it does then I think the attacks I was worried about would take too long
to be useful in a game.

Now that I better think of it, sequence number is only added to reliable
and unreliable-ordered packets, so we can't really count on that.

I would still prefer to be sure that this is actually exploitable in a real
case scenario before adding more code, but in any case, if we are going to
implement it, it should be limited to the ENet peer, not affecting the
whole MultiplayerAPI.

[raymoo]

A non-complete list: https://www.manrs.org/isps/participants/ Feel free to go over it, don't expect ISP names there, but bigger consortiums, e.g. for Italy MiX includes all ISPs I know and some I've never heard of.

Thanks. This doesn't really give a proportion, but I do recognize some large US ISPs when I searched.

You mean the host+join scenario? Well, in that case, the join part is most likely on loopback, which is pretty hard to spoof.

I mean where a player sometimes hosts a game for other people to join, and sometimes joins other people's games. If they advertise their games in (for example) a Discord server, other players may know their IP address from previous interactions.

I would still prefer to be sure that this is actually exploitable in a real case scenario before adding more code, but in any case, if we are going to implement it, it should be limited to the ENet peer, not affecting the whole MultiplayerAPI.

Does this mean it wouldn't affect RPC calls? Or do you just mean that the code changes should be localized to the ENet peer?

[Faless]

>

I mean where a player sometimes hosts a game for other people to join, and
sometimes joins other people's games. If they advertise their games in (for
example) a Discord server, other players may know their IP address from
previous interactions.

It seems to me it's a case that requires some quite special conditions and
a lot of effort from the attacker, with limited chance of success, and
little to gain.
I wouldn't worry too much about it, as we are talking about games here, not
mission critical systems.

Does this mean it wouldn't affect RPC calls? Or do you just mean that the
code changes should be localized to the ENet peer?

That it should be localized to the ENet peer and made optional. Other
multiplayer peers (websocket and upcoming webrtc) don't need it, making
that a waste of precious bytes and computational power.

>

[raymoo]

It seems to me it's a case that requires some quite special conditions

It's a pretty common situation for some types of games, such as fighting games without matchmaking, where people typically post their IP address and invite people to join.

and a lot of effort from the attacker

It only requires the effort once to be reused by the attacker, and maybe other attackers who obtain the exploiting program from the original attacker.

with limited chance of success,

If there's a command that doesn't require a sequence number, and is a valid command at any point in the game (such as a client notifying disconnection), then it could be (I am guessing) exploited reliably.

and little to gain.

I don't think this is true, at least for moderately-successful games. Some potential rewards (using the example of disconnecting a player):

• They can cheat in games by disconnecting their opponent(s)
• They can experience schadenfreude through imagining their opponent(s) being angry
• They can gain notoriety for the exploit
• They can sell the exploit to script kiddies
• They may be able to generalize the exploit to many Godot games, which extends the above incentives to those other games.

That it should be localized to the ENet peer and made optional. Other multiplayer peers (websocket and upcoming webrtc) don't need it, making that a waste of precious bytes and computational power.

Sounds good to me.

[Faless]

>

If there's a command that doesn't require a sequence number, and is a
valid command at any point in the game (such as a client notifying
disconnection), then it could be (I am guessing) exploited reliably.

Well, I tried to explain you why this is not a trivial thing to do
(filtering, sequencing, session IDs, ecc), but since you prefer making
assumption and guesses that it's easy to do, there's clearly no point in
arguing, as you clearly made your final decision already.

Sounds good to me.

While I still don't think it's going to be much useful, feel free to open a
PR, I'll make sure to review it.

[raymoo]

(filtering, sequencing, session IDs, ecc), but since you prefer making assumption and guesses that it's easy to do, there's clearly no point in arguing, as you clearly made your final decision already.

I thought you said that not all packets used by Godot had sequence numbers? And session IDs are only a byte and probably not random.

[Faless]

But you assumed that no sequence was used for "disconnect" messages, and
you are now assuming that sessionId is not random

On Tue, Jun 11, 2019, 15:50 raymoo [e-mail] wrote:

(filtering, sequencing, session IDs, ecc), but since you prefer making
assumption and guesses that it's easy to do, there's clearly no point in
arguing, as you clearly made your final decision already.

I thought you said that not all packets used by Godot had sequence
numbers? And session IDs are only a byte and probably not random.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/13947?email_source=notifications&email_token=AAM4C3V66PNWILGEL2WNOO3PZ63SXA5CNFSM4EGQQ3K2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXNMJII#issuecomment-500876449,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAM4C3S45SQI776365B75ALPZ63SXANCNFSM4EGQQ3KQ
.

[organicpencil]

It'd be cool to have an encryption switch for the high-level API which would solve 2 problems at once. But I'm not using it, so don't expect a PR from me :]

[Faless]

>

It'd be cool to have an encryption switch for the high-level API which
would solve 2 problems at once. But I'm not using it, so don't expect a PR
from me :]

Yeah, encryption would be much more useful for a general case, and I'm
planning to investigate that (again, at ENet level) once I implement DTLS
support in Godot.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/13947?email_source=notifications&email_token=AAM4C3UTTGDKJYCSKWS3JQLPZ7HAXA5CNFSM4EGQQ3K2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXNWYTI#issuecomment-500919373,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAM4C3SJFCWOGCAZNNAAN43PZ7HAXANCNFSM4EGQQ3KQ
.

[raymoo]

But you assumed that no sequence was used for "disconnect" messages

ENet itself allows receiving the ENet disconnect packet as unsequenced. It looks like it has some sort of check (https://github.com/lsalzman/enet/blob/0eaf48eeb0d94a18d079378d8b76d588832ce838/protocol.c#L478) in unsequenced packets, but seems like it would accept a large range of values, so is probably meant for some other purpose than security. In particular it looks like if the command had unsequencedGroup set to 0, it would always be accepted (I could be wrong).

you are now assuming that sessionId is not random

I looked at the code for determining session IDs (https://github.com/lsalzman/enet/blob/0eaf48eeb0d94a18d079378d8b76d588832ce838/protocol.c#L340, https://github.com/lsalzman/enet/blob/0eaf48eeb0d94a18d079378d8b76d588832ce838/protocol.c#L340) and I believe it starts at zero for each peer and counts up with extra connections. I'm not sure, but in any case it looks pretty deterministic.

[nxrighthere]

The only realistic and well-known vulnerability in the ENet is the Blind Masquerade, which is explained in RFC 4960. We solved this by using four-way handshake and cookies. A similar attempt to solve this was done in this fork and further commits there, but no idea if his implementation works in practice.

For everything else we use encryption based on the Noise protocol.

[Faless]

In case someone is interested, this service provide spoofing tests and stats:
https://spoofer.caida.org/summary.php

[raymoo]

These graphs measure how many client addresses are capable of spoofing as a different source address. The portion is low but significant (a bit smaller than a quarter), so an attacker whose ISP performs filtering can look for a VPS in a region or behind an ISP that doesn't filter source addresses. Thanks for finding these statistics.

I referenced the abstract for the paper for that project for this post: https://www.usenix.org/legacy/events/sruti05/tech/beverly.html

[Razzeeyy]

Ability to kick/disconnect peers as the server

I believe this should be in an API for ~MultiplayerAPI~ NetworkedMultiplayerPeer not the specific peer implementations. Because it really feels inconsistent and awkward to cast to specific peer implementation, since the rest of the API pretty much would expect this kind of functionality to be present.

[Jummit]

The second issue was closed(Method to get networking stats): https://github.com/godotengine/godot/issues/8763#issuecomment-577178610

[Calinou]

Note: For any items not having a proposal open yet, please open one proposal per feature (don't lump features together).

---

Feature and improvement proposals for the Godot Engine are now being discussed and reviewed in a dedicated Godot Improvement Proposals (GIP) (godotengine/godot-proposals) issue tracker. The GIP tracker has a detailed issue template designed so that proposals include all the relevant information to start a productive discussion and help the community assess the validity of the proposal for the engine.

The main (godotengine/godot) tracker is now solely dedicated to bug reports and Pull Requests, enabling contributors to have a better focus on bug fixing work. Therefore, we are now closing all older feature proposals on the main issue tracker.

If you are interested in this feature proposal, please open a new proposal on the GIP tracker following the given issue template (after checking that it doesn't exist already). Be sure to reference this closed issue if it includes any relevant discussion (which you are also encouraged to summarize in the new proposal). Thanks in advance!