Triplea: Allow arbitrary bot hostnames for lobby?

[RoiEXLab]

There are a lot of people in our community willing to host bot games on their own.
However there are a lot of people that can't do so due to restrictions by NATs created by their ISPs blocking all incoming traffic.
There are some services like ngrok that allow private users to expose their local network to the internet and would allow them to host a game globally.
Besides the idea that ideally such a service would be built-in TripleA (which is a different topic, and more complicated than you'd actually think AFAIK), the current state of the engine does not allow for such features, because the lobby checks for every registering bot server, if the bot IP the lobby is connected to matches the actual hostname provided by the bot and throws an exception if they differ.

The question is: Is this check crucial to security?
This would potentially allow attackers to put arbitrary IP entries into the lobby list and make Users connect to them instead when trying to join a game. Might not be too terrible because we don't just send any userdata to the bots except for the username.
Could become bad if there's a severe security flaw in our bot networking code, but in this case we'd have worse things to fix.

Also the port can already be freely set by the client (for obvious reasons), so in theory someone could already redirect a specific port to somewhere else and potentially harm a user this way.

On the other side there are many benefits. As already mentioned: Bot hosting for everyone, potentially automated user hosting for everyone out of the box because we somehow managed to do all the NAT traversals using some services.
Also related to some more recent discussions: If we ever want to dream about having bot containers that are dynamically being created and shut down (google's JIB tool could become handy for that one), such a precondition would be very important because we can't expect a cloud service to hand out fresh IPs for every new container (VM would be no real benefit from the current situation which isn't too bad), but instead had to rely on some generic hostnames that point to the individual containers.
IPv6 _should_ be a lot better to deal with regarding this topic, due to the missing need for a NAT mechanism and potential direct access to a machine through a router, but I haven't been able to test this, because $ISP decided at some point that my connection doesn't really need IPv6 anymore. 😠

The checking code: https://github.com/triplea-game/triplea/blob/6a36d5906d7d9e0c51546d97e5add81f4fd662ab/game-core/src/main/java/games/strategy/engine/lobby/server/LobbyGameController.java#L70

Thoughts?

[prastle]

Your the rocket scientist but makes sense to me @RoiEXLab

[ron-murhammer]

@RoiEXLab I'm fine with removing the check.

Though my general long term thoughts are player hosted bots don't tend to be sustainable as people come/go too often. Being able to host more of our own bots with a smaller memory footprint and enabling more people to host games are generally a more sustainable direction IMO.

[prastle]

@ron-murhammer I believe @RoiEXLab is reffering to both since in order to host a bot you have to be able to host. He is just calling everything in the lobby a bot. Solving the one problem solves the other if they feel like hosting bots.

[ssoloff]

@RoiEXLab I didn't look at this too closely, so my analysis could be way off, but it seems that assertCorrectHost() is more of an anti-cheating mechanism (e.g. to prevent a maliciously-modified client from sending game state to other nodes) than a mitigation of a general security vulnerability. If that's the case, I wonder if you could just replace the host address value comparison with a reference comparison of the INode instances (e.g. from == description.getHostedBy())? I don't remember if there is only a single INode instance per node within a game. Node is serializable, so that may not be the case.

Also, I'm not sure if removing assertCorrectHost() (or changing it to not rely on the host address) is sufficient to allow NAT-enabled nodes to host. It looks like the testGame() method attempts to open a connection back to the node. Wouldn't that still fail given the scenario you described above?

[RoiEXLab]

@ssoloff I didn't look too closely either.
I just fiddled around with the client side code and tried to send a different Hostname to the server which resulted in an exception to be thrown from this line.
I stopped there because I didn't have any more time back then to setup a local lobby to test this scenario but perhaps @veqryn has any more insight on this?

[DanVanAtta]

@RoiEXLab as an idea, what if the lobby instead of crashing on mismatch instead chose to report/use the real IP. If I'm not mistaken, in that case the IP listed is the one you would connect to as a user, would that solve the problem of:This would potentially allow attackers to put arbitrary IP entries into the lobby list?

[RoiEXLab]

@DanVanAtta while it would solve the problem you quoted, we wouldn't gain anything from changing it.
The goal was to allow people to use "aliases"/port mapping servicea to redirect to their games even through NATs

[DanVanAtta]

I feel like there is something I'm missing.

My understanding is lobby gets a message from bot on connect where bot says "this is my hostname".
Lobby then checks this matches. If not we throw an error. My understanding is that with forwarding the hostname check would not match.

So my suggestion is instead of throwing an error, lobby would instead ignore what the bot reports, and just use it's verification check. In such case, bot probably does not need to send hostname at all.

Apologies if I'm confused and offtopic, please let me know where my understanding drifts if you would.

[RoiEXLab]

@DanVanAtta Your understanding is correct.
However what I want to be able to achieve is that is exactly that a bot could specify its Hostname in order to enable users behind ISP NATs or similar to hosts headless games by themselves.
There are several services (with a free plan for non commercial private use) that offer software that basically creates a local webserver on your machine, and redirects external requests to some auto-generated subdomain to the local machine, no additional configuration required.

[DanVanAtta]

Sorry for dropping the ball on this @RoiEXLab . That sounds interesting, I'd like to know more. When you have the chance let's discuss further. My thinking has been we would simplify/shrink our bot code to the point where it would be easy to have it be or replaced by a very simple network forwarding service. Being able to transparently spin up a bot and host a game would be huge for TripleA.

[RoiEXLab]

@DanVanAtta
I'll try to summarize my Idea from memory:
Basically the biggest challenge with hosting is to get a puplic ip + port to point to your local machine and thus allow anyone to join it.
There are free (with advanced paid plans as well of course) services out there that basically offer a relay server to use for everyone using the service.
So in order to expose let's say your port 80 to the web regardless of your ISPs network structure, you download their tool, run it and specify a port number (and username, password etc.) and it gives you a generic domain other people can simply connect to, which ultimately redirects to your local server.
However because the lobby checks if the host the bot "pretends to be" matches the other end of the TCP connection, specifying such a generic domain won't work.
By removing this server-side check we could allow power-users to pass a custom domain the bot then sends to the lobby, users are then going to connect to.
Of course, if this domain malicious, that could potentially be dangerous, but now that I'm thinking about it, this could happen already if someone modified their bot client accordingly, so not really an argument there.

The biggest benefit of this all is that this would allow the bots to be compatible with serverless infrastructures/be independent of the actual server configuration/location in the internet.
We could create a docker container or something containing all the dependencies, which also allow us to put them on AWS or Google Cloud or something to spin them up with a single click when we feel like there's not enough of them, and shut them down this way as well.

[RoiEXLab]

@prastle You should be able to test this feature now using the latest pre-release (1.10.14192) (if the staging lobby is updating itself which I'm assuming, but aint sure about) for yourself on the staging lobby.
(Note that you have to modify all 3 settings including the new HTTP override for the engine to actually recognize the settings)
In order to use this feature using your script, you'll have to add 2 things to it so the last part looks roughly like this:

java <some stuff that is here already> -DcustomHost=someurl.com-DcustomPort=1234 -jar bin/triplea.jar <insert all other stuff here>

It's important that the new properties are listed before the -jar part, otherwise this won't work.
If you are planning to use ngrok (which works and supports multiple hosts at once in theory), you'll have to replace 1234 with the port number you get when running .\ngrok tcp 3300 (where 3300 is the actual port number of your bot that is configured in the script somewhere else) and someurl.com with the provided ngrok.io domain.
I haven't actually tried running multiple instances, but according to the official docs it works even with the free pricing, just not as simple as running the command multiple times. (You need a config file AFAIK).
PM me on the forum if you have problems with something.

[RoiEXLab]

@DanVanAtta
It looks like there's something odd with the pre-release lobby. Here's a recent log snippet:

Feb 23 19:00:12 PreRelease systemd: Started TripleA Lobby Server.
Feb 23 19:00:12 PreRelease systemd: triplea-lobby.service: Changing to the requested working directory failed: No such file or directory
Feb 23 19:00:12 PreRelease systemd: triplea-lobby.service: Failed at step CHDIR spawning /home/triplea/lobby/1.10.13574/run_lobby.sh: No such file or directory
Feb 23 19:00:12 PreRelease systemd: triplea-lobby.service: Main process exited, code=exited, status=200/CHDIR
Feb 23 19:00:12 PreRelease systemd: triplea-lobby.service: Failed with result 'exit-code'.
Feb 23 19:00:12 PreRelease systemd: triplea-lobby.service: Service hold-off time over, scheduling restart. 

[DanVanAtta]

@RoiEXLab yes, the issue can be observed here: https://gitter.im/triplea-game/server, been busted for a little while. Reproducing on the server and it's a file not found for lobby properties.

[prastle]

@DanVanAtta If we could get this working it would bring hosting back for all? Just a noob ?

[DanVanAtta]

@RoiEXLab a question about:

There are free (with advanced paid plans as well of course) services out there that basically offer a relay server to use for everyone using the service.
So in order to expose let's say your port 80 to the web regardless of your ISPs network structure, you download their tool, run it and specify a port number (and username, password etc.) and it gives you a generic domain other people can simply connect to

Would the IP of the 'generic domain' match the publicly visible IP address of the host? For example:

• client sends request to server to host
• server responds "here is the IP address your request came from"
• the client gets this IP address back and then "reports" and uses it as its IP address, so when for example in the lobby game listing, that IP address will be reported as the one to use.

To be clear, this is all done by IP address and not hostname. I'm wondering if that jives just as well with the service you referred to.

[RoiEXLab]

Good point, totally forgot this one.
Not sure if this is still working without problems currently after all the networking changes you did in the last weeks, but basically a headless host connects to the lobby and gives the lobby an alternative Hostname.
The lobby then displays that one in the listing.
I don't recall exactly what checks are in place to prevent misuse, but basically that's it.

[RoiEXLab]

So to answer your question: No the domain is completely independent from the real public ip, it points to a relay server of the service your local application connects to and forwards all traffic from there

[RoiEXLab]

Ok, dug a little bit deeper into this and as I expected the functionality was partially broken in https://github.com/triplea-game/triplea/pull/5281/files#diff-c4c5746f01efb7b92c9a668b1523eef5

I wasn't able to exactly test this because I really need to redo my local postgres DB at some point in order to be able to start a local lobby again, but I had a deeper look into the code what confirmed my intuition.

Basically LobbyGameController has a mapping that maps any game id to a GameDescription that holds all the information the client needs to connect to a headless host. The "hostedBy" INode in GameDesciption still would hold the "remote relay hostname + port" newer clients (after #4673) would use to connect to a bot, so from a clients perspective the system still works as intended.
However here's what I didn't think of during the review of #5281:
I added another mapping to LobbyGameController that maps the "confirmed" TCP partner of any TCP connection to the lobby to a game id, in order to ensure game entries could only be modified from the node it was initially created. The reason this change had to be introduced is because after my PR, we could no longer reliably compare the host in GameDescription with the TCP partner because they could now officially differ.
So when you stopped looking up the GameDescription, you brought back the assumption that the TCP partner is also the provided hostname, which will cause the connectivity check to fail when the "custom host" option is the only option that can be used in a users environment.
-> So if the connectivity check fails the headless host will shut itself down, which is why this feature is partially broken, if we just remove the System.exit(1) call it should just continue to work, even if we get an error message.

My thoughts

This is kinda an arkward situation to have this bug:
I mean you already built a new backend for this type of thing in #5252 that obviously doesn't provide this functionality.
However in the long run it shouldn't even need this feature if we're planning to move the hosting load to the clients.
Because we don't know when exactly this might be the case it will probably be worth to "fix this bug" until we are at this point. An easy "fix" to this (if my assumptions are correct of course) would be to just disable the connectivity check if we set the properties, assuming that the users know what they're doing and because I'm not entirely sure if it will fail if the user messed up local config and the relay server can't forward anything, but the lobby or the client can connect to the relay server.
However if you think a clean fix would be desireable because it helps detect user errors and makes our lives easier supporting such a feature in the forum, then it might be worth replicating this behaviour in the HTTP lobby, which shouldn't be that difficult, just additional work.

I hope this clarifies a lot of things. If you still have questions about something, don't hesitate to reach out to me, I'll try to explain anything as good as I remember.

[DanVanAtta]

@RoiEXLab appreciate the response, useful and I'll need to see how things shape out a bit before weighing in further.

Re: DB

Notes are here: https://github.com/triplea-game/triplea/wiki/Local-Build-and-Testing
Should be easy, destroy your docker DB container, run ./build_docker, and then run ./launch_db, ./load_sample_data can be re-run after integ tests to restore a sample data set, 'launch_db' will automatically load sample data.

[prastle]

@RoiEXLab @DanVanAtta First of all Do we even have functioning scriptZ? If so I don't mind testing again once we have a functioning test lobby.

[DanVanAtta]

@prastle AFAIK yes, but I don't think the property in question was ever added to it. Have you tested this functionality, a custom hostname, before?

[RoiEXLab]

Likely he didn't, because it requires a compatible lobby which doesn't exist atm

[DanVanAtta]

Testing the bot launching scripts will be of use when we are ready nonetheless. @RoiEXLab do you by any chance have a test-case lined up somewhere to verify the custom host address feature?

[RoiEXLab]

Not really unfortunately, but I should be able to create a full guide to check this today.
It could get tricky automating that but it should allow you to reliably test the scenario until then

[DanVanAtta]

@RoiEXLab was consensus reached on this? Will we require bot hosts to start with 'bot' going forward?

[RoiEXLab]

@DanVanAtta

Will we require bot hosts to start with 'bot' going forward?

This was not the question this issue discussed. We were talking about hostnames, not actual bot names

[DanVanAtta]

Would you mind stating the outstanding discussion points @RoiEXLab ?

[RoiEXLab]

Previously the lobby just used the public ip of any hosted game to redirect everything to that wanted to join.
Then I changed it so that the hoster could define a separate public ip which may be used to point to a relay service, or some malicious software, we wouldn't be able to tell.
This issue discussed if we wanted to take this risk. (Actually the client itself could be malicious so not too much changes)
You then reopened it to ask about implementation details because you were about to migrate this component to HTTP.
We'd have to test if this currently works.
If it does this can be closed once again, otherwise we'd need to fix it.

[DanVanAtta]

Actually the client itself could be malicious so not too much changes

Exactly, the security is based on us not allowing dangerous XML code and for the socket client/protocol to disallow command injection to take place. Ensuring the publicly visible IP address matches does not really change that.

We will need to consider the case where a user could set an IP address that refers to one that is already in use. Offhand I don't think that will be a problem.

If it does this can be closed once again, otherwise we'd need to fix it.

Better to create a clean, new issue and not track everything related to this feature to the one issue. I created a card in the 2.0 release QA column to track this verification item: https://github.com/triplea-game/triplea/projects/14