Terraforming-mars: Bot/AI player

[SliceOfBread]

Has any thought been given to creating a bot (AI) player? I had been working on my own vision for this before I found out about this project.
To start, I think it would need to be best to extremely limit it to something like:
-base + CE (maybe)
-Original board only
-plays 2 player games only
-no draft

Ideas on these limits, bot types and pointers on how/where this might be hooked into the code welcome!

[chosta]

I think it is a good idea and a fun one to work on. My immediate thoughts on this:
There shouldn't be any restrictions on boards, nr players, draft etc. imo. Instead we start with core mechanics. E.g. teach the bot to only play on the board with cities, greeneries and tiles. Once the bot has the optimal algorithms for any map / corporation / game mode - we start teaching it to play with cards.
This though might be extremely tricky but with gradually introducing complexity a reasonable AI players could be done.
We also have deep learning ofc but lets not get ahead of ourselves :)

[SliceOfBread]

You can look at my automatm at https://github.com/SliceOfBread/tm as it is along the lines of what you are saying. It does not play corps (but has some of their features) and does not play cards. I added some rules (bots can buy production, bots can use steel/titanium to help build cities) to make things fun. It support Milestones but not Awards. It is designed to be played 2 bots vs 1 human (who does get to play cards).

The main obstacles I could not resolve were:
-figuring out a value of taking 1 vs 2 actions in a turn
-long term planning toward Milestones (i.e. the bot sees that another player will likely get Mayor so it pivots to Gardener)

My reasons for the limits are:
Without base + CE cards there is little interest for human players.
Original board - this is much more about the Milestones and Awards than the board layout. M&A can have a great effect on strategy.
2 Player - This is the most likely use (human vs. bot) and also the easiest to determine 'which bot is better'.
No draft - this is almost a completely separate skill. I think it just adds needless extra complication.

[vincentneko]

Having bots / AI players would be a great addition.

This is something that would take a tremendous amount of work, and I'm not sure if the result would be satisfying (see how the AI is playing in the official app...)

Anyway, thinking about it, here are some ideas:

• We should try to separate AI code whenever it's possible to avoid interfering too much with core classes (Player, Game)
• Cards with a callback action after being played or activated need an additional method that would be called when the player is a bot, like playAICallback and actionAICallback
• Generic methods like placeOceanAI, placeCityAI, placeGreeneryAI, etc will be needed.
• Each card needs a buyAIScore, playAIScore and actAIscore in order to help the bot decide to buy, play or activate the card.
• Random could be first used to speed up bot development.
• First restrict to core game (Corporate Era) to limit work on first iteration.

I will update this post if I think of other things.

[bafolts]

Isolating the code for the computer based player would be helpful. One strategy for this would be to have some code on the server running on some interval. On some time interval it goes through every running game running in memory and determines if one of the _waiting for_ is available for a computer player. It would then act on that waiting for making choices as a human would have to with those options. This shouldn't bog down the heroku service and worst case we would have to work through a lot of computer player moves. Most times while games are being played the heroku service is sitting idle anyway. It should also help isolate the computer player logic. I don't know much about making the computer player smart but for first steps you could have the computer player take random actions to work out the logic and go from there with improvements. The random actions will help work out the kinks involving being able to determine which action may be better than another. The computer player will need to know the limitations of it's move. Having to make an action only to learn that it wasn't a legal action will be more complicated than if all the options presented are legal.

[SliceOfBread]

@bafolts I really like the 'random action' idea for working out kinks.

Running things on an interval would probably work fine for actual play. On my previous projects, I've always used a type of genetic coding. A bot evaluates things based on weights. You randomly modify some of these weights then play the bots against each other. (With random games, like this, I play each game twice using the same seed but with the bots reversed.) Sometimes only a few games are needed to determine which bot is better. Other times 100s or 1000s are needed. The faster they can play, the faster they get better. An architecture that enables fast play is preferable. (I would assume this would be done on a private server and not heroku). If people have other ideas for bot implementation/learning I am all ears.

[ap3h3ad]

In order to allow Bot developers to access and play the game from outside the server (or at least get the current game state from outside, calculate the best move and show it to the user) - wouldn't it make some kind of API that upon HTTP request delivers actual information about the game?

For example if I connect to my game (via the player id) with some kind of web based application there are methods (e.g. REST based) that share information about the current game, such as all attributes of the current player, the shown attributes of the other players (TR, resources, etc.), played cards and so on and some general information about the game of course (milestones, awards, progress on the TR factors etc.). The log (what happened until now?), etc.

Therefore you wouldn't even have to implement any AI code in the application but make it more convenient for developers to create their own bot (which has a ton of advantages, such as the AI's 'thinking' doesn't strain the actual game server).

[jankeromnes]

@ap3h3ad I very much agree that such an API would be highly useful for Bot players to interact with the game, even if they're implemented outside of the application.

I started looking into this, and found that maybe this API already exists:

• When I start a game, and use one of the player links, I notice in the browser devtools that the player page makes a GET request to the URL /api/player?id=542b258c4f2. The response looks like this:

{
  "cardsInHand": [],
  "draftedCards": [],
  "milestones": [
    {
      "player_name": "",
      "player_color": "",
      "milestone": {
        "name": "Terraformer",
        "terraformRating": 35,
        "terraformRatingTurmoil": 26,
        "description": "Having a terraform rating of at least 35 or 26 with Turmoil."
      },
      "scores": [
        {
          "playerColor": "yellow",
          "playerScore": 20
        },
  /* ... a lot more stuff ... */
  "initialDraft": false,
  "deckSize": 178,
  "randomMA": false
}

• And then when I choose some card in the UI, and click on "Save", I notice that the page sends a POST request to the URL /player/input?id=542b258c4f2 with the following JSON payload:

[["Teractor"],["Viral Enhancers","Kelp Farming"]]

So I think I'll try to implement some sort of proof-of-concept to confirm whether this API can serve as a basis for Bot users to play the game.

Maybe I'll even make some sort of library that you can implement if you want to develop a Bot yourself. That could be useful and fun both for myself (I'd love to implement a Terraforming Mars bot) and for others (would be awesome to eventually have Bot competitions, like the Star Craft AI tournaments, if enough people are interested).

[SliceOfBread]

it would be very useful to have some sort of skeleton code that people can use to develop bots without having to understand too much how all this code fits together.

I have been working on some other projects but hope to get back to this in the not-too-distant future.

[jankeromnes]

Hello again! 👋

Quick update: I've started implementing a proof-of-concept Bot framework: https://github.com/jankeromnes/terraforming-mars-bot 🎁 🤖

It includes:

• This repository as a Git submodule, to automatically start a local Terraforming Mars server when opened in Gitpod

• A helper script node start-game (SERVER) that can start a new game of Terraforming Mars (single player for now). By default, this uses the local server (if available), but it should also work with any server URL

• A "random" bot (source code) that plays random moves (that's not a very effective strategy, but serves as a great starting point & skeleton code for smarter bots)

• A helper script node play-bot --bot=(BOT) (PLAYER LINK) that makes a bot (by default, the random bot) play an entire game (I've only tested single player so far)

The random bot is now able to finish a solo game of Terraforming Mars most of the time, but I've never seen it win yet.

I've added some basic documentation on how to implement your own bot. Please feel free to fork that repository, and/or to contribute your bot! I think I'll soon start implementing my own bot, to see how good I can make it. 🙂

[kevinb9n]

The random bot is now able to finish a solo game of Terraforming Mars most of the time, but I've never seen it win yet.

It hasn't played me yet!

This is so cool.

[bafolts]

If this AI player gets smart enough it would be easy enough to run it on the demo server and allow a player of artificial intelligence to be selected from the game setup screen.

[SliceOfBread]

This is great! There go my plans for the week. :)

[SliceOfBread]

Got a chance to play around with @jankeromnes stuff yesterday. This is great for anyone that wants to code up a bot (and I see Jan is working on a 'quantum' bot). While I might code up something simple, my plan is to evolve a bot and it's not clear to me if/how this will allow me to do that. (I assume for now, this won't be useful in creating my bot and will continue on my previous path for now. Any perspective on that is welcome.)

Last, I worry that if the interface changes, any coded bots will become obsolete. Is the current thinking that the interface will remain static?

[jankeromnes]

Thanks for the great feedback @SliceOfBread!

While I might code up something simple, my plan is to evolve a bot and it's not clear to me if/how this will allow me to do that.

If I'm not mistaken, I think for that you'll need to have some infrastructure in place in order to continuously update your bots' behavior (weights?) and test them by playing several games in a row.

The node start-game and node play-bot helper scripts I implemented could help with the repetitive tests (e.g. start a TM server & quickly play 100 games while keeping track of scores -- something that I'd like to add soon but isn't implemented yet). However, the evolution part (i.e. your evolving bot(s), weights, etc) would need to be custom-made anyway, so you can indeed continue on your current path and disregard what I'm doing.

But please feel free to copy anything that seems useful from my repository, or to request specific "bot infrastructure" features you would like to have (for example, the ability to run 100 games between two bots and show the scores).

Last, I worry that if the interface changes, any coded bots will become obsolete. Is the current thinking that the interface will remain static?

That is indeed a risk. That's why I've tried to make the bot API as generic as possible, and to abstract anything that seems too implementation-specific inside the play-bot.js script.

I believe that minor changes to the game's interface will not be difficult to cope with (by adapting the helper scripts, so that bot code doesn't need to change), but indeed if the game's REST API or internal representations see a massive change, that could require reimplementation of some bot code.

Luckily, terraforming-mars-bot is tied to a specific version of terraforming-mars (by means of its Git submodule) so I could (or rather should) implement tests to ensure that all bots continue to work before upgrading the game engine.

[jankeromnes]

Quick updates on terraforming-mars-bot:

• node play-bot now auto-starts a new game when needed (so it's no longer necessary to give it a fresh player link)
• I've implemented a new bot called "Quantum" (source code, with some "magic values" based on A Quantified Guide to TM Strategy)
• You can now run multiple games in a row with node play-bot --games=10 and it will summarize the scores

"Quantum" bot is still not quite able to win a solo game in 14 generations (I think the closest it got is temp=8/8, oxy=9/14, oceans=7/9), but at least it gets better scores than "Random" bot:

• Random bot (100 games): average 29.03 points (min 20, max 44)
• Quantum bot (100 games): average 50.69 points (min 30, max 72)