Triplea: Redesign map downloads

[RoiEXLab]

So there are a couple of issues opened about this, but I wanted to sum up all of them and present my plans for them.

Problems with the current system

1. Engine compatibility:
   
   
   
   • Currently there's no way to tell for the engine if the downloaded map is compatible with the latest version, so if the proposed map requires a newer engine version, then bad luck for the user, because the engine can't automatically download an older, compatible version although it would be available using github.
   
   
   • The triplea_maps.yaml file contains no information about any map revision, all the engine does is to check if the version number is different (I don't think it's checking if it's actually higher, but I might be wrong about that) and then downloads the map from the link in the file which is just a link to the master zip, so the engine might download a version that's too new even if the version string hasn't been altered at all, so this version string has essentially no meaning at all.
   
   
2. Hardcoded map information:
   
   
   
   • Currently the engine just has a hardcoded link to the triplea_maps.yaml file. If we ever wanted to move that file we couldn't do so because that would break compatibility+ users are not getting notified whenever a new version is available on the website, so they can't even tell their version is outdated.
   
   
3. Multiple Maps per repository
   
   
   
   • Currently a single repository often contains a couple of variations from the original map. This makes sense as all of those xml files often use the same assets, and therefore this reuses them efficiently, but it makes the whole organisation of stuff difficult, because if just one variation makes use of a newer feature, all of them suddenly need an engine update even if they don't actually require it.
   
   

Minor Problems (kinda unrelated)

Currently the map download folder is being located along with user preferences, while the default installation path is system-wide by default, ideally we'd store maps along with the installation, although that might cause problems because of missing admin permissions.
Of course the option to change the map folder should stay for our power users and other folks that want to run multiple (compatible!!! (This solves the issue TripleA had during the version change from 1.8 to 1.9)) triplea installations concurrently.

My Solution to this Problem

A Package Manager!
Not something super fancy like npm, yarn, apt, brew etc. but something lightweight that fits our needs.
The Idea being that we make it accept HTTP requests using something like https://maps.triplea-game.org/maps/:engineVersion which then returns a YAML or JSON array with all the information the current yaml file holds, with the difference it perfectly suits the current engine version.
Ideally this "Package Manager" integrates with github and queries all of the github repos on startup and caches them for "all versions" in a database, and keeps those lists up-to-date using webhooks.
Now this solves the problem of having the engine just download the "newest version", but it still doesn't solve the issue with the multiple map files. I can't think of a solution for this problem, but maybe it would make sense to split the xmls from the rest of the assets, and just treat the assets as a regular skin, and create a repo for it, so map xmls can just reference the revision, which is then being considered by the "package manager"

What are your thoughts about this?
Any Concerns?
I must admit I'd like to work on the package manager instead of the TripleA code to handle all of this, so I'd appreciate if someone would code the client side for this ^^

[DanVanAtta]

Engine compatibility: Currently there's no way to tell for the engine if the downloaded map is compatible with the latest version

I agree this is a problem. Not only this, but we can't make non-backward compatible map parsing changes as we are forced to use the latest parser with any XML. There is no ability to change parsing strategy. This also further impacts us as it means we can't start creating alternative simplified, non-backward compatible parsers. For example, war of relics is 88k lines long, we can fix the map specification so we don't need such long files - but it's really amazingly challenging to do that in a non-backward compatible ways. Because it is a non-starter to break parsing for all maps, we are pretty stuck.

Suggestion to fix: Spec Version

I think a "spec version" would solve this problem for us. The idea is embed a map parser version number in the map file to tell the game engine which parser to load. Based on this value we can load a newer parsing engine, or we can fall back and use a default. This lets us start upgrading the XML schema in non-backward compatible ways without breaking all existing maps.

For an example of this idea, I was experimenting with what it would look like to convert a XML map file to YAML. As part of that I created a fork in how the game parser is loaded, and introduced a spec version to the new YAML format:

• https://github.com/DanVanAtta/triplea/commit/5e1ae21c7e16ae952a9ad9cbb95ce75155c44937#diff-0dff504360f8239d76828118a02ebadd
• https://github.com/DanVanAtta/triplea/commit/5e1ae21c7e16ae952a9ad9cbb95ce75155c44937#diff-1ceac79f6b0e33e6f5e9825280784524
  For a proper spec version for XML's, it would be pretty similar, for now the number the of parsers depending on file extension type is still one-to-one.

[DanVanAtta]

Hardcoded map information: Currently the engine just has a hardcoded link to the triplea_maps.yaml file.

The file is not shipped with game installers, it should be downloaded fresh every time the download window is opened. A hardcoded link to a local file location is a game developer feature so you can test updates without updating the master file. It is a problem that this can't be moved, a known one at least.

Is this actually a problem as described? Beyond the location being fixed, updates should be live after commit and re-opening download window.

Suggestion to fix: consolidated link file

Move all links to one file that is downloaded by client. We mostly do this for lobby server already, so the update here would be to move all links to that file. Eventually, the one file would be the only hardcoded link. This would let us move around any of the other files or links, and would simplify the whole mechanism.

[prastle]

Wow.. I wish I knew more but sounds great!

[DanVanAtta]

Currently the map download folder is being located along with user preferences

This is by design, game engine updates are independent of map updates. When you download and install a game engine, you are not downloading maps. This means you can release map updates or game engine updates without having to release both. The user location for maps is a long running thing, it is so maps do not have to be re-downloaded between game installations.

After we moved maps to github so download speed is 5x faster, improved the UI to allow parallel downloads (don't have to wait and then click to downlaod the next), and also made the game engine download smaller, suddenly those original problems do not feel so bad :)

I would recommend we keep the current behavior.

[RoiEXLab]

@DanVanAtta

The user location for maps is a long running thing, it is so maps do not have to be re-downloaded between game installations.

I agree, however we could easily configure install4j to just keep map files around instead of removing them...

Suggestion to fix: consolidated link file

I believe we already have such a file, although we are not really using it like it: https://github.com/triplea-game/triplea/blob/master/latest_version.properties

Suggestion to fix: Spec Version

Yes, I thought about the semver compatibility syntax, although this would require us to finally alter the version scheme, or to simply transform it to a comoatible version string, however this doesn't solve the issue on what to do if the map is incompatible. Ideally the game engine should be able, using a remote server ("package manager") to find the latest compatible version of any map.

[prastle]

Is It possible to make the engine version in the lobby display example.8824? instead of 1.9?

[RoiEXLab]

@prastle We tried changing the version scheme multiple times now, and every single attempt failed with some sort of compatibility issue. The build number as last digit is really just a hack that's only available on the client side, this means if we wanted to change the behaviour we'd probably need to maked some breaking changes.

[prastle]

k

[prastle]

Just saying until all users have a clue what their opponent is using or they all have the same engine we will always have probs.

[DanVanAtta]

The user location for maps is a long running thing, it is so maps do not have to be re-downloaded between game installations.

I agree, however we could easily configure install4j to just keep map files around instead of removing them...

It was a really long debate to split all files from the game engine. Having the first time download be giant and then small on next versions does not seem like a win.

[DanVanAtta]

Suggestion to fix: consolidated link file

I believe we already have such a file, although we are not really using it like it: https://github.com/triplea-game/triplea/blob/master/latest_version.properties

We actually do use that file. The only problem is that we introduced a bug in the game about a year ago where if we modify it then the game client freezes on start-up.

When we look to do a forced update, we will be modifying that file to trigger notifications to users that they shoudl upgrade.

[prastle]

Anyways all bots are now running 8824
as of today

[prastle]

0 complaints in lobby so far basically i don't think most of them noticed.

[RoiEXLab]

@DanVanAtta

It was a really long debate to split all files from the game engine. Having the first time download be giant and then small on next versions does not seem like a win.

I think I wasn't clear enough here: I didn't mean to include map files with the game engine itself, I just wanted to point out it might be more consistent to just have the default map folder inside the installation directory, so the same maps are shared between users, and not between incompatible versions in case we want to change the default folder to something like TripleA1.9 at some point, so if a user installs TripleA1.10 the map files are strictly seperated.

[DanVanAtta]

Suggestion to fix: Spec Version

Yes, I thought about the semver compatibility syntax, although this would require us to finally alter the version scheme, or to simply transform it to a comoatible version string, however this doesn't solve the issue on what to do if the map is incompatible. Ideally the game engine should be able, using a remote server ("package manager") to find the latest compatible version of any map.

Actually no! It would not be semver compatibility syntax and would not map to the game engine version number. This would be a brand new version number for XML files to tell engine which parser to use.

So for example, in GameLoader we'd have the rough following logic depending on spec number:

null or 1.0 -> `GameParser.java`
1.1 -> `UpdatedGamerParser.java with certain enum flags`
1.2 -> `UpdatedGameParser.java with other enum flags`
2.0 -> `SomeOtherGameParser.java`

So to be a bit clear, the 'spec number' is a version number of the XML, it's like a doctype for HTML, or DTD number for XML.

[RoiEXLab]

@DanVanAtta

So to be a bit clear, the 'spec number' is a version number of the XML, it's like a doctype for HTML, or DTD number for XML.

Makes sense, however I'm concerned this will result in a lot of essentially-almost-never-used classes, unless we explicitly agree that each engine version just supports a small range of those specs.

[prastle]

Yes I understood that seems like a great idea! My question is how do we inform the players that George is using donkey 3621 or older while sue has 8824? or newer?

[DanVanAtta]

I just wanted to point out it might be more consistent to just have the default map folder inside the installation directory, so the same maps are shared between users

I see now, thanks. That could make sense and could have some benefits. I'd recommend to cut a new and dedicated issue for that topic.

[RoiEXLab]

@DanVanAtta
While we're at the GameParsing topic: I'm currently trying to get rid of the reflective access in the game parser.
I came up with a good system that is 100% backwards compatible (in theory, as always 😅) and works similar to the class name mapping the engine currently uses.
This should speed up parsing considerably, especially for large maps.

[prastle]

well what we also need is a way to update a map individually if that the only one that changed that week

[prastle]

instead of parsing all maps to get one

[DanVanAtta]

@DanVanAtta

So to be a bit clear, the 'spec number' is a version number of the XML, it's like a doctype for HTML, or DTD number for XML.

Makes sense, however I'm concerned this will result in a lot of essentially-almost-never-used classes, unless we explicitly agree that each engine version just supports a small range of those specs.

Not conveyed is that we would wind up supporting pretty much all old versions. I would expect a new parser for a new file type. I would also expect perhaps one new XML parser for an agreed upon simplified configuration. I think it probably makes more sense to go directly to YAML with a simplified spec.

The 1.1 vs 1.2 differences would be captured in the game parsing engine:

if( version > 1.2 ) {
  myNewAttribute = 'default'
} else {
  myNewAttribute = readFromFile();
}

[DanVanAtta]

well what we also need is a way to update a map individually if that the only one that changed that week

@prastle we should have that just change version number in triplea_maps.yaml and user would be notified the map is out of date the next time the game engine does a version check.

[prastle]

I am referring to updating a bot engine
Yes hepps is editing the aml file and we will add in next day or 2
roi helped me find
point is
Reloading a bot server with the map shouldn't require updating every map. just updating the specific map that changed.

[prastle]

other than shutting down lobby it is a 4 hr process for all bots.
when really it should be 10 min per server

[DanVanAtta]

@prastle

Reloading a bot server with the map shouldn't require updating every map. just updating the specific map that changed.

Ah yeah, that's an artifact of the download script, it's pretty basic. : |

other than shutting down lobby it is a 4 hr process for all bots.

You actually do not need to restart or stop the bot, the map file can be live-swapped while a game is being played. The updates to file are only read the next time a game is started.

[prastle]

OH
TY
that's a huge help kk
Ill do that next time i just need to close bots for engine update then

[DanVanAtta]

@RoiEXLab
to mostly finish responding to your points:

Engine compatibility: so if the proposed map requires a newer engine version, then bad luck for the user,

Indeed, this is a feature gap, TripleA does not support maps that are built for newer game engines. Maps should not merge such features until we have a forced upgrade release to support it, otherwise we blow up the map for anyone that has not yet upgraded their game engine.

Spec version should help as we'd have a way to know if a file is not supported.

The triplea_maps.yaml file contains no information about any map revision, all the engine does is to check if the version number is different

The version number in that file is only to flag to the game engine when a map should be prompted to user as out of date. It's for notification only, basically the same as the previous version number we used to have. It's only really to notify the downloader, no further usage.

We do have a problem that the download version is out of sync with the versions of each map.

Multiple Maps per repository
Currently a single repository often contains a couple of variations from the original map.

Yup, to boot it introduces a political problem when users don't want 'unofficial' versions to be added so we get forked copies that are copy/pasted.

A Package Manager!
Not something super fancy like npm, yarn, apt, brew etc. but something lightweight that fits our needs.
The Idea being that we make it accept HTTP requests using something like https://maps.triplea-game.org/maps/:engineVersion which then returns a YAML or JSON array with all the information the current yaml file holds, with the difference it perfectly suits the current engine version.

This basically seems like a way to deal with the fact that not all map versions are compatible. TripleA never has had a way to manage multiple versions on the client. It's still a huge problem when a user with map version 1.2 tries to play user with map version 1.3. We're for the most part assuming that those with save games will cope, and that otherwise everyone will upgrade their maps. These are not good assumptions as we dump the task of map compatibility management on the user.

Having a system to tell us which version of map to download for a certain game engine version is somewhat useful, but it still does not solve a number of other problems.

Suggested Fix: make game engine aware of major map versions and track multiple map versions and not just the latest version

This would be a pretty major update, but I think really well worth it so users do not have to worry about map compatibility when playing others and loading save games.

Some things we would need to do:

• Have the triplea_maps.yaml list the download URL of each major version of a map. So url would become a list of key/pair entries:

map_name: example
url:
  - 1.0 : http://..../commitSha
  - 2.0 : http://..../commitSha2

• client side we have a good bit of work, how to manage these on file system, load them, UI work to select between different versions.

[prastle]

WOW! So basically no new maps can be released until you guys have a new stable and force all to upgrade?

to mostly finish responding to your points:
Engine compatibility: so if the proposed map requires a newer engine version, then bad luck for the user,
Indeed, this is a feature gap, TripleA does not support maps that are built for newer game engines. Maps should not merge such features until we have a forced upgrade release to support it, otherwise we blow up the map for anyone that has not yet upgraded their game engine.

[prastle]

Thus all new map development should stop? Until the engine supports it? No testing? Unless we send space each other like the old days?

[DanVanAtta]

WOW! So basically no new maps can be released until you guys have a new stable and force all to upgrade?

If we want to take advantage of bleeding edge new features and are not backward compatible, yes.

[prastle]

Your the boss but i disagree

[DanVanAtta]

I don't think we are on the same page, it is reality/facts. If you push a map that has a required feature that is not in the current latest, anyone that upgrades their map from download manager will find it broken until they upgrade their game engine.

[prastle]

Correct. Except the current latest is from 12 months ago

[prastle]

Or when ever 7621 was made

[prastle]

i would also like to point out there has never been a forced upgrade since 1.9 from 1.8 . So when should we start working on new maps again?

[DanVanAtta]

i would also like to point out there has never been a forced upgrade since 1.9 from 1.8

If you play 1.8, it should tell you that 1.8 is out of date.

So when should we start working on new maps again?

That's a bit melodramatic : )
Nothing has changed suddenly.
If you have any maps that are waiting on a new feature, then those maps may be waiting. That is the same as it ever was.

[RoiEXLab]

@DanVanAtta Ideally the game engine would just ignore too new maps and just not update to them IMO, thus my suggestion, but the different url mappings could work out as well although this probably makes the file just harder to maintain.

[prastle]

all new maps in past 2 years have used new features pt number 1
no one uses 1.8 because i convinced them to upgrade or they quit playing online pt number 2
all the recent excitement in the forum on map making by all map makers are the changes to the engine in the last few months pt number 3

I would keep going but why bother

[DanVanAtta]

all new maps in past 2 years have used new features pt number 1

@prastle So you're saying that I can download game fresh from website, download all maps from within the game, and then play a map that would error out?

If so, a list of those maps would be useful, something should be done about them.

[DanVanAtta]

@DanVanAtta Ideally the game engine would just ignore too new maps and just not update to them IMO, thus my suggestion, but the different url mappings could work out as well although this probably makes the file just harder to maintain.

Player to player compatibility is the main concern for me. It's okay to let one engine version ignore the latest, but when you have that player playing against another player with an otherwise compatible game engine, then you run into problems. If the game engine keeps track of major map versions, then the player with the newer version could explicitly choose to play a downgraded version. Today you would be stuck if you upgrade, you can't play anyone with that same map unless they also upgrade. This is a real problem when it comes to save games, that is generally the motivation for whether to upgrade or not.

[prastle]

All maps will dl
a few will only work on 7621 or newer
tww and new iron war will only work on 8000 plus if i recall but either way new tww only works on 8824 or newer

[DanVanAtta]

but the different url mappings could work out as well although this probably makes the file just harder to maintain.

:grin: it's about the same as keeping SHA's in the yaml file

[prastle]

This is not the problem btw
The problem is old members refusing to upgrade

[DanVanAtta]

tww and new iron war will only work on 8000 plus if i recall but either way new tww only works on 8824 or newer

This is actually a problem : |
the game engine is not smart enough to do anything other than crash in this case.

[prastle]

had a game crash other night for a long time lobby member
who had upgraded
meanwhile his opponent using 3621 he said

[prastle]

Until you have all on a newer engine 4000 changes plus diff is not back compat

[DanVanAtta]

@prastle this issue is about map versioning, let's please stay on topic.

[prastle]

yes it is
and map versioning means as we just did old map is still avail with the new version. We will add to the yaml something newer than jan2018 engine is needed
;)

[prastle]

if you want to play the new version of course

[DanVanAtta]

Multiple Maps per repository
Currently a single repository often contains a couple of variations from the original map. This makes sense as all of those xml files often use the same assets, and therefore this reuses them efficiently, but it makes the whole organisation of stuff difficult, because if just one variation makes use of a newer feature, all of them suddenly need an engine update even if they don't actually require it.

There are still more problems with this. There is inertia to not modify a map you do not own. This means that one person gets stuck maintaining one file for a long while, or its weird in terms of ownership when a team comes in to update it. We then get the forks of maps that are copy/paste because it is easier than adding to an existing bundle.

The many repositories is nice for maps for a build point of view, but maintaining is challenging. I dont' think it really has picked up very much either.

Suggestion: break up map representation, inroduce 'unit packs', 'flag packs', 'sound packs', and go single repo for the map XMLs

Essentially we would break up teh map definitions so it would reference 'flag bundles', 'unit bundles', etc.. So for example if you download a XML where you are missing a few bundles, the game engine would be smart enough to download them. Then we could group all the XMLs to one repo. Nothing would change when looking at it from the download manager, but instead the engine would be smart enough to also download the right bundles..

This has a number of benefits:

• all XMLs could be in one repo. Easier to manage, easier to create new ones.
• space savings, units, flags, sounds would be re-used between maps
• splits graphics work from map details. A person can create a unit pack by dropping certain images, map makers could then include them by name from any map file and need to copy/paste them in.

[DanVanAtta]

@RoiEXLab this is a deep topic. I've responded to it all I think in some gory detail. To sum up:

• Server-side package manager would not necessarily solve the right problems

Instead I would suggest we would:

• make game client aware of major map versions and manage them so players can play against each other even if one has an updated version
• add a new version number, "spec version" to map definitions that tells the game engine how to parse that file
• redesign how maps are assembled and maintained in repo. Update game engine to be aware of this, and put all XMLs together in one repo and create a second repo that would keep zip files of graphics and audio assets .

[DanVanAtta]

Another far flung idea I would add to the list, update save game format to be unified with core map specification. If we combine this with the bundles, you would not need the original map XML for a save game or a hosted game as long as you have the needed gfx bundles. That would solve our save game compatibilty issue for maps : ) In other words save game XMLs would contain original map data too. I think this would be best done if we converted both to yaml;

[prastle]

@DanVanAtta i like the split thing good idea they are actually trying to build an image repo on the forum because of this. Great stuff. That might work as well.

[prastle]

The only prob i foresee is a guy doing all his own then needs to upload to multiple repos.

[prastle]

instead of one. this will require git hub streamlining

[DanVanAtta]

The only prob i foresee is a guy doing all his own then needs to upload to multiple repos.

yes & no, it depends. Actually, if we do static zip files, we'd probably host them without a repo. you can attach a zip to any github issue, that gives a link, we'd just need that link : )

For local map development, the only difference is that the 'units' and other graphics folders are not in the map folder itself. You'd tell the XML the name of that folder, create that folder in the right spot with the right images. Then when it comes time to publish, zip the graphics folders up, attach to a github issue to get a link, then submit a PR to XML repo with new XML.

[prastle]

Yup your back to GitHubease. it's similar to Chinese :) but ill try to learn :)

[prastle]

Main point is you need to make it easy for all contributors.

[DanVanAtta]

@RoiEXLab next steps I think are:

• do you still think a server side package manager is the way to go? If so, how would it fit in to the larger picture? There are a lot of problems with maps, client-to-client compatibility is my biggest concern, so I'd like to be sure we're addressing the right problems.
• otherwise, given everything, probably pick and choose what we want to follow up on and create independent tracking tickets for reach (perhaps assign a project to this).
• loop in forums, map makers will 100% want to give some feedback

[prastle]

@RoiEXLab @DanVanAtta At the present moment (After 24 hrs) all maps appear to be functioning fine with 8824 in the bots and live. So you can probably use that as a base moving forward. So far (knock on wood!) 0 complaints.

[DanVanAtta]

@RoiEXLab what's the latest? Do you have a plan? need help? Are you still thinking of a package manager?

[DanVanAtta]

I'm a no vote to the package manager, the in-game downloader is already basically that. We do have a lot of problems with map structure. I'm sorry to have buried this topic, but I wanted to give more context on how deep some of the issues go. There seem to be deeper solutions for us to go after.

[RoiEXLab]

@DanVanAtta agreed, closing this in favour of your newly created issues