Triplea: Map/Download Versioning

[Cernelius]

By looking at this pull request:
https://github.com/triplea-game/triplea/pull/4501
I was thinking that I don't like the way map updating notification is handled.

Regarding the above, if that would get merged now, anyone who downloaded the map in the last 5 days will receive the info that the map has been updated, while that would be not (necessarily) true. But, in general, even if the mapmaker consistently hurries to correctly push the yaml update notification minutes after the map has been updated, there will always be a time in between where the repository is being uncorrect (either not telling that the map has been updated or telling that it is, while it is not). Of course, it is possible to keep both updates ready, and push one a few seconds after the other one, each time you update a map, but it cannot be expected all mapmakers (or admins, in case of pull requests on unassigned maps) always doing so.

And the above is just the best case. I think so far we have seen an inconsistency, from the mapmakers, in updating or not updating the yaml (several times, I had to follow the process of deleting and redownloading). On top of this, there is the (likely recurring) matter that users see the download version number and the game version number, and wonder if the update went somehow wrong, not seeing the two being the same (a wrong assumption, of course).

So, I think a better way to handle this would be, either:

1) Somehow having the repository automatically telling that a map has been updated when that repository has undergone any kind of change (comprising revert), dating after the last time the user downloaded that map (also getting rid of the somewhat confusing version numbering, substituting it with dating).

2) Having the download number inside the repository, but outside the "map" folder of that repository. Then the yaml would look at that. This would keep the matter as a not constituent part of the map, and the process would be much alike the current one, except that such number change can actually happen exactly at the same time the update is made (rather than, at best, a few minutes thereafter). Additionally, this may encourage the mapmakers being consistent in actually doing that.

3) Having a map version somewhere inside that map main folder (maybe inside map.properties), that can be updated by the mapmaker exactly at the same time as anything else is (in case of pull requests, it should be the one making the pull that updates that too), the repository offering the update choice in case that version in the repository is bigger than the one the user is supposed to have. That would also cover the need of differentiating between map and game concepts, in that if I, for example, make some graphic enhancement to a map, without touching any xml, I would update the map version, but not any games' versions, while currently there is no way to reference such matters (add to this the case of having multiple games for a same map, and possibly adding more game thereafter (the map version would be increased to indicate that now the map has some more games, while the various old games' versions would not change)).

My favourite solution would be 1. I like 3 too, but I fear that the process may be not followed (mainly mapmakers changing games and updating games' version without changing that map's version too).

Point 3 would become my favourite if it would be also feasible to oblige the mapmaker to always change the map version number, each time anything is changed (like nothing can be pushed or merged if it doesn't change that too).

p.s.: For another recent example, you can see here that some maps were updated, 6 days ago, due to the removal of some now unsupported deprecates:

https://github.com/triplea-game/triplea/commit/70b2cc26fee3e7e579f61e535af17521946dab45

but some of those maps, for example this one:

https://github.com/triplea-maps/another_world/pull/1

had the changes applied over 3 months beforehand, thus somebody that downloaded that map 3 months ago would now receive the notification that the map has been updated, while the update already happened before he downloaded it. Now, this particular example is very unimportant, since anyways the removal of those useless elements has no functional effect at all, but to make the map unreadable by outdated engines, but I'm just saying such process will never be very self consistent until it is either automated (point 1) or somehow integrated in the map's repository itself (point 2 and 3).

p.p.s.: If this gets done, or it is going to, I'm available for updating the yaml, as to remove all occurrences of "version:", or whatever else this matter may need there. Please, let me know (I don't want to do it for nothing).

[DanVanAtta]

Now, this particular example is very unimportant, since anyways the removal of those useless elements has no functional effect at all, but to make the map unreadable by outdated engines

I think this might be incorrect. The version number in the triplea_maps.yaml file is only for update notification. All it does is trigger an out-of-date warning and puts the map in an 'update' column in the download maps window. In terms of engine compat, this has no effect. The value used by the engine for whether it can read a map or not is the min engine version found in the XML itself. I think there lies the problem, I'd agree we should handle things differently.

I want to first ensure that the version numer in triplea_maps.yaml is well understood to be somewhat 'weak', it is just for update notification and classification on the download map.

[Cernelius]

I should have said "unreadable by updated engines". What I meant is that people that would not update those maps, while having the new release without the "costPU" feature, will be unable to play those games. Anyways, very unimportant; was just a minor example.

---

Since here I'm trowing suggestions without an idea on their feasibility, I'll propose a better alternative to point 1, that would be practically the same:

1. 
   
   1. Instead of the download version nuber in the yaml, reference and use the commits number of that repository.
   
   

For example, currently this map/repository:

https://github.com/triplea-maps/total_world_war

has 48 commits, thus the download number would be 48, and this would work just like if the mapmaker would update the yaml at version 48, but automatically and immediately.

Practically, this is just an alternative to what I already said for number 1, with "48" instead of "Jan 9, 2019".

I actually would prefer this (commits number over latest commit date).

Again, no clue if any of this is even possible at all.

[DanVanAtta]

Context / How things Work Today

3 Version Number in Play

• map version found in map XML
• game engine min version found in map XML
• download version in triplea_maps.yaml

How the engine knows about maps on disk

• Engine searches for any .properties files in: $HOME/triplea/downloadedMaps

EG:

$ cat triplea/downloadedMaps/total_world_war-master.zip.properties
#Wed Jan 02 15:01:45 PST 2019
download.time=Wed Jan 02 15\:01\:45 PT 2019
map.url=https\://github.com/triplea-maps/total_world_war/archive/master.zip
engine.version=1.9
map.version=6.0

How the engine knows about maps available for download

Downloads latest triplea-maps.yaml
and compares to what is found on disk.

Compatibility check: new game, local disk contents vs remote player

When searching for installed maps on disk, the game identifies maps by their name. Using a given map name the engine check disk for corresponding XML and can compare version number.

How the engine handles map version on disk vs map version from save game

Save games contain info that is contained otherwise in the map XML, save games simply ignore the map version number in the XML file and they load as much as they can or they fail doing so.

Suggestions

1. Make the Game poly-version map aware, use git tags for map version

This means the game engine could have multiple versions of the same map simulatenously installed

Benefits:

• you could play the 'old' map versions and switch back to a 'new' map version from within the same game client without re-installing the map.
• if map versions can live side-by-side, it's generally 'safe' for the game to download map versions as needed
• opens door for 'immutable maps', if we have a repo+sha then we can always download an exact copy of a map
• When map makers wish to publish a new version, they can use the 'draft a new release' button to create it. Solves the problem for how to manage the problem of wanting to push an update vs an update that is a new version.
• Version information can be read automatically by software given a repo, so we could determine the latest version if we know the repo. Given we can list all repo's in triplea-maps automatically, we can get the latest (or all) version(s) for all maps automatically.

Costs:

• Will need to add features to show disk usage and/or prune maps.
• While there could be some simplifications, some new complexities will aries and there will be a transition cost

How it would work

• Game uses "Name - Tag - SHA" to identify maps on disk and not just "name" alone.
• 'Tag', or 'Map Version', indicates compatibility, and we'd remove the map version from the XML in favor of using it as a 'git tag' value.
• Essentially if we have 'Name - tag - SHA', we'll always know if we have that version of a map, and since we can generally find a repo from a map name we can then use the SHA to know how to download that map.
• Removes the concept of 'download version'

2. Automate triplea_maps.yaml with http_server

Idea here is that we use http-server to query github APIs to get repo information and construct in-memory triplea-maps.yaml and then serve that information from behind an API.

To do this:

• we'd move the snaphost URL and description info to a special file per repo
• picking up from (1), we can read version (tag) and SHAs automatically, we could also detect new repos in triple-maps, and could readily identify when there are new maps or versions
• game client would use the http-server to get download list instead of downloading a copy of triplea-maps.yaml

Benefits summary:

• No more triplea_maps.yaml to maintain
• New repos are added to the in-game download automatically
• Map makers can trigger whenever they wish new versions by using web UI button clicks to create releases (from 'releases' of any map repo, there is a 'draft new release' button that starts that process)
• new versions can be notified to game-clients automatically, the http-server can listen for new version as they are published and game-clients would be notified the next time they check for versions with the http-server.
• can always 'know' if a given version is installed on disk and if not how to get a copy
• no more re-installing maps to use different map versions, can always install the needed map and start playing

[RoiEXLab]

@DanVanAtta Excellent suggestion. I'd actually prefer going with an HTTP server because this approach allows us to change map layouts without breaking compatibility.
Also we're otherwise probably hitting github api limits rather soon.
Note that in any case the local parser system needs some work, because currently it expects the master suffix in the map name and doesn't expect a SHA.
We could also go one step further and have the HTTP server cache frequently used artifacts to solve the download size problem.

[Cernelius]

I don't actually fully understand the proposal, so I think I'm leaving this matter to the developers. I would just suggest any such changes being made only with unanimous agreement of all main developers.

For what I understand, if I'm understanding it correctly, I would agree that is better having a map versioning than a game versioning (if only one of the games of a map is updated, you can increase the map's version and just write in notes when the specific game was updated last time). Anyways, especially if removing game versioning, map versioning should stay somewhere inside the "map" folder.

[DanVanAtta]

@Cernelius this does not change game version, no changes to that.

To maybe help clarify, from a user-perspective, instead of the game engine allowing just one map version to be installed, it would allow multiple of the same. So you could have for example "World-War v1.0.0" and "World War v1.0.1" and "World War v2.0.0" all on your system, and the game engine could auto-find and download the right version when you're playing a save game or a remote opponent that has a version you do not.

[DanVanAtta]

@Cernelius for map makers, things get simpler:

• repos in github-maps will be automatically added to the in-game downloader
• map makers can push new versions whenever they want. ie:
  
  
  
  • a map repo initially would be a v1.0
  
  
  • map maker pushes changes to the repo, they can then 'draft a release' and create a v1.1 from within github. The in-game downloader would then see the new version
  
  
  • map maker could then continue to modify the repository, adding, deleting files, and again 'draft a release', a v1.2. Thanks to github storing snapshots of repositories at all points in time, the game engine could still get all the files from the v1.1 release, even they may have been deleted in the very latest v1.2.
  
  

so in really short, map makers would just need to create a repo, and whenever they 'draft a release' (a button click), they'll get a version of their map as a snapshot at that point in time, and the game will forever be able to download that map version no matter how much the repo is then changed.

[stale[bot]]

The TripleA team regrets this issue was not solved sooner. To keep focus on latest issues and the most pressing tasks, this issue has been automatically marked as stale because it has not had recent activity. If the issue may be closed, please do so. If there are remaining items, we encourage for those items to be resubmitted as new, independent, concise tasks. The strategy is divide and conquer. Thank you for your contributions.