Mapper: More Revision-Control-friendly file format

[otoomet]

Steps to reproduce

1. create a new file
2. add a few simple objects
3. save it as _.xmap_ file

Actual behaviour

The file will be large, and apparently includes undo information, zoom level, view and such. Could not find a description of the file format.

Expected behaviour

Option to save only strictly _map data_ (and preserve the order and formatting of objects saved). The undo/view information will interfere with RC systems, such as _git_ and create a lot of spurious differences even when the actual map is not changed.

I would like to keep some of the maps under RC and keep the diffs as simple as possible. Ideally there will also be a some sort of graphical diff-merge tool (see #741) on top of the text-diffs.

Configuration

Mapper Version: 0.8.4
Operating System: ubuntu 16

[dg0yt]

• The unstable version already creates additional linebreaks in the regular, compact .omap files, for better use in version control.
• The 'non-map' data is another story. At least undo history can be cleaned manually before saving for version control, via Edit > Clear undo/redo history. This should mitigate the most serious sub-issue.
• Side effects of color or symbol changes are probably the biggest show-stopper for this type of version control at the moment.

[dg0yt]

FTR the request for XML format documentation is tracked in #182.

[otoomet]

Thanks.

Side effects of color or symbol changes are probably the biggest show-stopper for this type of version control at the moment.

I am not sure I understand what this means...

As a side note, #1097 asks pretty much the opposite.

[dg0yt]

When you add/remove a color, it will trigger changes in a number of symbols.
When you add/remove a symbol, or changes symbol order, it will trigger changes in a number of objects.

I don't see a conflict with #1097, because it will affect objects only when they are modified anyway. It addresses revision control at an object level, while git and Co. can only do revision control on a file level. The question is how to implement #1097 without adding too much extra data.

[jmacura]

For the record: I was trying to write a conversion script from .omap to SVG, but found the file structure not really friendly to 3rd party tools. It contains the "noise" (undos & redos, map view), weirdly named elements ("barrier") and, despite being verbose in general, the coordinates itself are not atomic and hard to parse...

[dg0yt]

@jmacura If you want to process the XML, you should better use the .xmap variant, due to its verbosity. The .omap variant is optimized for fast saving and loading, and for small size. That is the reason why the coordinates are not explicit in this variant. Note that SVG uses a similar approach.

The barrier elements are used when a file format change would cause serious trouble (crash, misrepresentation) with slighly older versions of the software. This is easy to handle when reading an XML stream, but could be more difficult with an XML tree.

"Noise" isn't a fair assessment outside of the revision control perspective, at least as long as users expect to continue their work were they left it.

[otoomet]

I think this kind of metadata (like open files, views, zoom settings) is typically stored in a separate file by IDE-s and such. Not sure if this might apply to Mapper.

[dg0yt]

The barrier elements are used when a file format change would cause serious trouble (crash, misrepresentation) with slighly older versions of the software. This is easy to handle when reading an XML stream, but could be more difficult with an XML tree.

At the moment, the barrier element wraps elements which cause problems in older versions of Mapper. I would propose to modify this is as follows:

• Add an optional attribute action to <barrier>.
• If the action attribute has the value skip, readers shall skip the following element if they don't meet the version requirements expressed in the <barrier>.
• If there are multiple elements to be skipped, now wrapped by a single common <barrier>, each element gets an individual preceding <barrier action="skip">.

Once implemented in Mapper, this would be enough to safeguard current versions against misbehaviour with files written by future versions. Other software could handle it as needed, including just ignoring it.

Implementing this change for map reading now, it may become the standard for writing in a later version.

[dg0yt]

I think this kind of metadata (like open files, views, zoom settings) is typically stored in a separate file by IDE-s and such. Not sure if this might apply to Mapper.

My first thought was that Mapper is more like a document editor (e.g. Word, PowerPoint), less like an IDE. (How many map makers do know what an IDE is?) However, when looking at the collection of templates used with a map, the perspective changes. Still, I'm not convinced that yet another file is really desirable.

[dg0yt]

The file will be large, and apparently includes undo information, zoom level, view and such. Could not find a description of the file format.

Saving undo information can be turned off in the settings.

[aberlol]

Saving the same map file (without changes) in OOM on Linux and Android (same version number 0.9.4) does not produce exactly the same .omap file content.

• It seems as some object/tags-child nodes are swapping places. This adds unnecessary changes in the commit history. It would be nice to not re-ordering the tags.

• The template path-attribute is changed, even though the relpath is the same. Is (the absolute) path really necessary to save when we have a working relative path?

[dg0yt]

It seems as some object/tags-child nodes are swapping places. This adds unnecessary changes in the commit history. It would be nice to not re-ordering the tags.

Indeed. This is a QHash structure, and we don't care enough about the order of the keys.

The template path-attribute is changed, even though the relpath is the same. Is (the absolute) path really necessary to save when we have a working relative path?

Unsure. You are looking at Linux and Android which do have a common root directory /. But on Windows, there are drive letters e.g. C:. If you just save the map on a different drive, there is no relative path.
OTOH, I already try to avoid absolute paths in test maps, example maps and symbol sets. The absolute directory layout may give hints about operating system, user name, or map project which might not be meant to be disclosed in some cases.

[aberlol]

It seems as some object/tags-child nodes are swapping places. This adds unnecessary changes in the commit history. It would be nice to not re-ordering the tags.

Indeed. This is a QHash structure, and we don't care enough about the order of the keys.

I see! Then a quick fix would be to order the tags alphabetically every time you save. I understand that the order is un-important to the application... but it would really improve the revision-controlability at a low cost.

(In my case I just removed all tags with regex from the file since I didn't really need them anyway.)

The template path-attribute is changed, even though the relpath is the same. Is (the absolute) path really necessary to save when we have a working relative path?

Unsure. You are looking at Linux and Android which do have a common root directory /. But on Windows, there are drive letters e.g. C:. If you just save the map on a different drive, there is no relative path.
OTOH, I already try to avoid absolute paths in test maps, example maps and symbol sets. The absolute directory layout may give hints about operating system, user name, or map project which might not be meant to be disclosed in some cases.

True, I see your point.

I like the idea posted earlier by @otoomet about creating a metadata-file that contains user specific information (even template settings). This would be a great solution in my opinion.

Addition: The metadata file could then either be gitignored or shared between users if current behavior is desired.

[dg0yt]

Then a quick fix would be to order the tags alphabetically every time you save.

Sorting everytime you save is too expensive. For now, I switched away from QHash to a flat list, assuming that a) the number of items is low, and b) searching for particular tags happens only occassionally (e.g. on import with symbol assignment).

The new data structure is unit-tested, but I would appreciate if it could be complemented by some real-world testing of yesterday's master release.