Fonttools: [TTX] Discussing alternative dump formats for TTX

[justvanrossum]

In #117 we're talking about using using json, and some other formats, like TOML. Let's use this issue for wildly speculating about what dump format could be better than what we have now.

[justvanrossum]

Our TTX format is not so bad, but sometimes feels a little verbose. XML is not really that great for data. But it does give us a lot of freedom.

json is clean and simple, but lacks:

• comments
• multi line strings (useful for language-within-the-language, such as TT instructions or T2 charstrings)

toml is very flexible, but that also means we need to put effort into outputting a suitable variation of the format.

yaml exists.

Are there more standard formats that could potentially be useful for us?

[anthrotype]

I just came across this "style-preserving" TOML library for python: https://github.com/sdispater/tomlkit
Also has an API for writing new TOML docs from scratch. Might be worth a look.

[chrissimpkins]

While not a serialization approach, GH flavored Markdown might be helpful from a practical collaborative data use/sharing standpoint. A number of tools built on fonttools are emitting data formatted in Markdown for the automated rendering to formatted HTML that is available in discussion threads (e.g., Github) and HTML/PDF with platform-specific MD->HTML/PDF tools (e.g., pandoc).

[anthrotype]

yeah, markdown is not an option.

[chrissimpkins]

markdown is not an option

Sorry, this should be a separate issue. I read the title too broadly. I do think that MD as a supplemental ttx data dump format could be helpful to those who are sharing and discussing these data.

[anthrotype]

I think that if TTFont and all the ttLib objects had an asdict() and fromdict() methods, that unstructured and structured the objects to/from primitive python data structures, then the serialisation format could be anything, json, toml, yaml, pickle or whatever.

[justvanrossum]

The more I think of it, the more I think our TTX format isn't all that bad. json is great for simple dict/list-like structures, but its lack of multi line strings, comments and hex notation for integers (!) makes it not a great contender to _replace_ TTX.

Whether toml is an options remains to be seen. The format is quite flexible, but we'd need clever formatting instead of default formatting, and I'm not sure that's possible in a generic way, without knowledge of the data structures.

Either way — Cosimo's suggestion to implement asdict() and fromdict() is an excellent one. We could even have a default generic implementation that serializes the XML structure to a dict/list tree, as Behdad suggests in #117, and write cleaner table-specific versions incrementally.

[behdad]

write cleaner table-specific versions incrementally.

That breaks the format though. But agree with general direction.

[be5invis]

Should we break the proposal into two parts?

• Dump for human edit
• Dump for machine processing

JSON is excellent for purpose 2, since handling it could be very effective.