Cascadia-code: Suggestion: Use UFO format for source storage

[madig]

Description of the new feature/enhancement (with images if possible)

Currently, the font source is provided as a .glyphs file and a TTF binary with VTT code inside. This probably works fine for the designer, but both formats are not made for version control. There is another way: use the UFO format for source storage, a TTX dump for VTT code storage and put that in version control. Glyphs files and VTT binaries can then be generated.

Granted, the UFO format has its share of problems, but it is the only format explicitly made for version control. At my company, we use https://pypi.org/project/glyphsLib/ to round-trip from UFO to Glyphs and back. The UFOs are compiled with https://pypi.org/project/fontmake/, VTT hinting applied with https://github.com/daltonmaag/vttLib/.

Quick drive-by modifications to the UFOs could even be made with free tools like FontForge and TruFont.

Proposed technical implementation details (optional)

You can have a look at https://github.com/daltonmaag/ubuntu and https://gitlab.gnome.org/GNOME/cantarell-fonts/ to see what this can look like.

The workflow is roughly

1. ufo2glyphs CascadiaCode.ufo
2. Work on the Glyphs file
3. glyphs2ufo CascadiaCode.glyphs (creates a Designspace file after the first time, which contains more information and is stored next to the UFO)
4. Look through changes to UFO and commit desired changes.

Compiling the font is done with fontmake -u CascadiaCode.ufo -o ttf (or -m CascadiaCode.designspace). Note that the cubic-to-quadratic curve conversion algorithm used has an effect on the VTT hinting and can completely destroy it. I wrote a hint-transplanter for this reason, have it somewhere. Hinting is injected with python -m vttLib mergefile input.ttx output.ttf && python -m vttLib compile --ship output.ttf, where input.ttx is obtained by python -m vttLib dumpfile CascadiaCode-Regular-VTT.ttf input.ttx.

The VTT workflow is messy, but only because VTT remains a sore thumb in font production 😢

[DHowett-MSFT]

I am extremely on board with this. I love it so much.

[DHowett-MSFT]

FWIW: I've been making edits to the font using ... vim ... for the glyphs file and the ttx, and VTT for the "looking at the hinting" part. This sounds infinitely better.

[madig]

If the designer is on board with this, I can make an example PR. Note that the VTT dump would be editable in a text editor, but since there is no open-source compiler for VTT, you'd still have to use VTT to "Compile Everything", save, and then dump the code.

[DHowett-MSFT]

I think we've got a couple more design changes incoming, but if you're happy to work with us here while we shake the last of the Glyphs stuff our of our systems (and we can get confirmation from said designer) I'm happy to sit on that PR :smile:

[madig]

Sure :smiley: I'll keep using my delicious autohinted OTF in the meantime. Omnomnomnom.

[DHowett]

Hey @aaronbell, what do you think about this? I know that you primarily work in Glyphs, so this is very likely to impact you a lot more than us. The arguments I can see in favor of UFO are:

• Lets us individually source control glyph forms (the .glyphs file is pretty big :smile:)
• Allows us to establish an automatic build pipeline
• Frees a few interested folks from having to buy Glyphs (or a Mac)
• Transformable to and from Glyphs as required for editing

Regardless of whether we move the glyph forms to UFO, moving the hinting source to a version-controlled text document makes it easier to version control _that_ (as Github can diff text far bette than binary TTF).

I'm in support, but I'm also the engineer manifest. :smile:

[aaronbell]

@DHowett I can totally see why you’d be excited about using UFO as a source format. :) It is definitely a more universal format and enables anyone (with the right software) to make modifications to the source file fairly easily. In my experience, I think that Glyphs tends to work really well for small teams where only a few people are likely going to be working on a given file, whereas the flexibility of UFO works quite well for large teams or scenarios with a variety of font editors.

A few comments:

1) Automated pipeline is not unique to UFOs and we can totally build an automated build pipeline around Glyphs. In the case of Bahnschrift, the source is in Glyphs, and it is then routed through TTX to import hinting and make other font table data modifications. This was really advantageous because it allowed us to take advantage of the best features of both Glyphs and TTX.

IIRC, most Google Fonts are built using a similar process, and employ fontmake to process / build the TTF from the Glyphs source. So I wouldn’t say that it is an argument in favor of UFO.

2) My current workflow is not set up for UFOs, so switching formats will limit me to an extent.

(a) I can’t use the more advanced development features of Glyphs because they are not storable in the UFO format. For example, things like the bracket trick (which is quite useful as the font family expands in weight / width) cannot be saved in the UFO format. This isn’t to say that we can’t implement this differently, but it isn’t as simple. I also have not invested in the top tier tools commonly used for modifying UFOs (like Robofont, Superpolator, etc.). Most likely, I would edit the UFO in Glyphs, and re-export a UFO back out.

(b) If I did want to preserve the special features of Glyphs, one option is that I could maintain my own private Glyphs source and export UFOs for github, but that sort of defeats the purpose and means I need to update my source any time modifications are made on the server. Seems ripe for error.

Ultimately, I agree with your perspective that switching the core source files to UFO will allow more people (who don’t want to use Glyphs, or a Mac). For greater community engagement, it would likely be good. It just is a bit more of a hassle for me! I leave the call to your team.

One downside of all this is that there’s a good likelihood that, even if we keep the Glyphs source and build an automated export system in fontmake, the point order / number may change, necessitating the hinting to be corrected.

[mdtauk]

@aaronbell how would you suggest the community can offer suggestions or alternative glyphs to you for consideration or inclusion?

There are numerous vector formats, but not everyone is a designer, or has experience with don't design software like FontForge, FontLab, or Glyphs?

Also, do you design the source outlines with TrueType, or Type1 curves and nodes?

[madig]

I think the better argument in favor of UFO+DS is that it is an open, formally specified format that doesn't change unilaterally between point releases 😄. There is interest from the Noto owner to move to UFO+DS.

IIRC, most Google Fonts are built using a similar process, and employ fontmake to process / build the TTF from the Glyphs source. So I wouldn’t say that it is an argument in favor of UFO.

Under the hood, the Glyphs files are transcoded into UFO plus Designspace using glyphsLib. A pipeline is easier to write because there are libraries (fontTools and https://github.com/fonttools/ufoLib2) to script things outside Glyphs. glyphsLib could theoretically be used, too, but honestly, I don't trust it enough for that and I want to rip out a lot of accumulated code at some point.

At Dalton Maag, we store UFOs plus Designspace(s) plus VTT TTX dumps as the Source of Truth, designers then import them in FontLab 5/6 or Glyphs (via glyphLib's ufo2glyphs) and go in reverse for source control. It's not perfect, but works well enough for us.

The biggest headache is probably going to be that points are different when exported from Glyphs vs. ufo2ft, so you'd have to choose one if you want (or are obliged) to use VTT and stay sane. Or you experiment with https://github.com/adobe-type-tools/afdko/, which Glyphs uses under the hood IIRC.

(a) I can’t use the more advanced development features of Glyphs because they are not storable in the UFO format.

Brace and bracket layers are supported losslessly through glyphsLib. Corner components for example, aren't. What specific features do you need?

[aaronbell]

I gotta say, @madig, that workflow sounds super messy 😆. Also it relies on fonttools not changing their processing unilaterally between point releases (ask me how I know! 😉).

My preference is usually to keep things in a single source format (rather than constantly converting back and forth, which introduces risk), but if it works for DM then it probably would work well enough here too. This font project probably won't require some of the features that glyphsLib can't handle.

@mdtauk I'm not sure there is an optimal solution. Even @madig's suggested route requires some experience and understanding of scripting in order to prepare the source files for proper editing (unless they use RoboFont and can do it natively, but I'm not anticipating the average user or even type designer has that software). I guess one option is to for folks to take the TTF file and make their modifications there, but any changes made would need someone on the project management side to port that change back to the source.

The source is built in cubics currently, and converted to quadratic on export.

[mdtauk]

@aaronbell I think any suggested changes would have to be finalised and implemented by those type designers like yourself, maintaining the source, if only for stroke and character consistency across the whole type family.

But being able to help you all along with better fidelity mock ups and ligatures, icon, glyph submissions. Would TTF files containing the new glyphs work? Exported from our font design software of choice. Or even vector and SVG files perhaps?

[aaronbell]

@mdtauk Any and all work. Already-made vectors does certainly make the process simpler (rather than having to trace), but yeah, any way that you can demonstrate the idea is ok!

[madig]

It is messy, because no editor except RoboFont supports DS+UFO3 properly, and that's precisely what we don't use >:( It's the best army we currently have though. Also, we use a project-specific requirements.txt to pin down deps and don't change them until it's needed, so we can unilaterally keep everything as is 🙂👍 Font collaboration remains a messy business though :(

@DHowett: say... do you happen to have connections to the VTT team or the relevant managers? Some of the messiness could be resolved if we had an open-source VTT compiler, wink wink nudge nudge :)

[davelab6]

If only vtt could be open source....

[mekkablue]

Sorry for being late at the party, only saw this thread now. Two common misconceptions about the .glyphs file format are:

1. that it would not be compatible with source control (it is),
2. and that it would not be open source (it is).

Parallelising a second format and keeping two sources round-trippable just for the sake of it, seems weird to me. It is a potential source of errors (all of what has been written above, and what if they get out of sync in a more crucial way?) and an additional layer of work.

• Lets us individually source control glyph forms (the .glyphs file is pretty big 😄)

One of the points of source control is that it handles changes _within_ a file.

• Allows us to establish an automatic build pipeline

Can be done with .glyphs files as well, as pointed out above already.

• Frees a few interested folks from having to buy Glyphs (or a Mac)

You can edit .glyphs files in FLVI on Windows as well. So in reality, .ufo is just as Mac-centric.

I am not sure this really is an issue at the moment because the vast majority of type designers already have Macs, and the few who don’t, have Windows and can use FLVI as an editor.

• Transformable to and from Glyphs as required for editing

Maybe I am misunderstanding this, but that sounds like a disadvantage to me…?

I don’t want to appear dismissive, but perhaps the problem that needs to be solved should to be pointed out first. Otherwise shoehorning .ufo into it appears to be a solution without a problem. And it may backfire because it may make it more complicated for designers to contribute, and the VTT steps in the process description are scaring off enough of them already.

And the planned expansions for 2020, especially Arabic, are easier in the .glyphs format.

[madig]

The idea is to store the data in UFOs exclusively and generate .glyphs files only for working in Glyphs, i.e. don't keep it in version control.

One of the points of source control is that it handles changes within a file.

The other point is that the user can see what changed where without necessarily having to use special tools to inspect huge blobs. UFO is the only format that allows this. The official .glyphs file differ is Mac-only and frequently failed me when I tried to use it.

Can be done with .glyphs files as well, as pointed out above already.

Not reliably. glyphsLib is a volatile piece of software because Glyphs is (and the .glyphs file format, there is no formal specification, just whatever Glyphs produces, and that can and does change). UFO is the only format you can reliably script and pipeline outside of a Mac CI service running the version of Glyphs that the file was made with.

You can edit .glyphs files in FLVI on Windows as well.

Last time I tried it, FL6's support was... more a technical preview than something you'd use in production. Im- and exporting UFOs and Designspaces works reasonably fine. I hope support for both improves in the upcoming versions.

I am not sure this really is an issue at the moment because the vast majority of type designers [...]

It is for the reasons above. It's true that many designers are Glyphs-based, and for good reasons, don't get me wrong, but Glyphs has made design decisions that make it harder to use outside the single-freelance-designer-who-doesn't-use-Git use-case.

And it may backfire because it may make it more complicated for designers to contribute, and the VTT steps in the process description are scaring off enough of them already.

This may be a valid point. I personally actually don't expect many people to contribute, because from what I have seen so far, font projects are usually done by one person/group and then never touched again until the client asks for an extension, etc.; It's usually just small drive-by fixes that can probably be done in any format. Glyphs 2.6.2 boasts improved UFO handling though, so maybe we don't need the glyphsLib step and can open things directly in Glyphs?

The VTT step is unfortunate indeed, but using Glyphs' hinting functionality would tie the font to Glyphs absolutely, so...

And the planned expansions for 2020, especially Arabic, are easier in the .glyphs format.

What would prevent them from being done in Glyphs and exported to UFO?

Edit: Just to make this clear, I know the UFO format has its share of problems and limitations, so if the designer and project lead deem it easier to stay 100% Glyphs-based, I'm fine with closing this issue.

[aaronbell]

Closing as source is converted.