Hack: Technology/place for alternate, extended char set, & outsourced glyphs

[mynetx]

The Hack team wants to focus on a certain character range, see #307. Like a breakfast buffet, there are additional and alternate glyphs that should be available to anyone wanting them, maintained by someone else. This requires a technology for incorporating them into the build process as well as a place where they are hosted and showcased for selection.

Let’s discuss details about the idea, workflow and implementation in this issue.

[chrissimpkins]

I like the term á la carte ;) Thanks for starting the conversation @jublo

From my standpoint, it doesn't need to be maintained by someone else (though absolutely could be), simply external to this repository and able to support things like the following:

• alternate glyph shapes (e.g. replace current zero design with an alternate that may have been designed by anyone, including numerous alternate choices at the same Unicode position)
• extended character sets that are not "deemed appropriate" for upstream (conversation underway about that definition and considerations associated with it as linked above)

What I have always had in mind was a tool that automates replacement of glyphs within the Hack sets with alternates and supports extension of the Hack sets from an external repository of "glyph choices". Implementation details TBD, but I think that it would be nice to be able to define individual and ranges of glyphs to replace with alternates, & individual or ranges of glyphs to extend the base sets with. I also would love to see this repository of alternates/extensions easily support contributions via PR from others so that anyone can add an alternate that anyone else can use. We provide the appropriate metrics for the design and a minimal UFO source template and anyone can go to work on a shape that they like, then contribute it to the general pool of available shapes.

Thoughts? And if this is something that carries some interest out there, how do we approach the implementation details?

[chrissimpkins]

cc: @iamjamestl

@jublo meet @iamjamestl who has an interest in a configurable build from source approach in order to support hinting configuration. Perhaps this approach could also be used to support the alternate glyph/extended glyph character set approach here? Will let him detail what he has in mind to start the discussion.

[mynetx]

Suggesting to provide download sets in the form of branches in a repo that are not merged to master, but base off master. There could be a branch U0030-slashed that contains a slashed zero glyph. Does this idea throw any issues with PRs? GitHub offers existing branches for merging …

[chrissimpkins]

Maintainer could create and define the naming scheme for branches with any PR's but that is bound to create a very heavily branched repository that might prove difficult to manage. If it takes off and there are hundreds (thousands) of glyphs contributed you are at 1:1 with git branches...

[mynetx]

Could better be a subfolder strategy then:
/menus/slashed-zero/some/path.ufo/some/subpath/zero.glif

[chrissimpkins]

my thought would be to eliminate the ufo directory structure and simply include the glif files for designs that are intended to be overwrites of existing glyphs. for those that represent new glyphs (not currently in Hack source) we will have to figure out a different approach. new glyphs will require some tinkering with the plist files in the ufo source. we could create a tool to configure the plist files so that they include the new glyphs possibly? perhaps the expansion of sets becomes a down the road goal and replacement is immediate goal with a design that keeps that longer term solution in mind.

[chrissimpkins]

would be nice to be able to display the design in an image in some fashion too.

[mynetx]

I think everyone would love a simple structure like /menus/slashed-zero/zero.glif. For new designs, we would offer them to upload files as unicode point glif files, and a bot would automatically rename files that have names. Yes, editing plists is rather easy once you have a complete set of names and their codepoints.

[chrissimpkins]

tbh, it may make more sense (and be much simpler) to maintain all extended sets (new glyphs) in the Hack source or as new forks of Hack if it is really felt that these are problematic for any reason other than for web font file sizes. For web fonts we can subset the fonts as we are doing. This will greatly simplify the process for redistribution teams who want to distribute the 'max set' of glyphs and the storage issues will not be notable for the average desktop user. There is minimal impact on file size across the sets under consideration. We are not dealing with CJK glyphs where files can get larger and even then this is not an issue with current drive storage capacities.

Thoughts about making the external repository alternates only and creating a design objective to prioritize design optimizations in some sets but not others in this repository? Anything that is added to Hack can then have alternates supported in the other repo. Would require a base glyph here to support alternates there. This would eliminate the need to create an entirely new build process to support the new / extended glyphs.

[chrissimpkins]

/menus/slashed-zero/zero.glif

Would need to figure out an approach that supports multiple alternates. For instance a dotted zero and slashed zero would both carry the same glif file path for replacement. I like the inclusion of this in a file path structure as above. How do you imagine this being specified in the build process? As the end of the file URL? (e.g. slashed-zero/zero.glif)

[chrissimpkins]

Also, would be useful to include a way to collect all Unicode code point alternates under a single directory:

e.g.

u0030/slashed-zero/zero.glif
u0030/dotted-zero/zero.glif

Users will need to be able to look through the directories in the repository to determine what is available for builds. Or could shorten to something like:

u0030-slashed/zero.glif
u0030-dotted/zero.glif

More directories but still an organization under code points.

[mynetx]

I was thinking about this:

• Your suggestion as per u0030-slashed/zero.glif.
• Another folder menus/fancy-numbers/glyphs/u0030-slashed.txt with a description of these styles combining multiple glyphs at menus/fancy-numbers/README.md.

[chrissimpkins]

If I understand correctly, the design files (*.glif) live in individual directories to facilitate incorporation into the source and description/documentation files live under the menus directory? Like it! Down the road the menus files could turn into a web site where it would be easier to view images of the examples if we start to accumulate multiple variants.

I assume menus is intended to follow the breakfast buffet theme? If so we need a catchy repo name!

all-you-can-eat-pancake-fest ? :)

Too American?

ontbijtkoek-beschuit (cc:@burodepeper)
Es-ist-mir-Wurst (cc:@jublo, @texhex)

[burodepeper]

My suggestion is $codepoint/$alternate-name/$official-glif-filename as was suggested earlier. The $codepoint directory can contain a README with a description of the various alternatives, if a simple description like 'slashed zero' isn't clear enough. Ideally, there'd be a symlink from a canonical name to a codepoint, ie /zero points to /u0030. All other information should just go in a general README.

@chrissimpkins: ontbijtkoek-beschuit would be horrible ;)

[chrissimpkins]

The build tooling may help to determine the path plan. What do you have in mind for this @jublo? Seems we could either (1) build from a configurable Hack repository installer (e.g. something tied into @iamjamestl idea about autotools based ./configure && make builds); (2) build from a configurable external repository installer that would sit in this new repo and pull Hack upstream for all of the modifications (similar to approach used by @texhex with Windows installer but with source modifications before external installer approach used).

From a user standpoint, likely not a big deal either way. From a developer standpoint the latter approach is likely to be better or we are going to modify our source when we test the glyph replacements (unless this is all via temp directories, source copies, and a good cleanup plan so that these changes don't hit Hack commits, this is the workaround that we are using for web font subsets at the moment...).

[iamjamestl]

I really like this idea, both as a way of extending the character set with "community supported" glyphs, selecting among several options for built-in glyphs, and as a way to potentially target different renderers for fussy glyphs.

I think for this to work and be manageable, I'd agree with Chris and say we'd have to stop treating the ufo data as the authoritative source, and instead formulate a simpler structure over which changes can be "overlaid" at build time and then "compiled" into a ufo structure.

As far as I can tell, the three major components that go into making the ufo and, ultimately, ttf file are:

1. Metadata
2. Glyph data
3. ttfautohint control data

Metadata is not something I'd imagine we'd want to change a la carte. Much of it is static, and the rest, like lib.plist can be generated.

Glyph data and hint data are tightly coupled, and developers of an "a la carte" option (I like the term "overlay") may want to provide ttfautohint control data for their custom glyphs.

With these things in mind, I envision a source structure like:

/source/<style>/features.fea
/source/<style>/fontinfo.plist.in
/source/<style>/u0030-zero.glif
/source/<style>/u0030-zero.ttfautohint
/source/<style>/u0061-a.glif
...

Comments:

1. Remove references to "Hack" in the source tree--it can be filled in by autoconf/automake where necessary and then the project can be forked and the font name only has to be changed in one place for the fork to be in compliance with the license.
2. lib.plist would be generated. The glyph names contain all the information needed to do this: the unicode code point to indicate the order, and the glyph name.
3. The structure inside each style is intentionally flat, to be easier to navigate, and to be easier to reason about in Make.

I imagine overlays to have the same structure:

/source/overlays/overlay-name/<style>/u0030-zero.glif
/source/overlays/overlay-name/<style>/ue0b0-unie0b0.glif
/source/overlays/overlay-name/<style>/ue0b0-unie0b0.ttfautohint
/contrib/overlays/community-supported-overlay-name1/<style>/u0069-i.glif

Comments:

1. "Officially" supported overlays can go in the main source directory. "Community" contributions can be maintained in a separate "contrib" tree, and may have less stringent testing. This can help users discover additional overlays without having to find and pull them in from some random GitHub repository, while not polluting the "official" source tree.
2. Overlays can deliver both glif and autohint control data. If they deliver a glif file without autohint data, and the glyph is intended to replace an existing one in the source tree that does have autohint data, the autohint data will be removed.

Overlays could be selected at build time, either from the official source tree or the contrib tree, like:

./configure --with-overlays=overlay-name1,overlay-name2,community-supported-overlay,...
make

and Make would be responsible for producing an intermediate source with all the specified overlays:

%-combined: %/* <for each selected overlay, overlay-name/*>
        rm -rf $@
        cp -R $< $@
        for glif in <for each selected overlay, overlay-name/*.glif>; do cp $glif . && rm -f ${glif%.glif}.ttfautomake; done
        for ttfautomake in <for each selected overlay, overlay-name/*.ttfautomake>; do cp $ttfautomake .; done

This is a bit of pseudocode, but I think you can see the point: the source is copied to an intermediate "combined" source, and then the glifs and ttfautomake control data from each selected overlay is copied into the combined source tree. If the overlay delivers glif data without hint control data, and the source contains hint control data, it is removed because the source control data would not necessarily be valid anymore. Make can track dependencies among all the source and overlay data such that if any of it is changed, the combined source tree is rebuilt, triggering off other builds down the line.

The combined source tree could be compiled into a ufo source with make:

%.ufo: %-combined/*.plist %-combined/*.glif
        rm -rf $@
        mkdir $@
        copy metadata
        generate lib.plist
        copy glif data into $@/glyphs/

Again, pseudocode. Make can ensure that the "compiled" ufo data is updated exactly when necessary. Also, this could be broken down into several subtargets to avoid removing the whole compiled ufo when a single source file is updated. Then you could work on this ufo in an external editor and continue to git pull updates without it blowing away your work.

Likewise, the control data would be "compiled" from the combined source tree:

%-control-data.txt: %-combined/*.ttfautohint
        cat $^ > $@

I think there is probably a lot of room for discussion about the exact form of the source data, but I hope this overlay concept makes sense. I think autoconf would provide a user-friendly way of selecting the desired overlays and Make would allow us to implement this fairly difficult concept very easily, in a declarative way, with complete dependency management.

I also like the idea of compiling to a ufo rather than keeping the ufo data in the source tree because it will encourage cleaner merge requests. Rather than getting ufo data as written by some editor like FontForge, the contributor would have to explicitly copy their modifications into the source tree. They can still use FontForge to edit the compiled ufo, and make could be used to trigger off subsequent builds whenever an external editor changes it, but it won't pollute our source tree.

[mynetx]

@texhex Thanks for your extensive input! Much appreciated.

tl;dr

Let's resumé so far:

• Create a new authoritative source for official Hack. The Base™.
• Create central metadata file. Contains Hack font name, license and more.
• Compile UFO from source.
• source contains style being font-weight (100-900 in CSS) paired with font-style (regular, italic).
• style contains glyph data, features and ttfautohints.
• overlays contain named sets of additional characters beyond our well-defined base set. Glyphs that not everyone needs, but that we still want to keep in the main repo.
• contrib/overlays are user countributions, otherwise identical to overlays.
• Any overlays can be pulled during build process using command line parameters.

Did I forget anything @chrissimpkins @burodepeper?

[chrissimpkins]

@iamjamestl Thanks for the extremely detailed overview here James. I really appreciate your input and feedback! This is a very novel and frankly exciting idea that could address things that we have had in mind to do but without the approach to do them.

Question 1 from me is whether you envision this as the structure of the Hack repository or as a separate Hack fork repository that is designed around the idea of broad customization options?

If the above is intended as a redesign for the Hack repository source and tooling, I will need some time to think through all of the implications of the proposed changes. The introduction of a build step to create properly spec'd UFO source files adds a great deal of complexity to what is currently a reasonably simple build process in the name of broad support for customization and is bound to have implications for end users (including platform specific consequences with the new reliance on configure to create UFO spec'd source), redistribution teams that build from source, our CI testing approach and the CI testing tools that are built around UFO source code as spec'd (and not for a new "spec" that we will be creating), repository maintenance needs, and likely more. Customization is a good thing and will be a benefit IMO. Customization at the expense of access by entire platforms to the source code to modify and build the fonts (because it relies on a source code compile step that is not readily supported in their situation) may not be. Turning manageable, testable properly spec'd source code into our own new "pre-source" specification with the introduction of a build step to form properly spec'd source in the name of customization support is not something that initially strikes me as a good idea.

If the above is intended to be a fork of the Hack repository that restructures the source in this format in the name of extensive configuration options, I am very much in favor of it from the get go.

[iamjamestl]

Oh, yes, I do mean for the Hack repository. I am well aware of the work you put into ufolint and ufodiff and this would not take away any of that. At worst, your .travis.yml will go from the equivalent of:

script:
  - ufolint src/Hack-Regular.ufo

to

script:
  - ./configure && make -C src Hack-Regular.ufo
  - ufolint src/Hack-Regular.ufo

One extra step that takes

> time (./configure && make -C src Hack-Regular.ufo)
0.15s user 0.11s system 21% cpu 1.247 total

1.2 extra seconds on my system, using tools that are supported and have been supported on every operating system, including Windows, for decades.

For that one line and one extra second, consider what you get:

1. Configurable font names without any source code changes.
2. The ability to tune both glyph shapes and their hinting per renderer.
3. The ability to support character variants.
4. Glyph and hint data side-by-side.
5. Cleaner pull requests--only the glyph (and possibly hinting) data; none of the metadata; and no chance of stupid tools like FontForge overwriting every file in the repository with it's own unique flavor of ufo.

Rather than think of it as some "pre source" format, consider thinking about it as "pre-formatted" or "pre-processed" source. Every C project you've ever touched has probably been littered with *.c.in files that have to be processed before they can be linted.

This particular "pre-processed" source format is just something I came up with in about 30 minutes this afternoon. I'm not married to it or anything. Frankly, we could probably get the idea of overlays to work with the raw UFO data, but you'll have to tell me how to generate lib.plist. If it could be abstracted away in a new tool like ufomerge even better.

[iamjamestl]

So I gave the kind of worst case scenario, at least from a CI point of view. The best case scenario is that, from a UI perspective, the change is almost completely transparent. We add the *.ufo dependencies to the ufolint target, and make takes care of the rest.

So your script would go from:

script:
  - make ufolint

to

script:
  - ./configure && make ufolint

Edit: Removed an opinion. I'll stick to the facts.

[chrissimpkins]

Commented in #310, will add it here as well. Educating myself on this approach and digesting all of this information. Intend to be in touch in the next few days with thoughts.

[iamjamestl]

Thanks, Chris. I hope it didn't come across that I wouldn't be interested in working on this unless it made it into the official project my way. I'm excited by this idea, but I'm sensitive to the fact that it would be a big change, and if working on that change and letting it soak in a fork for a while makes that easier to swallow, I'm all for it. I'll wait and see what your thoughts are.

[chrissimpkins]

Not at all! See #310 ;) I think you may actually be undershooting with the opportunity that this tool affords us and others out there.

[chrissimpkins]

I am digging into autotools documentation this weekend so that I have a better grasp of scope / capabilities. Will be much better prepared to comment once I educate myself.

[iamjamestl]

Well, if it helps you to have some sort of concrete example as you're reading the Autotools documentation, I'll link you to my current branch: https://github.com/iamjamestl/Hack/tree/autotools, as long as you understand that what's there so far is not representative of what I want to achieve in #310, rather it's the result of about 2 hours of work and is the bare minimum necessary to get the compiled ttfs.

[mynetx]

I feel overwhelmed right now. It feels like a complex and powerful thing, but I cannot get behind it right now. I will trust in you building, testing and I can just call it and it works. :)

[chrissimpkins]

I feel overwhelmed right now. It feels like a complex and powerful thing, but I cannot get behind it right now.

No worries JM. Interest in designing alternate glyphs once this is set up?

[mynetx]

Interest in designing alternate glyphs once this is set up?

Course, but needs to be demand driven.

[chrissimpkins]

Course, but needs to be demand driven.

If you make a slashed zero, dotted zero you are going to make a lot of friends out there... Most popular requests for years.

[chrissimpkins]

Let's figure out an approach to the alternate library and we can revisit glyphs that have been requested. Give the designs some thought in the meantime and let's see if we can come up with something new that has character. Slash in opposite direction to common approach? Vertical slash? Would be fun to play around with design and see what emerges.

[chrissimpkins]

@jublo here is where @burodepeper is going with the default oval filled zero. Work in progress that we intend to release as part of the v3.x builds.

https://github.com/source-foundry/Hack/pull/276

[mynetx]

@chrissimpkins Vertical slash can be easily confused with theta θ.

[chrissimpkins]

OK, set your design mind on it and let's see what you come up with. Standard slash and dotted will be very popular options. People have requested these repeatedly. The oval filled zero gives Hack a big part of its identity IMO so I have never been swayed. There are people who despise it and you will attract users to the fonts with alternatives to the zero. Clearly the goal is to differentiate from O and there have been a number of approaches to this in other fonts (e.g. more rectangular zeros, fills of different styles, different sizes like shorter or taller zero vs. O). If this is something that interests you, have a look through other typeface designs out there and see if you can come up with some ideas about how we might approach it. Standard approaches are absolutely fine and will be desired because they appear "normal" to users who are used to viewing the zero this way. Different approaches are experimental and you might find something that is very different from how anyone has approached the legibility issue in the past.

[chrissimpkins]

This thread should set you on your way... https://github.com/source-foundry/Hack/issues/200

[mynetx]

Right. I will start some ideas at some point. do we have a sep. issue for the alt-zeroes?

[chrissimpkins]

@jublo https://github.com/source-foundry/Hack/issues/200

[chrissimpkins]

@jublo Once I complete my work on the v3.0 release and this is going through the review process, I will have some spare cycles to build this alternate glyph repository structure. If you are free to design any glyphs for it by the time we push the v3.0 release, it would be great. It would be nice for something to be available when we push this major release to support this new capability. We do not need to have the configurable build tool available to use this. Users can simply drag/drop the glyph *.glif file into the source and build a set of fonts that includes any alternate combinations that they like. Does not need to be a polished v1.0 repository immediately. Ideally it is a library of alternate glyphs that over time permits simple visual examination of the alternate shapes in some form (website, Markdown files in repo, etc).

I am thinking alt-hack for the repo name :) Thoughts?

cc: @burodepeper - in case you have a burning desire to draw after all of this non-design transition...

[mynetx]

@chrissimpkins What glyph requests are most valued? Where to host them until your struct is done?

[chrissimpkins]

@jublo keep them locally for now. will only need the *.glif files when ready.

[mynetx]

@chrissimpkins What glyph requests are most valued?

[chrissimpkins]

@jublo follow this thread up ;)

[mynetx]

@chrissimpkins I meant, besides variants of 0.

[chrissimpkins]

@jublo this is definitely an area where it is best to experiment. If it seems like something of interest to you or anyone else out there it is worth considering as an alternate, so long as you stick with some semblance of the metrics of the overall design. This is an opportunity to experiment with the design in a granular way at the individual glyph level so that new and interesting combinations can be built from the alternates.

To address your question directly, I don't know what will be of most interest out there. Remains to be seen. Could be something entirely new and very different that hasn't been attempted in a face intended for source code before. Could be old standards that feel familiar like the slab serif style lower case i or the slashed zero. These are so widely used that they feel 'normal' to those who frequently view code. To me, the challenge and enjoyment of this project is trying things that are not common in other faces (eg mixture of regular and semibold glyphs in regular set, different zero shape, very wide spacing on brackets, etc) and seeing how they play in source code across users who view a wide range of languages (source). Without the concern about whether something is acceptable for the upstream main source as a default shape, I am hoping that this alternate glyph approach encourages experimental designs that could turn into a wide array of derivative possibilities for the base design. We reserve the right to merge things that we really like upstream too so it will be a good testing ground for these experiments.

[mynetx]

@chrissimpkins Thanks for the background info! So all what’s missing is a proper playground to showcase/discuss.

[chrissimpkins]

@jublo and glyph designs!

[mynetx]

@chrissimpkins I will get round to that at some point next week.

[chrissimpkins]

A new repository for alternate glyph styles is now available and we are adding stylistic alternates.

https://github.com/source-foundry/alt-hack

New glyphs are being added and contributions are welcomed. Will be able to build these into the Hack sets with the new build tooling coming in v3.0 and we will have detailed documentation re: how to do this available at release.