Hack: Proposal: Support build-time options and simplify builds with Autotools

[iamjamestl]

Some great work is going on in the dev-build-scripts branch to automate building all the various iterations of Hack. This is done primarily with shell scripts.

I'd like to support alternative hinting strategies at build time (see #306). There also seems to be some desire to support glyph and character set customization at build time (see #308). To support these and future endeavors, I think the current shell scripts, while excellent, will become cumbersome to deal with very quickly.

I would like to propose to turn the current build shell scripts into sustainable, maintainable, high quality Makefiles, and tie together all of the various proposed build options with autoconf.

Building these fonts is fundamentally all about transformation, from source data, to unhinted ttf, to hinted ttf, to various web font formats. Make is the perfect tool for representing the transformation from one format to another in a simple declarative syntax, complete with automatic dependency management, lazy evaluation, and parallel building. Autoconf is a time-honored, portable solution for detecting build requirements and selecting build time options.

These are the various options I would like to support:

./configure --with-hints=[default|minimal|none] \
    --enable-webfonts \
    --disable-ttf \
    --with-subset=[latin|??] \
    --with-overlays=dotted-zero,slab-i,... \
    --with-target-platform=[macos|freetype|windows]
    + a few others related to installation I don't want to mention here

• --with-hints=[default|minimal|none]: Controls the ttfautohint behavior. default is the current behavior. minimal disables automatic x-height increasing. none disables ttfautohint altogether.
• --enable-webfonts: Produces woff and woff2 artifacts (disabled by default).
• --disable-ttf: Don't produce ttf artifacts (useful in conjunction with --enable-webfonts, possibly).
• --with-subset=[some-subset-name]: Build the fonts with a selected subset of characters.
• --with-overlays=some,set,of,overlay,names,...: See #308.
• --with-target-platform=[macos|freetype|windows]: Implicitly select an overlay of the same name, even if it's empty, and also output a #define like PLATFORM_<NAME> such that we can preprocess files, like ttfautohint control files on a per-platform basis.

Make would present an interface like:

Recipes would be generic, capturing how to build, not what to build, like:

%-combined: %

%.ufo: %-combined

master_ttf/%.ttf: %.ufo

%.ttf: master_ttf/%.ttf

%.woff: %.ttf

%.woff2: %.ttf

(build commands omitted for clarity)

What to build will be left to one or two declarations:

ttf_DATA = Hack-Regular.ttf Hack-Bold.ttf Hack-BoldItalic.ttf Hack-Italic.ttf
webfonts_DATA = hack-regular.woff hack-regular.woff2 etc...

The Makefiles will be processed with Automake, which gets us all of this:

https://www.gnu.org/software/automake/manual/automake.html#Use-Cases

for free, most useful among which includes:

1. Installation: Great for our Linux package maintainers or users who wish to build their own version, Automake can support installation. Type make install and it will install the font to the correct place for your operating system, like /usr/share/fonts or /System/Library/Fonts, or the location of your choosing with the --with-fontsdir configure option. DESTDIR support makes this a great option for preparing releases.

2. VPATH Builds: Build multiple releases from the same source tree:

mkdir build-default; cd build-default
../configure
make
--
mkdir build-hinting-minimal; cd build-hinting-minimal
../configure --with-hints=minimal
make

1. Renaming: https://www.gnu.org/software/automake/manual/automake.html#Renaming Without any source code changes, users can fork, customize, and build their own version with their own name to be in compliance with the license.

I will build tools and documentation on top of this for the release engineer. The ability to produce release-ready archives with one command will be preserved.

I will also build in whatever make targets are necessary for automated unit testing.

I have a working proof of concept of this proposal. It needs some polish, and I will publish it to my Hack fork on or around Sep 20.

[mynetx]

We're getting professional. There’s nothing to add for me at this point. Will the autotools configuration also support Homebrew?

[iamjamestl]

Yes, Autotools produces POSIX-compliant scripts and Makefiles so it will work on Linux, macOS, BSD, Solaris, and Cygwin. I will opt to use a few GNUisms in make for readability's sake, but GNU make is practically standard, even if it has to be accessed with a separate command, gmake.

[chrissimpkins]

Will have a look through the details of proposal here and comment in #308 that suggests how we might approach alternate glyph builds over the weekend James. I really appreciate all of these efforts. Will need a bit of time to understand this and how it will impact various aspects of the project (in particular the suggestion about source code modification to something that is not valid UFO spec with on demand source compiles from #308). This would definitely be a way to implement customized builds which is something that we've wanted for a long time.

More questions for clarification to come...

[chrissimpkins]

@iamjamestl

Knowing myself and my own default verbosity flags, I am going to have extensive commentary about this as the approach that we use in Hack. As I've started to think through the approach a bit, I have a bit more grandiose thought and want to pitch this back at you in a way that should not create more work but should take on a broader scope.

We have identified a set of FLOSS compilation and post-compile modification tools that are incorporated into the current shell scripting approach for builds in the dev-build-scripts branch. These are under active review by the Debian and Fedora groups against free open software guidelines for UFO source based font builds to final compiled and post-compilation modified ttf, woff, and woff2 build artifacts. Their guidelines require that the build tools (and all dependencies of the build tools) must meet their criteria in order to build from source and distribute on their respective distros. Currently, this is a limitation for those who might want to release fonts on Debian/Ubuntu/Fedora (and other distros that support similar guidelines) and my understanding is that groups that are currently doing this build with FontForge, the only compilation tool for UFO source that is available at the moment on their distros. Once these reviews are complete, the build toolchain that we establish here will permit any free open source typeface development group to redistribute according to DFSG, the Fedora free open software redistribution guidelines, and I assume others without further review of any tooling outside of the project's own repository and any build dependencies that they add.

I propose that rather than build the configurable Hack compilation workflow, we build the definitive FLOSS UFO to ttf/woff/woff2 compilation workflow that will permit any group to have a "build-in-a-box" approach to support the following things:

• development of ttf build files with configurable ttfautohinting settings to allow for platform specific hinting adjustments, including a sensible set of defaults. @lemzwerg has provided a great deal of assistance with the instruction sets (and manual modification of those instruction sets) in Hack as well as ttfautohint compilation modifications to support OS X source builds of ttfautohint to support our automation of the compilation process currently in place. Perhaps he would be willing to pitch in a "best practices" approach to ttfautohinting defaults for projects that target FreeType on the various distros based upon how they approach the use of included instruction sets as defined by their FreeType rendering defaults (that in some cases are distro specific). This would address your original rationale for the proposal.
• development of woff and woff2 web fonts (these appear to have emerged as the current default for current browser support)
• simple, configurable build artifact type declarations
• simple, configurable alternate glyph replacements that support an approach that exists outside of OpenType use of .alt style glyphs and works with the UFO source code itself (in a fashion along the lines of the discussion in #308) so that features.fea scripting with recompile or manipulation of application specific alternate glyph settings is unnecessary for the rendering of alternate glyph shapes. This would allow developers to support in repository or external repository extra glyph libraries that can be compiled as overlays into the master default font set.
• simple, configurable subsetting of "master glyph sets" based on modifications of the UFO source used for compilation through the configure process rather than through the use of postcompilation modifications of compiled files (as is the common approach now). i.e. build the appropriate set at build time, not in the post-compilation phase.
• well defined specification for the new UFO spec based "pseudo-source" (*.in style files) that are likely to emerge from this build approach. Or if really ambitious, entirely new source spec that better serves the collaborative development workflow with compilation to something that works in the current era of font compile tooling.
• clear, simple to follow documentation targeted to user base, developer base, and maintainer base.
• DFSG (among other free software guideline) compliance

And if we do it, let's do it right by finding a good approach to testing it supported through existing or spawned tools based upon where this development leads. Testing support should (IMO) be available for this workflow for any development groups that adopt it (including the Hack project). We can create UFO v2 and UFO v3 spec'd mock type source to use as a basis for this. I would love to further develop ufolint to support linting of the UFO source based upon how the above project develops, extending this to linting of "pseudo-UFO" source as necessary. There may be other tooling needs for linking glyphs, subsetting glyphs, naming builds, modifying OpenType tables via plist files or their pseudo-brethren, etc that emerge as a result. These are all issues of interest for us.

You have landed on an idea that provides implementation details for something that is sorely needed and missing (to my knowledge) in open source typeface development and that is a wrangling together of the large array of excellent, available FLOSS tools out there into something that those who really don't have an interest in tooling but simply want to design/configure/build/distribute their typefaces need. My own wrangling process for Hack required (literally) a couple of months of investigation. This should not be the case for others and I would love to offer other groups a significantly simplified approach based upon the knowledge that we develop here. This affords us the opportunity to define these tools and this approach ourselves, and to allow the community that (hopefully) develops around the project to guide and assist in its development.

My comments from a Hack project perspective are better served as issue reports on your repository IMO. Once this reaches a stable stage of development and the approach / documentation is in place (all things that would need to happen to support use in Hack anyways), I think that is when we push for a decision about adoption of the new build tooling approach here. I am definitely in favor of the initial pitch. Sending you an invitation to the Source Foundry organization account. We would love to have this project hosted here if you would be willing to lead its development and allow us to take part in a collaborative effort on its design.

PS: yes, I failed on the verbosity goal...

cc: @spstarr @paride

[iamjamestl]

There's a lot in there to parse, but at least on first reading, I really like what you're saying. This can be made into a project that is consumed by other font projects, including Hack. Autotools is very flexible about where it lives in relation to both the source and build directories. In fact, they can be three completely separate trees:

/font_project/source/...
/font_project/tools/configure(.ac)
/font_project/tools/Makefile(.am)
/some/completely/separate/build/dir

Such that you can:

cd /some/completely/separate/build/dir
/font_project/tools/configure --with-some-options
make && make release

Then, the whole tools directory can just become a git subtree of the font project. Bug fixes and enhancements in the tools would apply to all of the font projects that use it. Automated testing could be performed directly in the tools project. Font names and versions, which the tools would need to know, can be determined by the tools programatically by convention or configuration.

I'm gonna be in the car for at least 12 hours this weekend, so that'll give me a lot of time to think about what this looks like and how to achieve it, but I suspect I'd be happy to lead this effort under the Source Foundry umbrella in collaboration with you.

[iamjamestl]

So now that I've had some time to digest this exciting new direction, I have a better idea of how to get started, which I will lay out as simply as I can as a series of steps:

1. I will accept the invitation to join the Source Foundry organization.
2. I would ask that @chrissimpkins create the project with permissions as he
   sees fit and with the name and license [1] of his choosing. I guess names can always be
   changed, but a lot of expectations are established up front by the name alone.
   If the goal is to possibly say, "hey, designers, drop this code along side your
   UFOs and we'll take care of building it" then the name needs to accurately reflect
   that. The best I've come up with is some variation on 'autofont'.
3. Establish enough of a skeleton to run make test and connect it up to Travis.
4. Create automation for Travis to run autoreconf and push configure
   and Makefile.in files back to the repository. Traditionally, those files are
   not stored in the repository, but they are distributed with the source code (as
   generated by make dist); however, I want to make it as easy as possible for
   designers to incorporate these tools into their projects, and git subtrees are
   one way I'd like to offer that. I think it is also important to make traditional releases
   (.zip, .tar, etc.) for those who want it and as a good software engineering
   practice. Having Travis create the configure and Makefile.in files from
   their configure.ac and Makefile.am sources will eliminate any doubt whether
   they're up-to-date, what Autotools versions they were generated from, etc.; and
   it will allow contributors to make small or trivial changes without having to
   install Autotools themselves.
5. Begin to implement roughly what I outlined in the OP, with tests. The
   exact requirements and milestones can be laid out in the new project as
   separate issues, rather than in this issue, which could be closed. For now,
   we can use Hack as a test fixture, but I like Chris's suggestion of creating a
   couple mock UFO sources.

There's more to say about some implementation details I've been playing around with, but they can be discussed in the new project. I will at least say here:

• I see no reason why this can't work as a simple drop-in to any downstream
  UFO-based source. I think one result of this project will be to define a
  "suggested" font project layout, and if those suggestions are followed, then
  the tools will work without any special configuration. But we can also
  "source" a configuration file in the downstream project to give developers an
  opportunity to override any defaults.
• Given all the tools and standards around UFO, I want to say I am committed to
  making this work with unmodified UFO sources as much as possible. I think
  that would be a powerful thing to tell designers: you don't have to modify your
  source to work with these tools.
• I will be watching the Debian/Fedora packaging progress with interest. These
  same issues are likely to come up in Gentoo, which is a source-based distro
  that doesn't currently package fontmake.

[1] We are not bound by the autoconf license: http://git.savannah.gnu.org/gitweb/?p=autoconf.git;a=blob_plain;f=COPYING.EXCEPTION;hb=HEAD. automake makes similar concessions.

[chrissimpkins]

license [1] of his choosing

We generally license under MIT. You ok with that James?

The best I've come up with is some variation on 'autofont'

Love it! Quick web and Github specific search didn't lead to any name collisions of significance for the project. Let's start there and if you change your mind we can always change the repo name.

I see no reason why this can't work as a simple drop-in to any downstream
UFO-based source. I think one result of this project will be to define a
"suggested" font project layout, and if those suggestions are followed, then
the tools will work without any special configuration. But we can also
"source" a configuration file in the downstream project to give developers an
opportunity to override any defaults.
Given all the tools and standards around UFO, I want to say I am committed to
making this work with unmodified UFO sources as much as possible. I think
that would be a powerful thing to tell designers: you don't have to modify your
source to work with these tools.
I will be watching the Debian/Fedora packaging progress with interest. These
same issues are likely to come up in Gentoo, which is a source-based distro
that doesn't currently package fontmake.

All of above sound great! Let's start opening issue reports on the new repository where these issues can be discussed by anyone who has an interest.

[chrissimpkins]

@iamjamestl

https://github.com/source-foundry/autofont

gtg! Travis CI active on the repo, MIT license added to get started. New issue reports for both should you need to discuss anything with the initial setup. You have full admin access to the repo and should treat it as your own. Establish the model for collaboration that you'd like to use and we will abide by those rules. Feel free to invite any collaborators to your repo for work. Let me know if you need anything else to get going. Will continue the conversation over there.

[chrissimpkins]

and let me/us know how we can help! :)

[iamjamestl]

Thanks, @chrissimpkins. I am really looking forward to this.

I think I will leave this issue open for a little while longer, just to get some extra visibility from the community, but I would otherwise direct future comments to the autofont project: https://github.com/source-foundry/autofont/issues.

[chrissimpkins]

Project in development at link in https://github.com/source-foundry/Hack/issues/310#issuecomment-331683019. Closing this IR.