Fontbakery: System for declaration of subjective font quality scores

[davelab6]

https://fontstand.com/news/knowledge/evaluating-the-quality-of-a-typeface/ has a nice list of things to consider a font family "good quality", and many are unlikely to be easy to implement as font bakery checks.

However, along with the project's fb config, it should be possible to have a profile of checks for these things, which simply checks that config file for attestations that these things have been done.

This is a bit vague, so I'm assigning it to Felipe and also Simon, Denis and Adam, to think about and get back with a more concrete plan in the coming weeks :)

[felipesanches]

Today we had a video call in which we discussed the structure of the Unified Font Repo and when we were talking about the FontBakery report and discussing ways to deal with FAILs that are silenced by disabling specific checks, the fb config file was brought up again as a potential solution.

Then I once more suggested that when we do so, it should be mandatory to provide at least a statement (a single sentence) explaining the reason why the designer decided to skip each check, so that we have it properly documented.

I believe that your proposal here in this issue would likely use a similar mechanism.

[felipesanches]

Here's a somewhat crazy idea that might actually make a lot of sense.

What if we come up with a CreativeCommons-like system composed of a set of mnemonic badges corresponding to those subjective quality aspects of a font project and then we created mechanisms for those badges to be placed on the README.md of a repo, or in a website hosting the font files.

Such badges would have corresponding human-readable text as well as a machine-readable (read: fontbakery can easily read it) metadata file format.

To mitigate the risk of someone gaming the system by simply applying all the badges without actually reviewing the font design, we should make it mandatory that badges are "signed" by simply providing the name of the reviewer. I am not implying any sort of crypto-signing here, but merely trusting what people say, the same way we trust their usernames/real names/pseudonyms, for copyright and licensing matters, when they contribute to a free software project.

Members of the type community would taint their reputation if they were caught excessively overstating the qualities of a design to game the system. And faking signatures (labeling a design review with someone else's name without their consent) would pretty much be identity fraud.

It should be also easy to declare these subjective font quality scores. So we might create a website where a reviewer could answer a set of questions and then the website would generate a metadata file that can be downloaded and saved on the root of a font project directory/repo. Badges would be automatically rendered on README.md based on the contents of that file and provide links to the more verbose human-readable description of the review, which could even showcase the reviewer info if available on the Google Fonts Designers Catalog.

All of these things are completely analogous to the Creative Commons framework.

Oh! One last thing: even though I mentioned the Google Fonts Designers Catalog here, I think this seems generic enough that it should be a system that could be used on any font project, not meant to be exclusive only for Google-hosted fonts.

[davelab6]

Nice ideas!!! :)

[felipesanches]

• The description I gave above would probably result in way too many badges. Perhaps several similar aspects in a review could be grouped on a single badge so that there's a smaller set of badges.
• But it would still employ numeric scores based on tallying the results of the individual questions, and all details can be seen in a full report for those who click to learn more.

[felipesanches]

Could maybe use https://github.com/SorkinType/EQX as a starting point for the design review website.

[simoncozens]

Okay, so let's combine a number of thoughts we've been having recently:

• Badges for quality assurance
• Checks should have "severity" as well as PASS/FAIL/WARN
• Some checks can't be automated
• Different design versus technical profiles
• Returning FAIL is maybe too judgmental to designers and off-putting to people who use CIs.

Here's an idea, then:

• Every check gets a severity score. That means every profile gets a potential "perfect" score which is the sum of the severity scores in the checks that are not skipped.
• Passes get full points allocated to the total. Warns also get full points, but emit a warning.
• Some checks just check for the existence of a "yes, I thought about this" declaration in a configuration file.
• Now each profile produces not a series of pass/fails but a score out of 100%.
• This score gets reported to the user at the end, and a badge is created.
• Separate badges can be created for separate profiles - technical, design, etc.

[felipesanches]

I suppose that a single profile may have multiple badges. As discussed at #3336, we never really extensively used Sections, but maybe that's their fate :-) A badge for each section of a profile.

But badges could also perhaps cross the profile boundary because they seem more likely to be defined by topic. For instance, on the googlefonts profile I could pretty well see a vendor-specific badge for METADATA.pb-related checks, while the rest of the checks would probably add to the scores of a badge that also takes into account checks from universal or opentype profiles. A "Variable Fonts" badge would be an example (containing both universal as well as vendor specific checks).

When thinking of badges, I picture in my mind things like the "Achievement Unlocked!" sort of goals one often see in video-games. These often have:

• A short catchy title
• A slightly longer descriptive text
• Some kind of progress UI. In some cases I've seen it depicted as "1 to 5 stars", in other cases "bronze, silver, gold, etc.". In our case it would likely be just a percentage score as @simoncozens described.

Grouping existing checks on those "Sections" was already a long-postponed goal and I had not put much energy on that because it was unclear for me if doing it would have any real value. I have been really planning to completely deprecate Sections to shave complexity out of the code-base unless we actually used them extensively.

Note: Sections might not be best way to implement this if we decide to give Badges the ability to group checks across profiles.

It might be a nice brainstorming exercise for us to try to come up with a small set of initial badge proposals in that spirit and then decide if it really makes sense to go forward.

[felipesanches]

I'll keep the word "subjective" in the title of this issue for now because that's what @davelab6 initially suggested us to address here, even though this conversation is clearly evolving to also embrace the existing technical checks.

We need to define how we'll store these "subjective quality scores" in a way that is user-friendly.

[simoncozens]

"Achievements" would be nice, but there's already a kind of de-facto standard for build/QA badges:

[simoncozens]

https://shields.io/#dynamic-badge

[felipesanches]

YES! But I think the idea would be to have a larger set of badges such as:

We would have to draft what else makes sense. Maybe based on the original article that @davelab6 pointed out when opening the issue.

[simoncozens]

Just have one per section. This is what sections are for. :-)

[felipesanches]

@twardoch, @RosaWagner, @moyogo, @vv-monsalve, @eliheuer, or anyone else interested:

I'd be glad to have some input from you on this conversation

[eliheuer]

I like it!

The only thing I would add, as the article Dave links in the first post says in the conclusion:

And then there are, of course, typefaces that are intentionally designed to be weird, wonky, imperfect, distressed, uneven, casual or handmade etc. They are not meant to be evaluated by the same set of technical criteria, but you get the idea.

Maybe a way to indicate fonts with artistic and/or cultural significance is needed so you know the standard subjective font quality scores can be ignored for specific typefaces?