Fontbakery: [new-arch] A testsuite for the testsuite!

[felipesanches]

While porting fontbakery checks to the new architecture (#1388) I dealt with a bunch of crash bugs (introduced by a few silly mistakes by myself during the port). I think it would be wise to write a testsuite for the checks themselves and doing this while implementing the checks can be good.

@graphicore do you have any suggestion on how to approach that?

[felipesanches]

Here's one example of a check implementation and the code-paths I think a TDD approach should cover:

@register_test
@old_style_test(
    id='com.google.fonts/test/029'
  , conditions=['license']
  , priority=CRITICAL
)
def check_copyright_entries_match_license(fb, ttFont, license):
  """Check copyright namerecords match license file"""
  failed = False
  placeholder = PLACEHOLDER_LICENSING_TEXT[license]
  entry_found = False
  for i, nameRecord in enumerate(ttFont['name'].names):
    if nameRecord.nameID == NAMEID_LICENSE_DESCRIPTION:
      entry_found = True
      value = nameRecord.string.decode(nameRecord.getEncoding())
      if value != placeholder:
        failed = True
        fb.error(('License file {} exists but'
                  ' NameID {} (LICENSE DESCRIPTION) value'
                  ' on platform {} ({})'
                  ' is not specified for that.'
                  ' Value was: "{}"'
                  ' Must be changed to "{}"'
                  '').format(license,
                             NAMEID_LICENSE_DESCRIPTION,
                             nameRecord.platformID,
                             PLATID_STR[nameRecord.platformID],
                             unidecode(value),
                             unidecode(placeholder)))
  if not entry_found:
    fb.error(("Font lacks NameID {} (LICENSE DESCRIPTION)."
              " A proper licensing entry must be set."
              "").format(NAMEID_LICENSE_DESCRIPTION))
  elif not failed:
    fb.ok("licensing entry on name table is correctly set.")

We should have one check for each of these fb ok/error calls:

• Test A: At the first fb.error: validate codepath is reached with a font project that has a license file, but does not have a LICENSE DESCRIPTION properly set in the name table.
• Test B: The second fb.error should be reached if we test it with a font that lacks a LICENSE_DESCRIPTION in its name table.
• Test C: A good font should trigger the fb.ok check result.

[felipesanches]

One concern that I have is that such a test suite could lead to a very large data folder size if it had one font file for each test case of each FB check. So a smarter strategy would be needed. Or, alternatively, a separate testdata repo should be setup as an optional git submodule.

One smarter strategy could involve using fonttools to craft bad fonts on purpose. But on the other hand we may also want to use real-world cases of bad fonts sometimes. (such as using the actual bad fonts that made us create a new check)

[graphicore]

One smarter strategy could involve using fonttools to craft bad fonts on purpose. But on the other hand we may also want to use real-world cases of bad fonts sometimes. (such as using the actual bad fonts that made us create a new check)

I like that, how awesome would it be to have the "one perfect font" and then cripple it on purpose to fail specific tests :-D I know, it's sadistic, but hey, we're talking fonts, right?

[felipesanches]

A-ha!!! I love this concept! Let's do it! :-D

Which family could we use for that purpose ?

[davelab6]

Encode is the largest family so far, so I suggest to fork and rename them
to FontbakeryTest. Probably you can also make "one awful font" that has
many things wrong with it to keep down the clutter LOL

On Jun 30, 2017 11:05 PM, "Felipe Corrêa da Silva Sanches" <
[email protected]> wrote:

A-ha!!! I love this concept! Let's do it! :-D

Which family could we use for that purpose ?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/googlefonts/fontbakery/issues/1413#issuecomment-312406617,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAP9y0DRY3GP83H-wXioWXl4g7-3taJPks5sJbd1gaJpZM4OLG4y
.

[felipesanches]

I'd probably stick to "the perfect family" only, because I want to increase trust on FontBakery. Dealing with a very broken family does not sound too reliable.

PS: Go rest and enjoy your vacations, dude! ;-)

[graphicore]

Hmm, Encode is huge but it lacks a monospaced cut. Maybe we need to use different sources to make it complete. We can still unite them as FontbakeryTest. But hey, if that's going to be the one perfect family, how do we keep people from using it :smiling_imp:

[felipesanches]

well... If we're going to put an effort to get a perfect family, I think it would be great to set it up as an actual real world example that people can use. So the strategy I have in mind would involve:

• selecting the top ten or so high-scoring families based on that fontbakery dashboard frontpage that lists all scores.
• from those ten or so we gotta select one (or 2) which (a) is very popular on Google Fonts (by downloads or usage count perhaps) (b) covers the special cases of monospaced, non-monospaced and any other cases we know we'll need to cover the whole testsuite and (c) ideally have an active designer/team maintaining it on a git repo so that we can submit pull requests to make our fontbakery-informed fixes official in that family on the upstream project.

[felipesanches]

With a bonus-point if the family has all of the 18 possible styles.

[felipesanches]

@graphicore While I'm working on porting the remaining checks, would you be able to allocate some of your time to help me bootstrap the code infrastructure for running a testsuite on our testsuite ?

[graphicore]

So what are the goto unittest frameworks? I could already live with the unittest module, but maybe there's something better. Suggestions?

[davelab6]

Nose?

On Jul 12, 2017 6:43 PM, "Lasse Fister" [e-mail] wrote:

So what are the goto unittest frameworks? I could already live with the
unittest module, but maybe there's something better. Suggestions?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/googlefonts/fontbakery/issues/1413#issuecomment-314918542,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAP9y1T2MI3g5NcGdqwLU4jKe-vXBDfaks5sNUwHgaJpZM4OLG4y
.

[m4rc1e]

@graphicore in #1400 I used the standard unittest frame work with nose. PR is still open though (I'm working on it)

[graphicore]

awesome, I'll look into it!

[graphicore]

I looked a bit around in the python testing world: Since we're on python 2.7.something we're using
unittest2 by default as unittest (unittest2 is a unittest backport for python 2.4-2.6).

nose is not actively maintained, we shouldn't use it:

Nose has been in maintenance mode for the past several years and will likely cease without a new person/team to take over maintainership.

instead nose2 or pytest or pure unittest(2) could be used.

I had a look into all of these and, big surprise, all of these support unittest TestCases.

What I liked was the fixtures feature of pytest, it's a very similar concept to our @condition, including the dependency injection via the argument names in the test cases (that's reassuring for me).

While I'm not very sure what nose2 really adds for us (it's a test runner, OK, but pure unittest can do so as well AFAIK, see Test Discovery) I really like the added fixtures and assertstyle of pytest, so, I'd opt for that. We can still write tests using unittest, so that's not going lost.

On the other hand, I'd understand if you guys prefer using just builtins i.e. unittest to write tests, in that case, I'd give all three test runners a "test run" and see which one produces the best reports.

[graphicore]

There are two examples in the PR #1463 test_id_029_shorter is the preferred one for the case Felipe asked for. Of course, each unit test could be split into more individual functions, I rather mean the style of importing and calling the font-test function directly is better.

The two helper functions are rather quick and dirty, but it triggers all three cases of the test, if all assertions pass.

[davelab6]

Thanks to @graphicore for laying out how to do this, and @felipesanches for plugging away at writing tests :)

When should this be closed?

[felipesanches]

Whenever we reach 100% self-test coverage, perhaps?

[davelab6]

Cool - moving to 0.5.0

[felipesanches]

I feel it is straightforward for us to continue increasing coverage of the code-tests without needing to keep this issue open.