Awesome: test failure with spec/beautiful/init_spec.lua

[blueyed]

Appears to be flaky, or caused/triggered by some recent unpinned update:
https://travis-ci.com/awesomeWM/awesome/jobs/178666425

[100%] Running unit tests
●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●◼●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
330 successes / 1 failure / 0 errors / 0 pending : 2.610266 seconds
Failure → spec/beautiful/init_spec.lua @ 14
beautiful init Check beautiful.get_font
spec/beautiful/init_spec.lua:15: Expected objects to be the same.
Passed in:
(userdata) 'lgi.rec 0x14300f0:Pango.FontDescription'
Expected:
(userdata) 'lgi.rec 0xb18520:Pango.FontDescription'
stack traceback:
    spec/beautiful/init_spec.lua:15: in function <spec/beautiful/init_spec.lua:14>

Code: https://github.com/awesomeWM/awesome/blob/e63ec6f25b748b0f58f25bbecc987fe17f72d59b/spec/beautiful/init_spec.lua#L15-L16

/cc @Sorky

[blueyed]

Anyone else seen this?

Might be related to the more-coverage PR (#2672), where it just happened again.
Again with LUA=5.2 LUANAME=lua5.2 DO_COVERAGE=1.

[actionless]

it would be nice to print the fields of the object to see why is_same() didn't passed

btw, can is_same() actually effectively compare userdata?

[psychon]

btw, can is_same() actually effectively compare userdata?

It checks for equality, I think. And beautiful contains a cache for fonts that should actually make sure things match here.

Perhaps the following would help in figuring out what is going on, perhaps it does not.

diff --git a/spec/beautiful/init_spec.lua b/spec/beautiful/init_spec.lua
index 07edb0ac8..529ab94dd 100644
--- a/spec/beautiful/init_spec.lua
+++ b/spec/beautiful/init_spec.lua
@@ -12,8 +12,10 @@ describe("beautiful init", function()

     -- Check beautiful.get_font and get_merged_font
     it('Check beautiful.get_font', function()
-        assert.is_same(beautiful.get_font("Monospace Bold 12"),
-            beautiful.get_merged_font("Monospace 12", "Bold"))
+        local a = beautiful.get_font("Monospace Bold 12")
+        local b = beautiful.get_merged_font("Monospace 12", "Bold")
+        assert.is_same(a:to_string(), b:to_string())
+        assert.is_same(a, b)
     end)

     -- Check beautiful.get_font_height

[blueyed]

Would make sense / it better.
I've only seen it so far with https://github.com/awesomeWM/awesome/pull/2672 however.

[Sorky]

Most of my efforts in creating "spec/beautiful/init_spec.lua" went into the "Check beautiful.init" case

As I didn't work on the other functions I only added cursory tests as a framework for later enhancement (ie: should be expanded, particularly with failure cases)

Given that it has passed many builds from introduction till now I expect it is valid in principle but some edge case or environment condition must now be in play

Focusing on the code being tested...

function beautiful.get_font(name)
    return load_font(name).description
end

function beautiful.get_merged_font(name, merge)
    local font = beautiful.get_font(name)
    merge = Pango.FontDescription.from_string(merge)
    local merged = font:copy_static()
    merged:merge(merge, true)
    return beautiful.get_font(merged:to_string())
end

"beautiful.get_merged_font" reads ok (although the var naming could be better), it works with and returns via "beautiful.get_font" so I can't see anything obvious

Given that the merged font differs...

• Has "Pango.FontDescription" changed for some reason?
• Does the "Monospace 12" font exist?
• Does the "Monospace Bold 12" font exist separately [duplicate]?
• What happens if the test uses a different font?

As per Psychon - Breaking down the above or even digging into "load_font" should more shed light

[Sorky]

I grabbed your branch on my system and it built and busted worked without error

[blueyed]

I grabbed your branch on my system and it built and busted worked without error

I also cannot reproduce it myself, but it appears to happen only sometimes on Travis, i.e. it is flaky.

[psychon]

Uhm. This is related to GC behaviour. The following makes the test always fail:

diff --git a/spec/beautiful/init_spec.lua b/spec/beautiful/init_spec.lua
index 07edb0ac8..3ef8cb51f 100644
--- a/spec/beautiful/init_spec.lua
+++ b/spec/beautiful/init_spec.lua
@@ -11,9 +11,11 @@ describe("beautiful init", function()
     local shim

     -- Check beautiful.get_font and get_merged_font
-    it('Check beautiful.get_font', function()
-        assert.is_same(beautiful.get_font("Monospace Bold 12"),
-            beautiful.get_merged_font("Monospace 12", "Bold"))
+    it('Check beautiful.get_font_with_collect', function()
+        local a = beautiful.get_font("Monospace Bold 12")
+        collectgarbage("collect")
+        local b = beautiful.get_merged_font("Monospace 12", "Bold")
+        assert.is_same(a, b)
     end)

     -- Check beautiful.get_font_height

The key here is to have a GC cycle between the call to get_font and get_merged_font. This causes the font cache in beautiful to be cleared (because it is a table with weak values, but the values are tables that are not referenced from anywhere else, so are always collectable).

I guess this GC stuff could be fixed (but I will not try to fix that right now), but I am not sure if this "caching stuff" really should be part of the "contract" of the API of beautiful.load_font. If it shouldn't, the test needs to test for something else (and I am not quite sure what it tests for currently anyway).

[psychon]

Oh and to make it easier to reproduce the original problem before my patch, I can only reproduce running while busted --filter="get_font" --repeat 100 --no-keep-going ; do : ; done and waiting a bit (just takes some minutes here with Lua 5.2).