Boulder: TestRecheckCAAFail appears to be flakey

[rolandshoemaker]

Rarely so, but has hit a PR of mine recently.

[cpu]

If you can attach a raw build log with the flake that would be helpful.

[rolandshoemaker]

From here, about the only useful bit is:

--- FAIL: TestRecheckCAAFail (0.11s)
    ra_test.go:2094: false != true
FAIL
FAIL    github.com/letsencrypt/boulder/ra   21.051s

[rolandshoemaker]

(Can't repro locally after running ~100 times, likely due to a slow host.)

[cpu]

Saw this a few times in CI today. Probably worth pulling into next week's sprint. I don't mind taking it.

[cpu]

Adding some debugging prints and running the test a bunch locally with --count 10 helped me spot the bug:

SubErr: errors.SubBoulderError{BoulderError:(*errors.BoulderError)(0xc000158360), Identifier:identifier.ACMEIdentifier{Type:"dns", Value:"a.com"}}
SubErr detail: "CAA invalid for a.com"

SubErr: errors.SubBoulderError{BoulderError:(*errors.BoulderError)(0xc0003e68d0), Identifier:identifier.ACMEIdentifier{Type:"dns", Value:"b.com"}}
SubErr detail: "CAA invalid for c.com"

The ra_test.go:2094: false != true failure is from the unit test looking for a sub problem for the c.com identifier which is mocked to have invalid CAA.

In the buggy output above you can see in the second sub-problem the detail message correctly identifies c.com but the sub-problem is for the wrong identifier (b.com).

The fix is in https://github.com/letsencrypt/boulder/pull/4259