Cabal: Unit test Distribution.Utils.ShortText BinaryId fails

[fgaz]

https://travis-ci.org/haskell-pushbot/cabal-binaries/builds/258926423

$ Cabal/unit-tests
[snip]
  Distribution.Utils.ShortText
    ShortText Id:                                       OK
      +++ OK, passed 100 tests.
    ShortText Ord:                                      OK
      +++ OK, passed 100 tests.
    ShortText Monoid:                                   OK
      +++ OK, passed 100 tests.
    ShortText BinaryId:                                 FAIL
      *** Failed! Falsifiable (after 40 tests and 5 shrinks): 
      "\65534"
      Use --quickcheck-replay=551066 to reproduce.

ping @ezyang

[ezyang]

CC @hvr who introduced this test in 993d20a2e9b8fb29aefaa2c266f31177a00a5ee6

[hvr]

I need to look into why a BOM (which btw makes no sense whatsoever for UTF8 encodings) doesn't round-trip properly. Iirc I specifically tested such corner-cases in the implementation of http://hackage.haskell.org/package/text-short

PS: I just noticed this is with the GHC 7.6.3 configuration, so this may be a problem with the legacy fallback...

[hvr]

After some investigation, the issue is in fact for the String-backed legacy fallback, whose Binary instance relies on the roundtrip property of Distribution.Utils.String.{encode,decode}StringUtf8, which fails for the BOM:

> decodeStringUtf8 ( encodeStringUtf8 "\65534")
"\65533"

because decodeStringUtf8 (imho rightfully) considers a BOM invalid in an UTF8 stream, and maps it to the replacement-character.

[hvr]

I'll take a stab at harmonizing the decodeStringUtf8 semantics with the more round-tripping friendly ones from text and text-short.

[hvr]

I'm confident this one's been fixed via #4928; I ran unit-tests -p BinaryId --quickcheck-tests 999999 compiled for GHC 7.6.3 a few times; and also tried the replay value; everything passed so far.