Simplecov: Massive difference in .resultset.json output between 0.17.1 and 0.18 (no merging involved)

[fidothe]

I found a massive difference between simplecov's output in 0.18.x vs 0.17.1, which for me resulted in Code Climate reporter failure (exactly as codeclimate/test-reporter#418).

I'm not doing anything complex with simplecov - just requiring it and calling start in my spec_helper.rb:

require 'simplecov'
SimpleCov.start

The project has very few other Ruby dependencies, but it is a JRuby project, and makes extensive use of a Java library.

It's easy to see the results: the coverage/.resultset.json file is largely full of nulls in 0.18.4:

https://github.com/fidothe/saxon-rb/commit/16ea5f06bfcfd8ccb6a15f75852635c4ce9fef6d

and without any nulls in 0.17.1:

https://github.com/fidothe/saxon-rb/commit/91021def1c2f2b9b0904046d9c753510a6667870

The only difference between those two commits is that I pinned SimpleCov back to 0.17.1 in 91021d.

I've tried this under JRuby 9.2.9.0 and 9.2.10.0. I'm assuming that it's in some way related to #853, #854 et al, but since there's none of the merging or parallelizing going on here I thought it was worth reporting it separately. (Also the problem repo is public...)

[PragTob]

Hi @fidothe and thanks a ton for the report! :green_heart:

I'm still amazed at how many ways there are in which changes that shouldn't impact any of this and are completely passing in our relatively good test suite are breaking projects.

I'll put a day aside at the latest on the weekend to try and get to the bottom of this. I might start here as opposed to the others as the scenario appears to be simpler.

Sorry for the inconvenience y'all.

[fidothe]

To run the suite with simplecov correctly under JRuby you need to enable the --debug flag. In CI the suite is run with:

JRUBY_OPTS="--dev --debug" bundle exec rspec spec --profile 10 --format RspecJunitFormatter --out /tmp/test-results/rspec.xml --format progress

Running it with

JRUBY_OPTS="--dev --debug" bundle exec rspec spec --format progress

produces an identical .resultset.json, on my machine at least

[fidothe]

On a whim, I checked into exactly what the difference in the resultset files was.

From 0.17.1:

"/work/saxon-rb/lib/saxon-rb_jars.rb": [
  null,
  1,
  1,
  null,
  0,
  null,
  null,
  1,
  1,
  null
],

0.18.4:

"/work/saxon-rb/lib/saxon-rb_jars.rb": {
  "lines": [
    null,
    1,
    1,
    null,
    0,
    null,
    null,
    1,
    1,
    null
  ]
},

So, the value for each key (file path) in the coverage Object used to be an Array, and is now an Object with a single key: lines, containing the array that used to be the value for the path key...

That certainly looks like it would account for the explosion in the codeclimate test reporter, anyway.

[PragTob]

@fidothe yes that's the reason for the code climate test reporter breaking, see: https://github.com/codeclimate/test-reporter/issues/413

it doesn't explain the coverage itself being wrong though. Because the coverage itself is exactly the same

[fidothe]

Okay, my bad. Transforming the 0.18.4 output so it has no lines key, just the lines as an array and re-ordering the hash of path/lines pairs makes all the differences go away. My fault for using diff -u to investigate the difference between two JSON files.

Very sorry to have given you a fright, @PragTob.